Oct 29 22:30:51.150243 ip-10-0-140-74 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Oct 29 22:30:51.150252 ip-10-0-140-74 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Oct 29 22:30:51.150259 ip-10-0-140-74 systemd[1]: kubelet.service: Failed with result 'resources'. Oct 29 22:30:51.150467 ip-10-0-140-74 systemd[1]: Failed to start Kubernetes Kubelet. Oct 29 22:31:01.399619 ip-10-0-140-74 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Oct 29 22:31:01.399642 ip-10-0-140-74 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 0970734cb3cc455eb10715f90ef1f7ba -- Oct 29 22:32:55.915981 ip-10-0-140-74 systemd[1]: Starting Kubernetes Kubelet... Oct 29 22:32:56.311945 ip-10-0-140-74 kubenswrapper[2577]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 29 22:32:56.311945 ip-10-0-140-74 kubenswrapper[2577]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 29 22:32:56.311945 ip-10-0-140-74 kubenswrapper[2577]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 29 22:32:56.311945 ip-10-0-140-74 kubenswrapper[2577]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 29 22:32:56.315693 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.315599 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 29 22:32:56.315693 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.315692 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 29 22:32:56.315693 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.315696 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 29 22:32:56.315791 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.315699 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 29 22:32:56.315791 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.315702 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 29 22:32:56.315791 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.315705 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 29 22:32:56.315791 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.315707 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 29 22:32:56.315791 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.315710 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 29 22:32:56.315791 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.315713 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 29 22:32:56.315791 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.315716 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 29 22:32:56.315791 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.315719 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 29 22:32:56.315791 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.315722 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 29 22:32:56.315791 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.315724 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 29 22:32:56.315791 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.315727 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 29 22:32:56.315791 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.315730 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 29 22:32:56.315791 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.315734 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 29 22:32:56.315791 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.315738 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 29 22:32:56.315791 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.315742 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 29 22:32:56.315791 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.315745 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 29 22:32:56.315791 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.315748 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 29 22:32:56.315791 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.315750 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 29 22:32:56.315791 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.315754 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 29 22:32:56.316241 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.315757 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 29 22:32:56.316241 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.315759 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 29 22:32:56.316241 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.315762 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 29 22:32:56.316241 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.315764 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 29 22:32:56.316241 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.315767 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 29 22:32:56.316241 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.315770 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 29 22:32:56.316241 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.315772 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 29 22:32:56.316241 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.315775 2577 feature_gate.go:386] unrecognized feature gate: Example2 Oct 29 22:32:56.316241 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.315778 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 29 22:32:56.316241 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.315784 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 29 22:32:56.316241 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.315787 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 29 22:32:56.316241 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.315789 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 29 22:32:56.316241 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.315792 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 29 22:32:56.316241 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.315795 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 29 22:32:56.316241 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.315798 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 29 22:32:56.316241 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.315801 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 29 22:32:56.316241 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.315804 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 29 22:32:56.316241 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.315806 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 29 22:32:56.316241 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.315809 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 29 22:32:56.316241 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.315812 2577 feature_gate.go:386] unrecognized feature gate: Example Oct 29 22:32:56.316728 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.315815 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 29 22:32:56.316728 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.315817 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 29 22:32:56.316728 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.315819 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 29 22:32:56.316728 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.315822 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 29 22:32:56.316728 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.315824 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 29 22:32:56.316728 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.315827 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 29 22:32:56.316728 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.315829 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 29 22:32:56.316728 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.315832 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 29 22:32:56.316728 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.315834 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 29 22:32:56.316728 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.315837 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 29 22:32:56.316728 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.315839 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 29 22:32:56.316728 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.315842 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 29 22:32:56.316728 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.315844 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 29 22:32:56.316728 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.315848 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 29 22:32:56.316728 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.315851 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 29 22:32:56.316728 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.315853 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 29 22:32:56.316728 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.315856 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 29 22:32:56.316728 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.315858 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 29 22:32:56.316728 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.315861 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 29 22:32:56.316728 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.315863 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 29 22:32:56.317193 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.315866 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 29 22:32:56.317193 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.315869 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 29 22:32:56.317193 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.315871 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 29 22:32:56.317193 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.315874 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 29 22:32:56.317193 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.315878 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 29 22:32:56.317193 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.315880 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 29 22:32:56.317193 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.315883 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 29 22:32:56.317193 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.315886 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 29 22:32:56.317193 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.315889 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 29 22:32:56.317193 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.315891 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 29 22:32:56.317193 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.315894 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 29 22:32:56.317193 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.315897 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 29 22:32:56.317193 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.315899 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 29 22:32:56.317193 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.315901 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 29 22:32:56.317193 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.315904 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 29 22:32:56.317193 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.315907 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 29 22:32:56.317193 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.315909 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 29 22:32:56.317193 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.315913 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 29 22:32:56.317193 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.315916 2577 feature_gate.go:386] unrecognized feature gate: EVPN Oct 29 22:32:56.317651 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.315919 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 29 22:32:56.317651 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.315922 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 29 22:32:56.317651 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.315925 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 29 22:32:56.317651 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.315928 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 29 22:32:56.317651 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.315931 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 29 22:32:56.317651 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.315933 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 29 22:32:56.317651 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.315945 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 29 22:32:56.317651 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.315948 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 29 22:32:56.317651 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.315950 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 29 22:32:56.317651 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.315953 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 29 22:32:56.317651 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.315956 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 29 22:32:56.317651 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.315959 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 29 22:32:56.317651 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.315961 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 29 22:32:56.317651 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.315964 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 29 22:32:56.317651 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.315966 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 29 22:32:56.317651 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.315969 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 29 22:32:56.317651 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.315972 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 29 22:32:56.317651 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.315976 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 29 22:32:56.317651 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.315979 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 29 22:32:56.317651 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.315982 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 29 22:32:56.318124 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.315984 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 29 22:32:56.318124 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.315987 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 29 22:32:56.318124 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.315989 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 29 22:32:56.318124 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.315992 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 29 22:32:56.318124 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.315994 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 29 22:32:56.318124 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.316776 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 29 22:32:56.318124 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.316783 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 29 22:32:56.318124 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.316786 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 29 22:32:56.318124 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.316789 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 29 22:32:56.318124 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.316792 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 29 22:32:56.318124 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.316794 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 29 22:32:56.318124 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.316797 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 29 22:32:56.318124 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.316799 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 29 22:32:56.318124 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.316802 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 29 22:32:56.318124 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.316805 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 29 22:32:56.318124 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.316808 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 29 22:32:56.318124 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.316810 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 29 22:32:56.318124 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.316813 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 29 22:32:56.318124 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.316816 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 29 22:32:56.318124 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.316819 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 29 22:32:56.318609 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.316822 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 29 22:32:56.318609 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.316825 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 29 22:32:56.318609 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.316828 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 29 22:32:56.318609 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.316830 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 29 22:32:56.318609 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.316833 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 29 22:32:56.318609 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.316836 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 29 22:32:56.318609 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.316838 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 29 22:32:56.318609 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.316841 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 29 22:32:56.318609 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.316843 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 29 22:32:56.318609 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.316848 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 29 22:32:56.318609 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.316852 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 29 22:32:56.318609 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.316855 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 29 22:32:56.318609 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.316858 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 29 22:32:56.318609 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.316861 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 29 22:32:56.318609 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.316863 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 29 22:32:56.318609 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.316866 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 29 22:32:56.318609 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.316868 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 29 22:32:56.318609 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.316871 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 29 22:32:56.318609 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.316874 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 29 22:32:56.319073 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.316876 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 29 22:32:56.319073 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.316879 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 29 22:32:56.319073 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.316882 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 29 22:32:56.319073 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.316885 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 29 22:32:56.319073 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.316887 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 29 22:32:56.319073 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.316890 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 29 22:32:56.319073 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.316892 2577 feature_gate.go:386] unrecognized feature gate: EVPN Oct 29 22:32:56.319073 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.316895 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 29 22:32:56.319073 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.316897 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 29 22:32:56.319073 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.316902 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 29 22:32:56.319073 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.316905 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 29 22:32:56.319073 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.316908 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 29 22:32:56.319073 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.316910 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 29 22:32:56.319073 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.316913 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 29 22:32:56.319073 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.316915 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 29 22:32:56.319073 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.316918 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 29 22:32:56.319073 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.316920 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 29 22:32:56.319073 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.316923 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 29 22:32:56.319073 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.316925 2577 feature_gate.go:386] unrecognized feature gate: Example Oct 29 22:32:56.319073 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.316928 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 29 22:32:56.320343 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.316931 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 29 22:32:56.320343 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.316933 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 29 22:32:56.320343 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.316936 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 29 22:32:56.320343 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.316939 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 29 22:32:56.320343 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.316941 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 29 22:32:56.320343 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.316944 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 29 22:32:56.320343 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.316946 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 29 22:32:56.320343 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.316949 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 29 22:32:56.320343 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.316951 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 29 22:32:56.320343 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.316954 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 29 22:32:56.320343 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.316957 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 29 22:32:56.320343 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.316959 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 29 22:32:56.320343 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.316962 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 29 22:32:56.320343 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.316964 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 29 22:32:56.320343 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.316967 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 29 22:32:56.320343 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.316970 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 29 22:32:56.320343 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.316972 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 29 22:32:56.320343 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.316974 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 29 22:32:56.320343 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.316977 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 29 22:32:56.320799 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.316980 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 29 22:32:56.320799 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.316983 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 29 22:32:56.320799 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.316985 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 29 22:32:56.320799 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.316988 2577 feature_gate.go:386] unrecognized feature gate: Example2 Oct 29 22:32:56.320799 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.316991 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 29 22:32:56.320799 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.316993 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 29 22:32:56.320799 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.316996 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 29 22:32:56.320799 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.316998 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 29 22:32:56.320799 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.317001 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 29 22:32:56.320799 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.317003 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 29 22:32:56.320799 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.317005 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 29 22:32:56.320799 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.317008 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 29 22:32:56.320799 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.317010 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 29 22:32:56.320799 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.317013 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 29 22:32:56.320799 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.317015 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 29 22:32:56.320799 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.317018 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 29 22:32:56.320799 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.317021 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 29 22:32:56.320799 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.317023 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 29 22:32:56.320799 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.317026 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 29 22:32:56.320799 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.317028 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 29 22:32:56.321293 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.317032 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 29 22:32:56.321293 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.317036 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 29 22:32:56.321293 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.317039 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 29 22:32:56.321293 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.317042 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 29 22:32:56.321293 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.317045 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 29 22:32:56.321293 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.317050 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 29 22:32:56.321293 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.317053 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 29 22:32:56.321293 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.317055 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 29 22:32:56.321293 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.317058 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 29 22:32:56.321293 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.317061 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 29 22:32:56.321293 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.317063 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 29 22:32:56.321293 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.317066 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 29 22:32:56.321293 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.317068 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 29 22:32:56.321293 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.317587 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 29 22:32:56.321293 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.317593 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 29 22:32:56.321293 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.317597 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 29 22:32:56.321293 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.317600 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 29 22:32:56.321293 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.317602 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 29 22:32:56.321293 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.317605 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 29 22:32:56.321763 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.317608 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 29 22:32:56.321763 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.317611 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 29 22:32:56.321763 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.317614 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 29 22:32:56.321763 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.317616 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 29 22:32:56.321763 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.317619 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 29 22:32:56.321763 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.317622 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 29 22:32:56.321763 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.317624 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 29 22:32:56.321763 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.317627 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 29 22:32:56.321763 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.317630 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 29 22:32:56.321763 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.317633 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 29 22:32:56.321763 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.317635 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 29 22:32:56.321763 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.317652 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 29 22:32:56.321763 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.317655 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 29 22:32:56.321763 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.317658 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 29 22:32:56.321763 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.317660 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 29 22:32:56.321763 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.317663 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 29 22:32:56.321763 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.317666 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 29 22:32:56.321763 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.317669 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 29 22:32:56.321763 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.317672 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 29 22:32:56.322213 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.317675 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 29 22:32:56.322213 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.317677 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 29 22:32:56.322213 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.317680 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 29 22:32:56.322213 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.317683 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 29 22:32:56.322213 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.317685 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 29 22:32:56.322213 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.317688 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 29 22:32:56.322213 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.317690 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 29 22:32:56.322213 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.317693 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 29 22:32:56.322213 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.317695 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 29 22:32:56.322213 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.317698 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 29 22:32:56.322213 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.317701 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 29 22:32:56.322213 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.317704 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 29 22:32:56.322213 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.317706 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 29 22:32:56.322213 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.317709 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 29 22:32:56.322213 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.317712 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 29 22:32:56.322213 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.317714 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 29 22:32:56.322213 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.317717 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 29 22:32:56.322213 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.317719 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 29 22:32:56.322213 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.317722 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 29 22:32:56.322213 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.317724 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 29 22:32:56.322727 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.317727 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 29 22:32:56.322727 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.317730 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 29 22:32:56.322727 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.317733 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 29 22:32:56.322727 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.317737 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 29 22:32:56.322727 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.317741 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 29 22:32:56.322727 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.317744 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 29 22:32:56.322727 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.317747 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 29 22:32:56.322727 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.317750 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 29 22:32:56.322727 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.317752 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 29 22:32:56.322727 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.317755 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 29 22:32:56.322727 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.317757 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 29 22:32:56.322727 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.317760 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 29 22:32:56.322727 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.317764 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 29 22:32:56.322727 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.317768 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 29 22:32:56.322727 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.317771 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 29 22:32:56.322727 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.317775 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 29 22:32:56.322727 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.317777 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 29 22:32:56.322727 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.317780 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 29 22:32:56.322727 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.317783 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 29 22:32:56.323279 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.317785 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 29 22:32:56.323279 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.317788 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 29 22:32:56.323279 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.317791 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 29 22:32:56.323279 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.317793 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 29 22:32:56.323279 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.317796 2577 feature_gate.go:386] unrecognized feature gate: EVPN Oct 29 22:32:56.323279 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.317799 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 29 22:32:56.323279 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.317801 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 29 22:32:56.323279 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.317804 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 29 22:32:56.323279 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.317806 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 29 22:32:56.323279 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.317809 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 29 22:32:56.323279 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.317811 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 29 22:32:56.323279 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.317814 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 29 22:32:56.323279 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.317817 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 29 22:32:56.323279 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.317819 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 29 22:32:56.323279 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.317822 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 29 22:32:56.323279 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.317825 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 29 22:32:56.323279 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.317827 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 29 22:32:56.323279 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.317829 2577 feature_gate.go:386] unrecognized feature gate: Example2 Oct 29 22:32:56.323279 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.317832 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 29 22:32:56.323279 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.317834 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 29 22:32:56.323772 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.317837 2577 feature_gate.go:386] unrecognized feature gate: Example Oct 29 22:32:56.323772 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.317839 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 29 22:32:56.323772 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.317842 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 29 22:32:56.323772 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.317845 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 29 22:32:56.323772 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.317847 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 29 22:32:56.323772 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.317850 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 29 22:32:56.323772 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.317853 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 29 22:32:56.323772 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.317855 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 29 22:32:56.323772 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.317858 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 29 22:32:56.323772 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.317860 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 29 22:32:56.323772 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.317863 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 29 22:32:56.323772 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.317866 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 29 22:32:56.323772 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.317868 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 29 22:32:56.323772 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.317871 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 29 22:32:56.323772 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.317873 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 29 22:32:56.323772 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.317876 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 29 22:32:56.323772 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.317878 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 29 22:32:56.323772 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.317882 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 29 22:32:56.323772 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.317884 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 29 22:32:56.324261 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.317887 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 29 22:32:56.324261 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.317889 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 29 22:32:56.324261 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.317892 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 29 22:32:56.324261 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.318812 2577 flags.go:64] FLAG: --address="0.0.0.0" Oct 29 22:32:56.324261 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.318822 2577 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 29 22:32:56.324261 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.318828 2577 flags.go:64] FLAG: --anonymous-auth="true" Oct 29 22:32:56.324261 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.318832 2577 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 29 22:32:56.324261 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.318837 2577 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 29 22:32:56.324261 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.318840 2577 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 29 22:32:56.324261 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.318844 2577 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 29 22:32:56.324261 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.318849 2577 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 29 22:32:56.324261 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.318852 2577 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 29 22:32:56.324261 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.318855 2577 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 29 22:32:56.324261 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.318858 2577 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 29 22:32:56.324261 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.318862 2577 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 29 22:32:56.324261 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.318865 2577 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 29 22:32:56.324261 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.318868 2577 flags.go:64] FLAG: --cgroup-root="" Oct 29 22:32:56.324261 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.318871 2577 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 29 22:32:56.324261 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.318874 2577 flags.go:64] FLAG: --client-ca-file="" Oct 29 22:32:56.324261 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.318877 2577 flags.go:64] FLAG: --cloud-provider="external" Oct 29 22:32:56.324261 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.318880 2577 flags.go:64] FLAG: --cluster-dns="[]" Oct 29 22:32:56.324261 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.318885 2577 flags.go:64] FLAG: --cluster-domain="" Oct 29 22:32:56.324261 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.318888 2577 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 29 22:32:56.324261 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.318891 2577 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 29 22:32:56.324848 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.318893 2577 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 29 22:32:56.324848 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.318897 2577 flags.go:64] FLAG: --container-log-max-files="5" Oct 29 22:32:56.324848 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.318900 2577 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 29 22:32:56.324848 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.318903 2577 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 29 22:32:56.324848 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.318906 2577 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 29 22:32:56.324848 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.318909 2577 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 29 22:32:56.324848 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.318912 2577 flags.go:64] FLAG: --contention-profiling="false" Oct 29 22:32:56.324848 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.318916 2577 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 29 22:32:56.324848 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.318919 2577 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 29 22:32:56.324848 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.318922 2577 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 29 22:32:56.324848 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.318925 2577 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 29 22:32:56.324848 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.318932 2577 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 29 22:32:56.324848 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.318935 2577 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 29 22:32:56.324848 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.318938 2577 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 29 22:32:56.324848 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.318941 2577 flags.go:64] FLAG: --enable-load-reader="false" Oct 29 22:32:56.324848 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.318944 2577 flags.go:64] FLAG: --enable-server="true" Oct 29 22:32:56.324848 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.318947 2577 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 29 22:32:56.324848 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.318951 2577 flags.go:64] FLAG: --event-burst="100" Oct 29 22:32:56.324848 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.318954 2577 flags.go:64] FLAG: --event-qps="50" Oct 29 22:32:56.324848 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.318957 2577 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 29 22:32:56.324848 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.318960 2577 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 29 22:32:56.324848 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.318963 2577 flags.go:64] FLAG: --eviction-hard="" Oct 29 22:32:56.324848 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.318968 2577 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 29 22:32:56.324848 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.318970 2577 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 29 22:32:56.325452 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.318975 2577 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 29 22:32:56.325452 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.318978 2577 flags.go:64] FLAG: --eviction-soft="" Oct 29 22:32:56.325452 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.318981 2577 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 29 22:32:56.325452 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.318984 2577 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 29 22:32:56.325452 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.318987 2577 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 29 22:32:56.325452 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.318990 2577 flags.go:64] FLAG: --experimental-mounter-path="" Oct 29 22:32:56.325452 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.318993 2577 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 29 22:32:56.325452 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.318996 2577 flags.go:64] FLAG: --fail-swap-on="true" Oct 29 22:32:56.325452 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.318999 2577 flags.go:64] FLAG: --feature-gates="" Oct 29 22:32:56.325452 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.319003 2577 flags.go:64] FLAG: --file-check-frequency="20s" Oct 29 22:32:56.325452 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.319006 2577 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 29 22:32:56.325452 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.319009 2577 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 29 22:32:56.325452 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.319012 2577 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 29 22:32:56.325452 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.319015 2577 flags.go:64] FLAG: --healthz-port="10248" Oct 29 22:32:56.325452 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.319019 2577 flags.go:64] FLAG: --help="false" Oct 29 22:32:56.325452 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.319022 2577 flags.go:64] FLAG: --hostname-override="ip-10-0-140-74.ec2.internal" Oct 29 22:32:56.325452 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.319026 2577 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 29 22:32:56.325452 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.319029 2577 flags.go:64] FLAG: --http-check-frequency="20s" Oct 29 22:32:56.325452 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.319032 2577 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 29 22:32:56.325452 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.319037 2577 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 29 22:32:56.325452 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.319040 2577 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 29 22:32:56.325452 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.319043 2577 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 29 22:32:56.325452 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.319046 2577 flags.go:64] FLAG: --image-service-endpoint="" Oct 29 22:32:56.326019 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.319049 2577 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 29 22:32:56.326019 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.319052 2577 flags.go:64] FLAG: --kube-api-burst="100" Oct 29 22:32:56.326019 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.319054 2577 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 29 22:32:56.326019 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.319058 2577 flags.go:64] FLAG: --kube-api-qps="50" Oct 29 22:32:56.326019 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.319061 2577 flags.go:64] FLAG: --kube-reserved="" Oct 29 22:32:56.326019 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.319064 2577 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 29 22:32:56.326019 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.319066 2577 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 29 22:32:56.326019 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.319070 2577 flags.go:64] FLAG: --kubelet-cgroups="" Oct 29 22:32:56.326019 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.319073 2577 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 29 22:32:56.326019 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.319077 2577 flags.go:64] FLAG: --lock-file="" Oct 29 22:32:56.326019 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.319080 2577 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 29 22:32:56.326019 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.319083 2577 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 29 22:32:56.326019 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.319087 2577 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 29 22:32:56.326019 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.319092 2577 flags.go:64] FLAG: --log-json-split-stream="false" Oct 29 22:32:56.326019 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.319095 2577 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 29 22:32:56.326019 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.319098 2577 flags.go:64] FLAG: --log-text-split-stream="false" Oct 29 22:32:56.326019 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.319101 2577 flags.go:64] FLAG: --logging-format="text" Oct 29 22:32:56.326019 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.319104 2577 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 29 22:32:56.326019 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.319108 2577 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 29 22:32:56.326019 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.319110 2577 flags.go:64] FLAG: --manifest-url="" Oct 29 22:32:56.326019 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.319113 2577 flags.go:64] FLAG: --manifest-url-header="" Oct 29 22:32:56.326019 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.319118 2577 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 29 22:32:56.326019 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.319121 2577 flags.go:64] FLAG: --max-open-files="1000000" Oct 29 22:32:56.326019 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.319125 2577 flags.go:64] FLAG: --max-pods="110" Oct 29 22:32:56.326019 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.319128 2577 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 29 22:32:56.326605 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.319131 2577 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 29 22:32:56.326605 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.319140 2577 flags.go:64] FLAG: --memory-manager-policy="None" Oct 29 22:32:56.326605 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.319143 2577 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 29 22:32:56.326605 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.319147 2577 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 29 22:32:56.326605 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.319150 2577 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 29 22:32:56.326605 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.319153 2577 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 29 22:32:56.326605 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.319161 2577 flags.go:64] FLAG: --node-status-max-images="50" Oct 29 22:32:56.326605 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.319164 2577 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 29 22:32:56.326605 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.319167 2577 flags.go:64] FLAG: --oom-score-adj="-999" Oct 29 22:32:56.326605 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.319170 2577 flags.go:64] FLAG: --pod-cidr="" Oct 29 22:32:56.326605 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.319173 2577 flags.go:64] FLAG: --pod-manifest-path="" Oct 29 22:32:56.326605 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.319176 2577 flags.go:64] FLAG: --pod-max-pids="-1" Oct 29 22:32:56.326605 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.319179 2577 flags.go:64] FLAG: --pods-per-core="0" Oct 29 22:32:56.326605 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.319182 2577 flags.go:64] FLAG: --port="10250" Oct 29 22:32:56.326605 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.319184 2577 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 29 22:32:56.326605 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.319187 2577 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0beae7170e413ea33" Oct 29 22:32:56.326605 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.319192 2577 flags.go:64] FLAG: --qos-reserved="" Oct 29 22:32:56.326605 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.319195 2577 flags.go:64] FLAG: --read-only-port="10255" Oct 29 22:32:56.326605 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.319198 2577 flags.go:64] FLAG: --register-node="true" Oct 29 22:32:56.326605 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.319201 2577 flags.go:64] FLAG: --register-with-taints="" Oct 29 22:32:56.326605 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.319205 2577 flags.go:64] FLAG: --registry-burst="10" Oct 29 22:32:56.326605 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.319208 2577 flags.go:64] FLAG: --registry-qps="5" Oct 29 22:32:56.326605 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.319211 2577 flags.go:64] FLAG: --reserved-cpus="" Oct 29 22:32:56.326605 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.319214 2577 flags.go:64] FLAG: --reserved-memory="" Oct 29 22:32:56.326605 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.319218 2577 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 29 22:32:56.326605 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.319221 2577 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 29 22:32:56.327242 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.319224 2577 flags.go:64] FLAG: --rotate-certificates="false" Oct 29 22:32:56.327242 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.319226 2577 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 29 22:32:56.327242 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.319229 2577 flags.go:64] FLAG: --runonce="false" Oct 29 22:32:56.327242 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.319232 2577 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 29 22:32:56.327242 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.319236 2577 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 29 22:32:56.327242 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.319239 2577 flags.go:64] FLAG: --seccomp-default="false" Oct 29 22:32:56.327242 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.319242 2577 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 29 22:32:56.327242 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.319244 2577 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 29 22:32:56.327242 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.319253 2577 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 29 22:32:56.327242 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.319257 2577 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 29 22:32:56.327242 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.319261 2577 flags.go:64] FLAG: --storage-driver-password="root" Oct 29 22:32:56.327242 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.319263 2577 flags.go:64] FLAG: --storage-driver-secure="false" Oct 29 22:32:56.327242 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.319266 2577 flags.go:64] FLAG: --storage-driver-table="stats" Oct 29 22:32:56.327242 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.319269 2577 flags.go:64] FLAG: --storage-driver-user="root" Oct 29 22:32:56.327242 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.319272 2577 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 29 22:32:56.327242 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.319275 2577 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 29 22:32:56.327242 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.319278 2577 flags.go:64] FLAG: --system-cgroups="" Oct 29 22:32:56.327242 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.319281 2577 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 29 22:32:56.327242 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.319286 2577 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 29 22:32:56.327242 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.319289 2577 flags.go:64] FLAG: --tls-cert-file="" Oct 29 22:32:56.327242 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.319292 2577 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 29 22:32:56.327242 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.319296 2577 flags.go:64] FLAG: --tls-min-version="" Oct 29 22:32:56.327242 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.319300 2577 flags.go:64] FLAG: --tls-private-key-file="" Oct 29 22:32:56.327242 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.319303 2577 flags.go:64] FLAG: --topology-manager-policy="none" Oct 29 22:32:56.327242 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.319306 2577 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 29 22:32:56.327864 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.319309 2577 flags.go:64] FLAG: --topology-manager-scope="container" Oct 29 22:32:56.327864 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.319312 2577 flags.go:64] FLAG: --v="2" Oct 29 22:32:56.327864 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.319316 2577 flags.go:64] FLAG: --version="false" Oct 29 22:32:56.327864 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.319320 2577 flags.go:64] FLAG: --vmodule="" Oct 29 22:32:56.327864 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.319324 2577 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 29 22:32:56.327864 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.319327 2577 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 29 22:32:56.327864 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.319457 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 29 22:32:56.327864 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.319460 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 29 22:32:56.327864 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.319463 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 29 22:32:56.327864 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.319466 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 29 22:32:56.327864 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.319469 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 29 22:32:56.327864 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.319471 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 29 22:32:56.327864 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.319474 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 29 22:32:56.327864 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.319477 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 29 22:32:56.327864 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.319479 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 29 22:32:56.327864 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.319482 2577 feature_gate.go:386] unrecognized feature gate: Example Oct 29 22:32:56.327864 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.319486 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 29 22:32:56.327864 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.319489 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 29 22:32:56.327864 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.319492 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 29 22:32:56.327864 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.319494 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 29 22:32:56.327864 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.319497 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 29 22:32:56.328367 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.319500 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 29 22:32:56.328367 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.319502 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 29 22:32:56.328367 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.319505 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 29 22:32:56.328367 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.319507 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 29 22:32:56.328367 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.319510 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 29 22:32:56.328367 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.319512 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 29 22:32:56.328367 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.319515 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 29 22:32:56.328367 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.319517 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 29 22:32:56.328367 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.319521 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 29 22:32:56.328367 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.319524 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 29 22:32:56.328367 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.319526 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 29 22:32:56.328367 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.319529 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 29 22:32:56.328367 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.319532 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 29 22:32:56.328367 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.319535 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 29 22:32:56.328367 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.319537 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 29 22:32:56.328367 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.319540 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 29 22:32:56.328367 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.319543 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 29 22:32:56.328367 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.319545 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 29 22:32:56.328367 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.319548 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 29 22:32:56.328367 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.319550 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 29 22:32:56.328855 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.319553 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 29 22:32:56.328855 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.319555 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 29 22:32:56.328855 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.319558 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 29 22:32:56.328855 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.319561 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 29 22:32:56.328855 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.319563 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 29 22:32:56.328855 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.319566 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 29 22:32:56.328855 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.319568 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 29 22:32:56.328855 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.319572 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 29 22:32:56.328855 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.319577 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 29 22:32:56.328855 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.319580 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 29 22:32:56.328855 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.319584 2577 feature_gate.go:386] unrecognized feature gate: EVPN Oct 29 22:32:56.328855 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.319586 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 29 22:32:56.328855 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.319589 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 29 22:32:56.328855 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.319592 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 29 22:32:56.328855 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.319595 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 29 22:32:56.328855 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.319597 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 29 22:32:56.328855 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.319600 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 29 22:32:56.328855 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.319602 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 29 22:32:56.328855 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.319605 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 29 22:32:56.329299 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.319607 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 29 22:32:56.329299 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.319611 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 29 22:32:56.329299 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.319614 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 29 22:32:56.329299 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.319617 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 29 22:32:56.329299 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.319619 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 29 22:32:56.329299 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.319622 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 29 22:32:56.329299 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.319625 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 29 22:32:56.329299 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.319627 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 29 22:32:56.329299 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.319630 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 29 22:32:56.329299 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.319633 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 29 22:32:56.329299 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.319637 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 29 22:32:56.329299 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.319651 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 29 22:32:56.329299 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.319654 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 29 22:32:56.329299 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.319656 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 29 22:32:56.329299 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.319659 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 29 22:32:56.329299 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.319662 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 29 22:32:56.329299 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.319664 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 29 22:32:56.329299 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.319667 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 29 22:32:56.329299 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.319669 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 29 22:32:56.329754 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.319672 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 29 22:32:56.329754 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.319676 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 29 22:32:56.329754 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.319679 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 29 22:32:56.329754 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.319682 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 29 22:32:56.329754 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.319684 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 29 22:32:56.329754 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.319687 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 29 22:32:56.329754 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.319690 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 29 22:32:56.329754 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.319692 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 29 22:32:56.329754 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.319695 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 29 22:32:56.329754 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.319697 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 29 22:32:56.329754 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.319700 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 29 22:32:56.329754 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.319703 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 29 22:32:56.329754 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.319705 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 29 22:32:56.329754 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.319708 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 29 22:32:56.329754 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.319711 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 29 22:32:56.329754 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.319714 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 29 22:32:56.329754 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.319716 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 29 22:32:56.329754 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.319719 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 29 22:32:56.329754 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.319722 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 29 22:32:56.330198 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.319724 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 29 22:32:56.330198 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.319727 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 29 22:32:56.330198 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.319730 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 29 22:32:56.330198 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.319732 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 29 22:32:56.330198 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.319734 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 29 22:32:56.330198 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.319737 2577 feature_gate.go:386] unrecognized feature gate: Example2 Oct 29 22:32:56.330198 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.319740 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 29 22:32:56.330198 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.319743 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 29 22:32:56.330198 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.319745 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 29 22:32:56.330198 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.319748 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 29 22:32:56.330198 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.319750 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 29 22:32:56.330198 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.319753 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 29 22:32:56.330198 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.319755 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 29 22:32:56.330198 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.319758 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 29 22:32:56.330198 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.319784 2577 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 29 22:32:56.330198 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.319820 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 29 22:32:56.330574 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.319824 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 29 22:32:56.330574 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.319828 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 29 22:32:56.330574 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.319831 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 29 22:32:56.330574 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.319835 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 29 22:32:56.330574 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.319837 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 29 22:32:56.330574 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.319840 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 29 22:32:56.330574 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.319843 2577 feature_gate.go:386] unrecognized feature gate: Example2 Oct 29 22:32:56.330574 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.319845 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 29 22:32:56.330574 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.319848 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 29 22:32:56.330574 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.319851 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 29 22:32:56.330574 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.319853 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 29 22:32:56.330574 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.319856 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 29 22:32:56.330574 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.319859 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 29 22:32:56.330574 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.319862 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 29 22:32:56.330574 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.319864 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 29 22:32:56.330574 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.319867 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 29 22:32:56.330574 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.319870 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 29 22:32:56.330574 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.319872 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 29 22:32:56.330574 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.319875 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 29 22:32:56.331046 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.319878 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 29 22:32:56.331046 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.319880 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 29 22:32:56.331046 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.319883 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 29 22:32:56.331046 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.319886 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 29 22:32:56.331046 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.319888 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 29 22:32:56.331046 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.319891 2577 feature_gate.go:386] unrecognized feature gate: Example Oct 29 22:32:56.331046 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.319893 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 29 22:32:56.331046 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.319896 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 29 22:32:56.331046 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.319899 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 29 22:32:56.331046 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.319901 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 29 22:32:56.331046 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.319904 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 29 22:32:56.331046 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.319908 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 29 22:32:56.331046 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.319910 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 29 22:32:56.331046 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.319913 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 29 22:32:56.331046 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.319915 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 29 22:32:56.331046 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.319919 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 29 22:32:56.331046 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.319921 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 29 22:32:56.331046 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.319924 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 29 22:32:56.331046 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.319926 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 29 22:32:56.331046 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.319929 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 29 22:32:56.331542 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.319931 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 29 22:32:56.331542 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.319934 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 29 22:32:56.331542 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.319945 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 29 22:32:56.331542 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.319948 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 29 22:32:56.331542 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.319951 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 29 22:32:56.331542 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.319955 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 29 22:32:56.331542 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.319958 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 29 22:32:56.331542 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.319960 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 29 22:32:56.331542 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.319963 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 29 22:32:56.331542 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.319966 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 29 22:32:56.331542 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.319969 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 29 22:32:56.331542 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.319972 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 29 22:32:56.331542 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.319974 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 29 22:32:56.331542 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.319977 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 29 22:32:56.331542 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.319980 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 29 22:32:56.331542 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.319982 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 29 22:32:56.331542 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.319985 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 29 22:32:56.331542 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.319987 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 29 22:32:56.331542 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.319990 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 29 22:32:56.331542 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.319994 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 29 22:32:56.332109 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.319997 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 29 22:32:56.332109 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.320000 2577 feature_gate.go:386] unrecognized feature gate: EVPN Oct 29 22:32:56.332109 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.320003 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 29 22:32:56.332109 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.320007 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 29 22:32:56.332109 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.320009 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 29 22:32:56.332109 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.320012 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 29 22:32:56.332109 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.320014 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 29 22:32:56.332109 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.320017 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 29 22:32:56.332109 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.320020 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 29 22:32:56.332109 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.320023 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 29 22:32:56.332109 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.320025 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 29 22:32:56.332109 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.320028 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 29 22:32:56.332109 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.320030 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 29 22:32:56.332109 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.320033 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 29 22:32:56.332109 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.320035 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 29 22:32:56.332109 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.320038 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 29 22:32:56.332109 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.320041 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 29 22:32:56.332109 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.320044 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 29 22:32:56.332109 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.320047 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 29 22:32:56.332579 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.320049 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 29 22:32:56.332579 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.320053 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 29 22:32:56.332579 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.320058 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 29 22:32:56.332579 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.320061 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 29 22:32:56.332579 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.320063 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 29 22:32:56.332579 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.320066 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 29 22:32:56.332579 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.320069 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 29 22:32:56.332579 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.320072 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 29 22:32:56.332579 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.320075 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 29 22:32:56.332579 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.320077 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 29 22:32:56.332579 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.320080 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 29 22:32:56.332579 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.320082 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 29 22:32:56.332579 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.320085 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 29 22:32:56.332579 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.320087 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 29 22:32:56.332579 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.320090 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 29 22:32:56.332579 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.320093 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 29 22:32:56.332579 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.320099 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 29 22:32:56.332579 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.320102 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 29 22:32:56.332579 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.320104 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 29 22:32:56.333127 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.320107 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 29 22:32:56.333127 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.320109 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 29 22:32:56.333127 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.320112 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 29 22:32:56.333127 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.320115 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 29 22:32:56.333127 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.320117 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 29 22:32:56.333127 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.320120 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 29 22:32:56.333127 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.320123 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 29 22:32:56.333127 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.320126 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 29 22:32:56.333127 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.326329 2577 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 29 22:32:56.333127 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.326342 2577 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 29 22:32:56.333127 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.327012 2577 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 29 22:32:56.333127 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.327019 2577 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 29 22:32:56.333127 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327076 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 29 22:32:56.333127 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327081 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 29 22:32:56.333127 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327084 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 29 22:32:56.333127 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327087 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 29 22:32:56.333127 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327090 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 29 22:32:56.333127 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327093 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 29 22:32:56.333127 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327097 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 29 22:32:56.333609 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327101 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 29 22:32:56.333609 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327105 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 29 22:32:56.333609 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327108 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 29 22:32:56.333609 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327111 2577 feature_gate.go:386] unrecognized feature gate: Example2 Oct 29 22:32:56.333609 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327113 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 29 22:32:56.333609 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327116 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 29 22:32:56.333609 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327119 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 29 22:32:56.333609 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327122 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 29 22:32:56.333609 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327125 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 29 22:32:56.333609 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327128 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 29 22:32:56.333609 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327130 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 29 22:32:56.333609 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327133 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 29 22:32:56.333609 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327136 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 29 22:32:56.333609 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327138 2577 feature_gate.go:386] unrecognized feature gate: EVPN Oct 29 22:32:56.333609 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327141 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 29 22:32:56.333609 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327144 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 29 22:32:56.333609 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327147 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 29 22:32:56.333609 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327150 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 29 22:32:56.333609 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327152 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 29 22:32:56.333609 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327155 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 29 22:32:56.333609 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327158 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 29 22:32:56.334109 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327160 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 29 22:32:56.334109 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327164 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 29 22:32:56.334109 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327166 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 29 22:32:56.334109 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327170 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 29 22:32:56.334109 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327172 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 29 22:32:56.334109 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327175 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 29 22:32:56.334109 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327177 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 29 22:32:56.334109 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327180 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 29 22:32:56.334109 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327182 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 29 22:32:56.334109 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327185 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 29 22:32:56.334109 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327187 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 29 22:32:56.334109 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327190 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 29 22:32:56.334109 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327193 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 29 22:32:56.334109 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327195 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 29 22:32:56.334109 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327198 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 29 22:32:56.334109 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327200 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 29 22:32:56.334109 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327203 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 29 22:32:56.334109 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327206 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 29 22:32:56.334109 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327208 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 29 22:32:56.334109 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327211 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 29 22:32:56.334650 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327213 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 29 22:32:56.334650 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327216 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 29 22:32:56.334650 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327218 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 29 22:32:56.334650 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327221 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 29 22:32:56.334650 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327225 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 29 22:32:56.334650 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327229 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 29 22:32:56.334650 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327232 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 29 22:32:56.334650 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327235 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 29 22:32:56.334650 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327238 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 29 22:32:56.334650 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327241 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 29 22:32:56.334650 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327244 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 29 22:32:56.334650 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327247 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 29 22:32:56.334650 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327250 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 29 22:32:56.334650 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327254 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 29 22:32:56.334650 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327257 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 29 22:32:56.334650 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327259 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 29 22:32:56.334650 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327262 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 29 22:32:56.334650 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327265 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 29 22:32:56.334650 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327267 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 29 22:32:56.335134 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327270 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 29 22:32:56.335134 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327273 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 29 22:32:56.335134 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327275 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 29 22:32:56.335134 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327278 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 29 22:32:56.335134 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327280 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 29 22:32:56.335134 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327283 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 29 22:32:56.335134 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327286 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 29 22:32:56.335134 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327289 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 29 22:32:56.335134 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327291 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 29 22:32:56.335134 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327294 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 29 22:32:56.335134 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327297 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 29 22:32:56.335134 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327299 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 29 22:32:56.335134 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327302 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 29 22:32:56.335134 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327304 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 29 22:32:56.335134 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327307 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 29 22:32:56.335134 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327310 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 29 22:32:56.335134 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327312 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 29 22:32:56.335134 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327315 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 29 22:32:56.335134 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327317 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 29 22:32:56.335902 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327319 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 29 22:32:56.335902 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327322 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 29 22:32:56.335902 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327324 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 29 22:32:56.335902 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327327 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 29 22:32:56.335902 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327330 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 29 22:32:56.335902 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327333 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 29 22:32:56.335902 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327336 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 29 22:32:56.335902 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327339 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 29 22:32:56.335902 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327342 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 29 22:32:56.335902 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327344 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 29 22:32:56.335902 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327346 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 29 22:32:56.335902 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327349 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 29 22:32:56.335902 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327352 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 29 22:32:56.335902 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327354 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 29 22:32:56.335902 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327357 2577 feature_gate.go:386] unrecognized feature gate: Example Oct 29 22:32:56.335902 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327359 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 29 22:32:56.335902 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327362 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 29 22:32:56.335902 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327364 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 29 22:32:56.335902 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327367 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 29 22:32:56.336384 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327369 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 29 22:32:56.336384 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.327396 2577 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 29 22:32:56.336384 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327553 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 29 22:32:56.336384 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327558 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 29 22:32:56.336384 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327562 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 29 22:32:56.336384 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327565 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 29 22:32:56.336384 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327568 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 29 22:32:56.336384 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327571 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 29 22:32:56.336384 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327574 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 29 22:32:56.336384 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327577 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 29 22:32:56.336384 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327580 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 29 22:32:56.336384 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327583 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 29 22:32:56.336384 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327585 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 29 22:32:56.336384 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327588 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 29 22:32:56.336384 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327591 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 29 22:32:56.336384 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327594 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 29 22:32:56.336927 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327596 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 29 22:32:56.336927 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327600 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 29 22:32:56.336927 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327602 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 29 22:32:56.336927 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327605 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 29 22:32:56.336927 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327608 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 29 22:32:56.336927 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327610 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 29 22:32:56.336927 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327613 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 29 22:32:56.336927 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327616 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 29 22:32:56.336927 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327618 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 29 22:32:56.336927 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327621 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 29 22:32:56.336927 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327623 2577 feature_gate.go:386] unrecognized feature gate: Example Oct 29 22:32:56.336927 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327626 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 29 22:32:56.336927 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327628 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 29 22:32:56.336927 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327630 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 29 22:32:56.336927 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327633 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 29 22:32:56.336927 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327635 2577 feature_gate.go:386] unrecognized feature gate: EVPN Oct 29 22:32:56.336927 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327638 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 29 22:32:56.336927 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327655 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 29 22:32:56.336927 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327658 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 29 22:32:56.336927 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327661 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 29 22:32:56.337659 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327663 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 29 22:32:56.337659 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327666 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 29 22:32:56.337659 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327668 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 29 22:32:56.337659 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327671 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 29 22:32:56.337659 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327674 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 29 22:32:56.337659 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327676 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 29 22:32:56.337659 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327678 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 29 22:32:56.337659 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327681 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 29 22:32:56.337659 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327684 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 29 22:32:56.337659 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327686 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 29 22:32:56.337659 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327689 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 29 22:32:56.337659 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327691 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 29 22:32:56.337659 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327694 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 29 22:32:56.337659 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327696 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 29 22:32:56.337659 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327700 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 29 22:32:56.337659 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327702 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 29 22:32:56.337659 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327705 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 29 22:32:56.337659 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327708 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 29 22:32:56.337659 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327710 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 29 22:32:56.338368 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327713 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 29 22:32:56.338368 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327715 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 29 22:32:56.338368 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327718 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 29 22:32:56.338368 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327720 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 29 22:32:56.338368 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327725 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 29 22:32:56.338368 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327728 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 29 22:32:56.338368 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327731 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 29 22:32:56.338368 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327734 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 29 22:32:56.338368 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327737 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 29 22:32:56.338368 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327740 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 29 22:32:56.338368 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327743 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 29 22:32:56.338368 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327746 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 29 22:32:56.338368 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327749 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 29 22:32:56.338368 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327751 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 29 22:32:56.338368 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327754 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 29 22:32:56.338368 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327756 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 29 22:32:56.338368 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327759 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 29 22:32:56.338368 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327761 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 29 22:32:56.338368 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327764 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 29 22:32:56.338886 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327766 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 29 22:32:56.338886 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327769 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 29 22:32:56.338886 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327771 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 29 22:32:56.338886 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327774 2577 feature_gate.go:386] unrecognized feature gate: Example2 Oct 29 22:32:56.338886 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327776 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 29 22:32:56.338886 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327779 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 29 22:32:56.338886 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327782 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 29 22:32:56.338886 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327784 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 29 22:32:56.338886 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327787 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 29 22:32:56.338886 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327790 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 29 22:32:56.338886 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327792 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 29 22:32:56.338886 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327795 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 29 22:32:56.338886 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327797 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 29 22:32:56.338886 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327800 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 29 22:32:56.338886 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327802 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 29 22:32:56.338886 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327804 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 29 22:32:56.338886 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327807 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 29 22:32:56.338886 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327809 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 29 22:32:56.338886 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327812 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 29 22:32:56.338886 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327814 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 29 22:32:56.339471 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327817 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 29 22:32:56.339471 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327819 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 29 22:32:56.339471 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327823 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 29 22:32:56.339471 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327827 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 29 22:32:56.339471 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327830 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 29 22:32:56.339471 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327833 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 29 22:32:56.339471 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327836 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 29 22:32:56.339471 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327839 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 29 22:32:56.339471 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327841 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 29 22:32:56.339471 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327844 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 29 22:32:56.339471 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327846 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 29 22:32:56.339471 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327849 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 29 22:32:56.339471 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327852 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 29 22:32:56.339471 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327854 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 29 22:32:56.339471 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.327877 2577 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 29 22:32:56.339889 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327909 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 29 22:32:56.339889 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327913 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 29 22:32:56.339889 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327916 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 29 22:32:56.339889 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327920 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 29 22:32:56.339889 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327922 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 29 22:32:56.339889 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327925 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 29 22:32:56.339889 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327928 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 29 22:32:56.339889 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327933 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 29 22:32:56.339889 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327947 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 29 22:32:56.339889 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327951 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 29 22:32:56.339889 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327954 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 29 22:32:56.339889 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327956 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 29 22:32:56.339889 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327959 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 29 22:32:56.339889 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327961 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 29 22:32:56.339889 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327964 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 29 22:32:56.339889 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327967 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 29 22:32:56.339889 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327970 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 29 22:32:56.339889 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327972 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 29 22:32:56.339889 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327975 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 29 22:32:56.340388 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327978 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 29 22:32:56.340388 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327980 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 29 22:32:56.340388 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327983 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 29 22:32:56.340388 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327986 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 29 22:32:56.340388 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327989 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 29 22:32:56.340388 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327992 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 29 22:32:56.340388 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327995 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 29 22:32:56.340388 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.327997 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 29 22:32:56.340388 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.328000 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 29 22:32:56.340388 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.328003 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 29 22:32:56.340388 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.328006 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 29 22:32:56.340388 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.328008 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 29 22:32:56.340388 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.328011 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 29 22:32:56.340388 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.328013 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 29 22:32:56.340388 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.328016 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 29 22:32:56.340388 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.328018 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 29 22:32:56.340388 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.328021 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 29 22:32:56.340388 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.328023 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 29 22:32:56.340388 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.328026 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 29 22:32:56.340388 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.328028 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 29 22:32:56.341115 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.328031 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 29 22:32:56.341115 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.328035 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 29 22:32:56.341115 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.328038 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 29 22:32:56.341115 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.328040 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 29 22:32:56.341115 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.328043 2577 feature_gate.go:386] unrecognized feature gate: Example Oct 29 22:32:56.341115 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.328045 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 29 22:32:56.341115 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.328048 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 29 22:32:56.341115 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.328050 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 29 22:32:56.341115 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.328053 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 29 22:32:56.341115 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.328055 2577 feature_gate.go:386] unrecognized feature gate: EVPN Oct 29 22:32:56.341115 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.328058 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 29 22:32:56.341115 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.328060 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 29 22:32:56.341115 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.328063 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 29 22:32:56.341115 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.328065 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 29 22:32:56.341115 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.328068 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 29 22:32:56.341115 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.328070 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 29 22:32:56.341115 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.328072 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 29 22:32:56.341115 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.328075 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 29 22:32:56.341115 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.328078 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 29 22:32:56.341115 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.328080 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 29 22:32:56.341628 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.328083 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 29 22:32:56.341628 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.328087 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 29 22:32:56.341628 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.328090 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 29 22:32:56.341628 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.328092 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 29 22:32:56.341628 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.328095 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 29 22:32:56.341628 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.328097 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 29 22:32:56.341628 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.328100 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 29 22:32:56.341628 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.328103 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 29 22:32:56.341628 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.328105 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 29 22:32:56.341628 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.328108 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 29 22:32:56.341628 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.328110 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 29 22:32:56.341628 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.328113 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 29 22:32:56.341628 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.328116 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 29 22:32:56.341628 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.328118 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 29 22:32:56.341628 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.328121 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 29 22:32:56.341628 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.328124 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 29 22:32:56.341628 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.328126 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 29 22:32:56.341628 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.328130 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 29 22:32:56.341628 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.328134 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 29 22:32:56.342118 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.328137 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 29 22:32:56.342118 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.328139 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 29 22:32:56.342118 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.328142 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 29 22:32:56.342118 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.328144 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 29 22:32:56.342118 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.328147 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 29 22:32:56.342118 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.328149 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 29 22:32:56.342118 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.328152 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 29 22:32:56.342118 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.328154 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 29 22:32:56.342118 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.328157 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 29 22:32:56.342118 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.328159 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 29 22:32:56.342118 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.328162 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 29 22:32:56.342118 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.328164 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 29 22:32:56.342118 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.328167 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 29 22:32:56.342118 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.328169 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 29 22:32:56.342118 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.328173 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 29 22:32:56.342118 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.328176 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 29 22:32:56.342118 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.328179 2577 feature_gate.go:386] unrecognized feature gate: Example2 Oct 29 22:32:56.342118 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.328183 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 29 22:32:56.342118 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.328186 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 29 22:32:56.342118 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.328188 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 29 22:32:56.342561 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.328191 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 29 22:32:56.342561 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.328195 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 29 22:32:56.342561 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.328198 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 29 22:32:56.342561 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.328200 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 29 22:32:56.342561 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.328203 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 29 22:32:56.342561 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.328205 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 29 22:32:56.342561 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.328208 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 29 22:32:56.342561 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:56.328211 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 29 22:32:56.342561 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.328259 2577 server.go:957] "Client rotation is on, will bootstrap in background" Oct 29 22:32:56.342561 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.330895 2577 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Oct 29 22:32:56.342561 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.331701 2577 server.go:1014] "Starting client certificate rotation" Oct 29 22:32:56.342561 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.331813 2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 29 22:32:56.342561 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.332546 2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 29 22:32:56.342561 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.333697 2577 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 29 22:32:56.342561 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.334756 2577 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 29 22:32:56.344313 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.344299 2577 log.go:25] "Validated CRI v1 runtime API" Oct 29 22:32:56.347395 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.347377 2577 log.go:25] "Validated CRI v1 image API" Oct 29 22:32:56.348168 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.348146 2577 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 29 22:32:56.350743 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.350723 2577 fs.go:135] Filesystem UUIDs: map[033799f6-a5e8-4944-83ed-8553f8110b52:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 dead43f6-6552-4c20-83b8-c085fcfbbaa6:/dev/nvme0n1p3] Oct 29 22:32:56.350817 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.350742 2577 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Oct 29 22:32:56.356569 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.356453 2577 manager.go:217] Machine: {Timestamp:2026-10-29 22:32:56.355101739 +0000 UTC m=+0.319978253 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099949 MemoryCapacity:33164337152 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2f41544626a51e83d19832b219b9de SystemUUID:ec2f4154-4626-a51e-83d1-9832b219b9de BootID:0970734c-b3cc-455e-b107-15f90ef1f7ba Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:c5:a5:39:73 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:c5:a5:39:73 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:c6:bb:2c:00:80:2b Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164337152 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 29 22:32:56.356569 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.356562 2577 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 29 22:32:56.356740 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.356683 2577 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 29 22:32:56.359441 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.359418 2577 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 29 22:32:56.359565 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.359443 2577 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-140-74.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 29 22:32:56.359619 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.359567 2577 topology_manager.go:143] "Creating topology manager with none policy" Oct 29 22:32:56.359619 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.359575 2577 container_manager_linux.go:311] "Creating device plugin manager" Oct 29 22:32:56.359619 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.359608 2577 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 29 22:32:56.359710 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.359622 2577 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 29 22:32:56.360414 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.360404 2577 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 29 22:32:56.363863 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.363851 2577 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 29 22:32:56.363992 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.363982 2577 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 29 22:32:56.364679 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.364669 2577 kubelet.go:493] "Attempting to sync node with API server" Oct 29 22:32:56.364709 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.364683 2577 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 29 22:32:56.364709 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.364693 2577 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 29 22:32:56.364709 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.364701 2577 kubelet.go:405] "Adding apiserver pod source" Oct 29 22:32:56.364709 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.364709 2577 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 29 22:32:56.365068 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.365055 2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 29 22:32:56.367164 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.367148 2577 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 29 22:32:56.371895 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.371880 2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 29 22:32:56.372814 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.372799 2577 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 29 22:32:56.372881 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.372839 2577 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 29 22:32:56.373029 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.373016 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 29 22:32:56.373085 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.373035 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 29 22:32:56.373085 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.373044 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 29 22:32:56.373085 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.373053 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 29 22:32:56.373085 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.373061 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 29 22:32:56.373201 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.373096 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 29 22:32:56.373677 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.373666 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 29 22:32:56.373725 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.373680 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 29 22:32:56.373725 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.373691 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 29 22:32:56.373725 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.373701 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 29 22:32:56.373725 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.373724 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 29 22:32:56.373873 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.373738 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 29 22:32:56.374325 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.374315 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 29 22:32:56.374371 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.374328 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 29 22:32:56.377994 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.377979 2577 server.go:1263] "Started kubelet" Oct 29 22:32:56.378082 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.378036 2577 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 29 22:32:56.378173 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.378128 2577 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 29 22:32:56.378216 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.378194 2577 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 29 22:32:56.378759 ip-10-0-140-74 systemd[1]: Started Kubernetes Kubelet. Oct 29 22:32:56.384492 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.384469 2577 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 29 22:32:56.386044 ip-10-0-140-74 kubenswrapper[2577]: E1029 22:32:56.386021 2577 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Oct 29 22:32:56.386596 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.386577 2577 server.go:317] "Adding debug handlers to kubelet server" Oct 29 22:32:56.389329 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.389310 2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 29 22:32:56.389786 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.389770 2577 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 29 22:32:56.389875 ip-10-0-140-74 kubenswrapper[2577]: E1029 22:32:56.389852 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-74.ec2.internal\" not found" Oct 29 22:32:56.390319 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.390298 2577 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 29 22:32:56.390319 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.390299 2577 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 29 22:32:56.390463 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.390323 2577 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 29 22:32:56.390560 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.390547 2577 reconstruct.go:97] "Volume reconstruction finished" Oct 29 22:32:56.390560 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.390558 2577 reconciler.go:29] "Reconciler: start to sync state" Oct 29 22:32:56.390669 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.390566 2577 factory.go:55] Registering systemd factory Oct 29 22:32:56.390669 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.390579 2577 factory.go:223] Registration of the systemd container factory successfully Oct 29 22:32:56.390782 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.390770 2577 factory.go:153] Registering CRI-O factory Oct 29 22:32:56.390839 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.390785 2577 factory.go:223] Registration of the crio container factory successfully Oct 29 22:32:56.390887 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.390847 2577 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 29 22:32:56.390887 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.390865 2577 factory.go:103] Registering Raw factory Oct 29 22:32:56.390887 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.390879 2577 manager.go:1199] Started watching for new ooms in manager Oct 29 22:32:56.392114 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.392099 2577 manager.go:319] Starting recovery of all containers Oct 29 22:32:56.402174 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.402154 2577 manager.go:324] Recovery completed Oct 29 22:32:56.406156 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.406143 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 29 22:32:56.408301 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.408283 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-74.ec2.internal" event="NodeHasSufficientMemory" Oct 29 22:32:56.408362 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.408331 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-74.ec2.internal" event="NodeHasNoDiskPressure" Oct 29 22:32:56.408362 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.408345 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-74.ec2.internal" event="NodeHasSufficientPID" Oct 29 22:32:56.408913 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.408899 2577 cpu_manager.go:226] "Starting" policy="none" Oct 29 22:32:56.408979 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.408915 2577 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 29 22:32:56.408979 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.408937 2577 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 29 22:32:56.408979 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.408904 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 29 22:32:56.409604 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.409588 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-74.ec2.internal" event="NodeHasSufficientMemory" Oct 29 22:32:56.409702 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.409617 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-74.ec2.internal" event="NodeHasNoDiskPressure" Oct 29 22:32:56.409702 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.409630 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-74.ec2.internal" event="NodeHasSufficientPID" Oct 29 22:32:56.414424 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.414404 2577 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 29 22:32:56.416366 ip-10-0-140-74 kubenswrapper[2577]: E1029 22:32:56.416340 2577 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-140-74.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Oct 29 22:32:56.416468 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.416431 2577 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-140-74.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Oct 29 22:32:56.418921 ip-10-0-140-74 kubenswrapper[2577]: E1029 22:32:56.416336 2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-74.ec2.internal.18e320d52108d3ff default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-74.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-140-74.ec2.internal,},FirstTimestamp:2026-10-29 22:32:56.377938943 +0000 UTC m=+0.342815456,LastTimestamp:2026-10-29 22:32:56.377938943 +0000 UTC m=+0.342815456,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-74.ec2.internal,}" Oct 29 22:32:56.435713 ip-10-0-140-74 kubenswrapper[2577]: E1029 22:32:56.435606 2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-74.ec2.internal.18e320d522d81fd4 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-74.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-140-74.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-140-74.ec2.internal,},FirstTimestamp:2026-10-29 22:32:56.408301524 +0000 UTC m=+0.373178038,LastTimestamp:2026-10-29 22:32:56.408301524 +0000 UTC m=+0.373178038,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-74.ec2.internal,}" Oct 29 22:32:56.435801 ip-10-0-140-74 kubenswrapper[2577]: E1029 22:32:56.435752 2577 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Oct 29 22:32:56.435801 ip-10-0-140-74 kubenswrapper[2577]: E1029 22:32:56.435689 2577 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Oct 29 22:32:56.435801 ip-10-0-140-74 kubenswrapper[2577]: E1029 22:32:56.435689 2577 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-140-74.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Oct 29 22:32:56.487347 ip-10-0-140-74 kubenswrapper[2577]: E1029 22:32:56.468803 2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-74.ec2.internal.18e320d522d8acca default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-74.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-140-74.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-140-74.ec2.internal,},FirstTimestamp:2026-10-29 22:32:56.40833761 +0000 UTC m=+0.373214123,LastTimestamp:2026-10-29 22:32:56.40833761 +0000 UTC m=+0.373214123,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-74.ec2.internal,}" Oct 29 22:32:56.487347 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.469981 2577 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-fhqx8" Oct 29 22:32:56.487347 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.482212 2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-fhqx8" Oct 29 22:32:56.487347 ip-10-0-140-74 kubenswrapper[2577]: E1029 22:32:56.487257 2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-74.ec2.internal.18e320d522d8d88f default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-74.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-140-74.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-140-74.ec2.internal,},FirstTimestamp:2026-10-29 22:32:56.408348815 +0000 UTC m=+0.373225327,LastTimestamp:2026-10-29 22:32:56.408348815 +0000 UTC m=+0.373225327,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-74.ec2.internal,}" Oct 29 22:32:56.548395 ip-10-0-140-74 kubenswrapper[2577]: E1029 22:32:56.490754 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-74.ec2.internal\" not found" Oct 29 22:32:56.548395 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.515192 2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 29 22:32:56.548395 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.516325 2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 29 22:32:56.548395 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.516342 2577 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 29 22:32:56.548395 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.516358 2577 kubelet.go:2521] "Starting kubelet main sync loop" Oct 29 22:32:56.548395 ip-10-0-140-74 kubenswrapper[2577]: E1029 22:32:56.516388 2577 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Oct 29 22:32:56.548395 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.532222 2577 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 29 22:32:56.549011 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.548998 2577 policy_none.go:50] "Start" Oct 29 22:32:56.549078 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.549019 2577 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 29 22:32:56.549078 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.549033 2577 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 29 22:32:56.551073 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.551060 2577 policy_none.go:44] "Start" Oct 29 22:32:56.583931 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.583807 2577 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 29 22:32:56.591281 ip-10-0-140-74 kubenswrapper[2577]: E1029 22:32:56.591265 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-74.ec2.internal\" not found" Oct 29 22:32:56.606610 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.606593 2577 manager.go:341] "Starting Device Plugin manager" Oct 29 22:32:56.606691 ip-10-0-140-74 kubenswrapper[2577]: E1029 22:32:56.606616 2577 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 29 22:32:56.606691 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.606625 2577 server.go:87] "Starting device plugin registration server" Oct 29 22:32:56.606869 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.606857 2577 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 29 22:32:56.606906 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.606871 2577 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 29 22:32:56.606985 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.606970 2577 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 29 22:32:56.607091 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.607076 2577 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 29 22:32:56.607187 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.607109 2577 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 29 22:32:56.607532 ip-10-0-140-74 kubenswrapper[2577]: E1029 22:32:56.607515 2577 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 29 22:32:56.607604 ip-10-0-140-74 kubenswrapper[2577]: E1029 22:32:56.607552 2577 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-140-74.ec2.internal\" not found" Oct 29 22:32:56.617255 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.617238 2577 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-74.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-140-74.ec2.internal"] Oct 29 22:32:56.617317 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.617302 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 29 22:32:56.618111 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.618097 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-74.ec2.internal" event="NodeHasSufficientMemory" Oct 29 22:32:56.618175 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.618124 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-74.ec2.internal" event="NodeHasNoDiskPressure" Oct 29 22:32:56.618175 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.618136 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-74.ec2.internal" event="NodeHasSufficientPID" Oct 29 22:32:56.620236 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.620223 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 29 22:32:56.620382 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.620370 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-74.ec2.internal" Oct 29 22:32:56.620417 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.620396 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 29 22:32:56.620953 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.620937 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-74.ec2.internal" event="NodeHasSufficientMemory" Oct 29 22:32:56.620953 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.620951 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-74.ec2.internal" event="NodeHasSufficientMemory" Oct 29 22:32:56.621068 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.620965 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-74.ec2.internal" event="NodeHasNoDiskPressure" Oct 29 22:32:56.621068 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.620970 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-74.ec2.internal" event="NodeHasNoDiskPressure" Oct 29 22:32:56.621068 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.620976 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-74.ec2.internal" event="NodeHasSufficientPID" Oct 29 22:32:56.621068 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.620981 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-74.ec2.internal" event="NodeHasSufficientPID" Oct 29 22:32:56.622952 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.622938 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-74.ec2.internal" Oct 29 22:32:56.623047 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.622964 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 29 22:32:56.623697 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.623681 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-74.ec2.internal" event="NodeHasSufficientMemory" Oct 29 22:32:56.623757 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.623709 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-74.ec2.internal" event="NodeHasNoDiskPressure" Oct 29 22:32:56.623757 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.623718 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-74.ec2.internal" event="NodeHasSufficientPID" Oct 29 22:32:56.649080 ip-10-0-140-74 kubenswrapper[2577]: E1029 22:32:56.649060 2577 nodelease.go:50] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-140-74.ec2.internal\" not found" node="ip-10-0-140-74.ec2.internal" Oct 29 22:32:56.653051 ip-10-0-140-74 kubenswrapper[2577]: E1029 22:32:56.653033 2577 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-74.ec2.internal\" not found" node="ip-10-0-140-74.ec2.internal" Oct 29 22:32:56.655694 ip-10-0-140-74 kubenswrapper[2577]: E1029 22:32:56.655677 2577 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-74.ec2.internal\" not found" node="ip-10-0-140-74.ec2.internal" Oct 29 22:32:56.691394 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.691378 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4c81f7639feb5d07aad0278bf28e49a4-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-74.ec2.internal\" (UID: \"4c81f7639feb5d07aad0278bf28e49a4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-74.ec2.internal" Oct 29 22:32:56.691479 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.691401 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4c81f7639feb5d07aad0278bf28e49a4-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-74.ec2.internal\" (UID: \"4c81f7639feb5d07aad0278bf28e49a4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-74.ec2.internal" Oct 29 22:32:56.691479 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.691416 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ef9ac0a87cdfc5ee9b8eaa2797bc55b1-config\") pod \"kube-apiserver-proxy-ip-10-0-140-74.ec2.internal\" (UID: \"ef9ac0a87cdfc5ee9b8eaa2797bc55b1\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-74.ec2.internal" Oct 29 22:32:56.707698 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.707682 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 29 22:32:56.708370 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.708355 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-74.ec2.internal" event="NodeHasSufficientMemory" Oct 29 22:32:56.708437 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.708379 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-74.ec2.internal" event="NodeHasNoDiskPressure" Oct 29 22:32:56.708437 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.708389 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-74.ec2.internal" event="NodeHasSufficientPID" Oct 29 22:32:56.708437 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.708409 2577 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-140-74.ec2.internal" Oct 29 22:32:56.730907 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.730891 2577 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-140-74.ec2.internal" Oct 29 22:32:56.730955 ip-10-0-140-74 kubenswrapper[2577]: E1029 22:32:56.730909 2577 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-140-74.ec2.internal\": node \"ip-10-0-140-74.ec2.internal\" not found" Oct 29 22:32:56.774997 ip-10-0-140-74 kubenswrapper[2577]: E1029 22:32:56.774980 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-74.ec2.internal\" not found" Oct 29 22:32:56.791635 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.791617 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4c81f7639feb5d07aad0278bf28e49a4-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-74.ec2.internal\" (UID: \"4c81f7639feb5d07aad0278bf28e49a4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-74.ec2.internal" Oct 29 22:32:56.791725 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.791708 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4c81f7639feb5d07aad0278bf28e49a4-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-74.ec2.internal\" (UID: \"4c81f7639feb5d07aad0278bf28e49a4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-74.ec2.internal" Oct 29 22:32:56.791764 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.791742 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ef9ac0a87cdfc5ee9b8eaa2797bc55b1-config\") pod \"kube-apiserver-proxy-ip-10-0-140-74.ec2.internal\" (UID: \"ef9ac0a87cdfc5ee9b8eaa2797bc55b1\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-74.ec2.internal" Oct 29 22:32:56.791764 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.791714 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4c81f7639feb5d07aad0278bf28e49a4-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-74.ec2.internal\" (UID: \"4c81f7639feb5d07aad0278bf28e49a4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-74.ec2.internal" Oct 29 22:32:56.791828 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.791705 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4c81f7639feb5d07aad0278bf28e49a4-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-74.ec2.internal\" (UID: \"4c81f7639feb5d07aad0278bf28e49a4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-74.ec2.internal" Oct 29 22:32:56.791828 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.791801 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ef9ac0a87cdfc5ee9b8eaa2797bc55b1-config\") pod \"kube-apiserver-proxy-ip-10-0-140-74.ec2.internal\" (UID: \"ef9ac0a87cdfc5ee9b8eaa2797bc55b1\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-74.ec2.internal" Oct 29 22:32:56.875417 ip-10-0-140-74 kubenswrapper[2577]: E1029 22:32:56.875376 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-74.ec2.internal\" not found" Oct 29 22:32:56.956732 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.956715 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-74.ec2.internal" Oct 29 22:32:56.959044 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:56.959032 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-74.ec2.internal" Oct 29 22:32:56.976327 ip-10-0-140-74 kubenswrapper[2577]: E1029 22:32:56.976307 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-74.ec2.internal\" not found" Oct 29 22:32:57.076798 ip-10-0-140-74 kubenswrapper[2577]: E1029 22:32:57.076771 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-74.ec2.internal\" not found" Oct 29 22:32:57.177282 ip-10-0-140-74 kubenswrapper[2577]: E1029 22:32:57.177228 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-74.ec2.internal\" not found" Oct 29 22:32:57.277718 ip-10-0-140-74 kubenswrapper[2577]: E1029 22:32:57.277692 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-74.ec2.internal\" not found" Oct 29 22:32:59.139849 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:59.139827 2577 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Oct 29 22:32:59.140251 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:59.139992 2577 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 29 22:32:59.185936 ip-10-0-140-74 kubenswrapper[2577]: E1029 22:32:59.185911 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-74.ec2.internal\" not found" Oct 29 22:32:59.222314 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:59.222289 2577 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 29 22:32:59.275093 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:59.275035 2577 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 29 22:32:59.294526 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:59.294499 2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-28 22:27:58 +0000 UTC" deadline="2028-05-17 11:37:12.531719926 +0000 UTC" Oct 29 22:32:59.294526 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:59.294522 2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13573h4m13.237200618s" Oct 29 22:32:59.298741 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:59.298721 2577 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-74.ec2.internal" Oct 29 22:32:59.321715 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:59.321700 2577 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 29 22:32:59.323388 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:59.323375 2577 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-74.ec2.internal" Oct 29 22:32:59.328188 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:59.328157 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef9ac0a87cdfc5ee9b8eaa2797bc55b1.slice/crio-2c31fee02392017ba8b41b846564c28cb85ccd960603212936086d59f108fd8a WatchSource:0}: Error finding container 2c31fee02392017ba8b41b846564c28cb85ccd960603212936086d59f108fd8a: Status 404 returned error can't find the container with id 2c31fee02392017ba8b41b846564c28cb85ccd960603212936086d59f108fd8a Oct 29 22:32:59.328981 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:32:59.328963 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c81f7639feb5d07aad0278bf28e49a4.slice/crio-39746dc1a72c981c5e9698f1f0c72d7c28b892b2793847e6fcf516fb3f1c727f WatchSource:0}: Error finding container 39746dc1a72c981c5e9698f1f0c72d7c28b892b2793847e6fcf516fb3f1c727f: Status 404 returned error can't find the container with id 39746dc1a72c981c5e9698f1f0c72d7c28b892b2793847e6fcf516fb3f1c727f Oct 29 22:32:59.335580 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:59.335565 2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 29 22:32:59.343000 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:59.342985 2577 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 29 22:32:59.808705 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:32:59.808675 2577 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 29 22:33:00.173747 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.173666 2577 apiserver.go:52] "Watching apiserver" Oct 29 22:33:00.197791 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.197764 2577 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 29 22:33:00.198108 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.198088 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/node-ca-pcjl7","openshift-multus/multus-additional-cni-plugins-llm78","openshift-multus/network-metrics-daemon-hzlmg","kube-system/global-pull-secret-syncer-bp46g","kube-system/konnectivity-agent-54f9x","openshift-network-operator/iptables-alerter-wpzm2","openshift-ovn-kubernetes/ovnkube-node-khlhx","openshift-network-diagnostics/network-check-target-7jv5g","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c96pg","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-74.ec2.internal","openshift-multus/multus-v4q29","kube-system/kube-apiserver-proxy-ip-10-0-140-74.ec2.internal","openshift-cluster-node-tuning-operator/tuned-jtlmz","openshift-dns/node-resolver-dpsd9"] Oct 29 22:33:00.198407 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.198387 2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Oct 29 22:33:00.200824 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.200796 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-khlhx" Oct 29 22:33:00.204096 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.204076 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-jtlmz" Oct 29 22:33:00.206623 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.206604 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hzlmg" Oct 29 22:33:00.206723 ip-10-0-140-74 kubenswrapper[2577]: E1029 22:33:00.206677 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hzlmg" podUID="f8f92bfb-5cef-412e-a6c1-4bd793c59fe7" Oct 29 22:33:00.207716 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.207696 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/04195c1d-9fb7-479b-8061-f92e3c88aed2-host-run-netns\") pod \"ovnkube-node-khlhx\" (UID: \"04195c1d-9fb7-479b-8061-f92e3c88aed2\") " pod="openshift-ovn-kubernetes/ovnkube-node-khlhx" Oct 29 22:33:00.207716 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.207722 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/04195c1d-9fb7-479b-8061-f92e3c88aed2-run-systemd\") pod \"ovnkube-node-khlhx\" (UID: \"04195c1d-9fb7-479b-8061-f92e3c88aed2\") " pod="openshift-ovn-kubernetes/ovnkube-node-khlhx" Oct 29 22:33:00.207853 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.207745 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/04195c1d-9fb7-479b-8061-f92e3c88aed2-run-ovn\") pod \"ovnkube-node-khlhx\" (UID: \"04195c1d-9fb7-479b-8061-f92e3c88aed2\") " pod="openshift-ovn-kubernetes/ovnkube-node-khlhx" Oct 29 22:33:00.207853 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.207766 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/04195c1d-9fb7-479b-8061-f92e3c88aed2-ovnkube-config\") pod \"ovnkube-node-khlhx\" (UID: \"04195c1d-9fb7-479b-8061-f92e3c88aed2\") " pod="openshift-ovn-kubernetes/ovnkube-node-khlhx" Oct 29 22:33:00.207853 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.207784 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/04195c1d-9fb7-479b-8061-f92e3c88aed2-host-kubelet\") pod \"ovnkube-node-khlhx\" (UID: \"04195c1d-9fb7-479b-8061-f92e3c88aed2\") " pod="openshift-ovn-kubernetes/ovnkube-node-khlhx" Oct 29 22:33:00.207853 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.207801 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/04195c1d-9fb7-479b-8061-f92e3c88aed2-systemd-units\") pod \"ovnkube-node-khlhx\" (UID: \"04195c1d-9fb7-479b-8061-f92e3c88aed2\") " pod="openshift-ovn-kubernetes/ovnkube-node-khlhx" Oct 29 22:33:00.207853 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.207822 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/04195c1d-9fb7-479b-8061-f92e3c88aed2-var-lib-openvswitch\") pod \"ovnkube-node-khlhx\" (UID: \"04195c1d-9fb7-479b-8061-f92e3c88aed2\") " pod="openshift-ovn-kubernetes/ovnkube-node-khlhx" Oct 29 22:33:00.207853 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.207844 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/04195c1d-9fb7-479b-8061-f92e3c88aed2-etc-openvswitch\") pod \"ovnkube-node-khlhx\" (UID: \"04195c1d-9fb7-479b-8061-f92e3c88aed2\") " pod="openshift-ovn-kubernetes/ovnkube-node-khlhx" Oct 29 22:33:00.208083 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.207866 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/04195c1d-9fb7-479b-8061-f92e3c88aed2-run-openvswitch\") pod \"ovnkube-node-khlhx\" (UID: \"04195c1d-9fb7-479b-8061-f92e3c88aed2\") " pod="openshift-ovn-kubernetes/ovnkube-node-khlhx" Oct 29 22:33:00.208083 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.207882 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/04195c1d-9fb7-479b-8061-f92e3c88aed2-host-cni-bin\") pod \"ovnkube-node-khlhx\" (UID: \"04195c1d-9fb7-479b-8061-f92e3c88aed2\") " pod="openshift-ovn-kubernetes/ovnkube-node-khlhx" Oct 29 22:33:00.208083 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.207898 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/04195c1d-9fb7-479b-8061-f92e3c88aed2-ovn-node-metrics-cert\") pod \"ovnkube-node-khlhx\" (UID: \"04195c1d-9fb7-479b-8061-f92e3c88aed2\") " pod="openshift-ovn-kubernetes/ovnkube-node-khlhx" Oct 29 22:33:00.208083 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.207917 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tlxfr\" (UniqueName: \"kubernetes.io/projected/04195c1d-9fb7-479b-8061-f92e3c88aed2-kube-api-access-tlxfr\") pod \"ovnkube-node-khlhx\" (UID: \"04195c1d-9fb7-479b-8061-f92e3c88aed2\") " pod="openshift-ovn-kubernetes/ovnkube-node-khlhx" Oct 29 22:33:00.208083 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.207935 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/04195c1d-9fb7-479b-8061-f92e3c88aed2-ovnkube-script-lib\") pod \"ovnkube-node-khlhx\" (UID: \"04195c1d-9fb7-479b-8061-f92e3c88aed2\") " pod="openshift-ovn-kubernetes/ovnkube-node-khlhx" Oct 29 22:33:00.208083 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.207952 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/04195c1d-9fb7-479b-8061-f92e3c88aed2-node-log\") pod \"ovnkube-node-khlhx\" (UID: \"04195c1d-9fb7-479b-8061-f92e3c88aed2\") " pod="openshift-ovn-kubernetes/ovnkube-node-khlhx" Oct 29 22:33:00.208083 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.207965 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/04195c1d-9fb7-479b-8061-f92e3c88aed2-log-socket\") pod \"ovnkube-node-khlhx\" (UID: \"04195c1d-9fb7-479b-8061-f92e3c88aed2\") " pod="openshift-ovn-kubernetes/ovnkube-node-khlhx" Oct 29 22:33:00.208083 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.207979 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/04195c1d-9fb7-479b-8061-f92e3c88aed2-host-run-ovn-kubernetes\") pod \"ovnkube-node-khlhx\" (UID: \"04195c1d-9fb7-479b-8061-f92e3c88aed2\") " pod="openshift-ovn-kubernetes/ovnkube-node-khlhx" Oct 29 22:33:00.208083 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.207995 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/04195c1d-9fb7-479b-8061-f92e3c88aed2-host-cni-netd\") pod \"ovnkube-node-khlhx\" (UID: \"04195c1d-9fb7-479b-8061-f92e3c88aed2\") " pod="openshift-ovn-kubernetes/ovnkube-node-khlhx" Oct 29 22:33:00.208083 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.208013 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/04195c1d-9fb7-479b-8061-f92e3c88aed2-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-khlhx\" (UID: \"04195c1d-9fb7-479b-8061-f92e3c88aed2\") " pod="openshift-ovn-kubernetes/ovnkube-node-khlhx" Oct 29 22:33:00.208083 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.208041 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/04195c1d-9fb7-479b-8061-f92e3c88aed2-env-overrides\") pod \"ovnkube-node-khlhx\" (UID: \"04195c1d-9fb7-479b-8061-f92e3c88aed2\") " pod="openshift-ovn-kubernetes/ovnkube-node-khlhx" Oct 29 22:33:00.208083 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.208055 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/04195c1d-9fb7-479b-8061-f92e3c88aed2-host-slash\") pod \"ovnkube-node-khlhx\" (UID: \"04195c1d-9fb7-479b-8061-f92e3c88aed2\") " pod="openshift-ovn-kubernetes/ovnkube-node-khlhx" Oct 29 22:33:00.208761 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.208744 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bp46g" Oct 29 22:33:00.208838 ip-10-0-140-74 kubenswrapper[2577]: E1029 22:33:00.208807 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bp46g" podUID="6f50ee57-a710-4325-bd45-4f861452ede4" Oct 29 22:33:00.211209 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.211193 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-54f9x" Oct 29 22:33:00.214141 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.213363 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-wpzm2" Oct 29 22:33:00.215654 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.215631 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 29 22:33:00.215788 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.215774 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 29 22:33:00.218680 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.218663 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c96pg" Oct 29 22:33:00.219050 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.219028 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-pcjl7" Oct 29 22:33:00.219175 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.219157 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 29 22:33:00.219244 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.219180 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 29 22:33:00.219244 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.219181 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 29 22:33:00.219244 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.219234 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 29 22:33:00.219426 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.219265 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 29 22:33:00.221006 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.220987 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-llm78" Oct 29 22:33:00.223267 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.223248 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-dpsd9" Oct 29 22:33:00.225577 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.225557 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-v4q29" Oct 29 22:33:00.228223 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.228200 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7jv5g" Oct 29 22:33:00.228642 ip-10-0-140-74 kubenswrapper[2577]: E1029 22:33:00.228605 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7jv5g" podUID="a4e1c52c-1eeb-4617-a0c9-d052521360d3" Oct 29 22:33:00.250137 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.250118 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 29 22:33:00.250317 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.250300 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 29 22:33:00.250408 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.250354 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-vd45d\"" Oct 29 22:33:00.250408 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.250381 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-g9cd8\"" Oct 29 22:33:00.250509 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.250410 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-8l2jp\"" Oct 29 22:33:00.250509 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.250496 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-f9xcm\"" Oct 29 22:33:00.250642 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.250533 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-rfj6j\"" Oct 29 22:33:00.250642 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.250607 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-mhfn7\"" Oct 29 22:33:00.250804 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.250225 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-kr4qv\"" Oct 29 22:33:00.250804 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.250786 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-p6rt5\"" Oct 29 22:33:00.250928 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.250839 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-wccb2\"" Oct 29 22:33:00.250997 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.250218 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 29 22:33:00.253912 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.253888 2577 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 29 22:33:00.254006 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.253948 2577 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-gpqvs" Oct 29 22:33:00.265319 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.265302 2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-gpqvs" Oct 29 22:33:00.299480 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.299459 2577 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 29 22:33:00.308837 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.308810 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b23027ca-0748-4125-9bc8-9d149a3418e0-kubelet-dir\") pod \"aws-ebs-csi-driver-node-c96pg\" (UID: \"b23027ca-0748-4125-9bc8-9d149a3418e0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c96pg" Oct 29 22:33:00.308937 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.308850 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/04195c1d-9fb7-479b-8061-f92e3c88aed2-node-log\") pod \"ovnkube-node-khlhx\" (UID: \"04195c1d-9fb7-479b-8061-f92e3c88aed2\") " pod="openshift-ovn-kubernetes/ovnkube-node-khlhx" Oct 29 22:33:00.308937 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.308901 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/04195c1d-9fb7-479b-8061-f92e3c88aed2-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-khlhx\" (UID: \"04195c1d-9fb7-479b-8061-f92e3c88aed2\") " pod="openshift-ovn-kubernetes/ovnkube-node-khlhx" Oct 29 22:33:00.308937 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.308926 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/04195c1d-9fb7-479b-8061-f92e3c88aed2-node-log\") pod \"ovnkube-node-khlhx\" (UID: \"04195c1d-9fb7-479b-8061-f92e3c88aed2\") " pod="openshift-ovn-kubernetes/ovnkube-node-khlhx" Oct 29 22:33:00.309074 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.308933 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/04ff31d0-a870-4527-a0c3-285737359f95-iptables-alerter-script\") pod \"iptables-alerter-wpzm2\" (UID: \"04ff31d0-a870-4527-a0c3-285737359f95\") " pod="openshift-network-operator/iptables-alerter-wpzm2" Oct 29 22:33:00.309074 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.308978 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/04195c1d-9fb7-479b-8061-f92e3c88aed2-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-khlhx\" (UID: \"04195c1d-9fb7-479b-8061-f92e3c88aed2\") " pod="openshift-ovn-kubernetes/ovnkube-node-khlhx" Oct 29 22:33:00.309074 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.308993 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b23027ca-0748-4125-9bc8-9d149a3418e0-device-dir\") pod \"aws-ebs-csi-driver-node-c96pg\" (UID: \"b23027ca-0748-4125-9bc8-9d149a3418e0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c96pg" Oct 29 22:33:00.309074 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.309031 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/93dbae94-f92d-4157-a89b-d3c984a56013-serviceca\") pod \"node-ca-pcjl7\" (UID: \"93dbae94-f92d-4157-a89b-d3c984a56013\") " pod="openshift-image-registry/node-ca-pcjl7" Oct 29 22:33:00.309247 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.309087 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wf78m\" (UniqueName: \"kubernetes.io/projected/93dbae94-f92d-4157-a89b-d3c984a56013-kube-api-access-wf78m\") pod \"node-ca-pcjl7\" (UID: \"93dbae94-f92d-4157-a89b-d3c984a56013\") " pod="openshift-image-registry/node-ca-pcjl7" Oct 29 22:33:00.309247 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.309132 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/04195c1d-9fb7-479b-8061-f92e3c88aed2-host-slash\") pod \"ovnkube-node-khlhx\" (UID: \"04195c1d-9fb7-479b-8061-f92e3c88aed2\") " pod="openshift-ovn-kubernetes/ovnkube-node-khlhx" Oct 29 22:33:00.309361 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.309238 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/04195c1d-9fb7-479b-8061-f92e3c88aed2-host-run-netns\") pod \"ovnkube-node-khlhx\" (UID: \"04195c1d-9fb7-479b-8061-f92e3c88aed2\") " pod="openshift-ovn-kubernetes/ovnkube-node-khlhx" Oct 29 22:33:00.309361 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.309295 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/04195c1d-9fb7-479b-8061-f92e3c88aed2-host-slash\") pod \"ovnkube-node-khlhx\" (UID: \"04195c1d-9fb7-479b-8061-f92e3c88aed2\") " pod="openshift-ovn-kubernetes/ovnkube-node-khlhx" Oct 29 22:33:00.309361 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.309311 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/04195c1d-9fb7-479b-8061-f92e3c88aed2-ovnkube-config\") pod \"ovnkube-node-khlhx\" (UID: \"04195c1d-9fb7-479b-8061-f92e3c88aed2\") " pod="openshift-ovn-kubernetes/ovnkube-node-khlhx" Oct 29 22:33:00.309496 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.309362 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/04195c1d-9fb7-479b-8061-f92e3c88aed2-host-run-netns\") pod \"ovnkube-node-khlhx\" (UID: \"04195c1d-9fb7-479b-8061-f92e3c88aed2\") " pod="openshift-ovn-kubernetes/ovnkube-node-khlhx" Oct 29 22:33:00.309496 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.309376 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b23027ca-0748-4125-9bc8-9d149a3418e0-socket-dir\") pod \"aws-ebs-csi-driver-node-c96pg\" (UID: \"b23027ca-0748-4125-9bc8-9d149a3418e0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c96pg" Oct 29 22:33:00.309496 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.309419 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f190a1a0-6f6a-44d0-aeaa-b053cff4bd9c-etc-modprobe-d\") pod \"tuned-jtlmz\" (UID: \"f190a1a0-6f6a-44d0-aeaa-b053cff4bd9c\") " pod="openshift-cluster-node-tuning-operator/tuned-jtlmz" Oct 29 22:33:00.309496 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.309443 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8a972cc8-dce5-4f7c-b5d5-cb24ccf73bad-cnibin\") pod \"multus-additional-cni-plugins-llm78\" (UID: \"8a972cc8-dce5-4f7c-b5d5-cb24ccf73bad\") " pod="openshift-multus/multus-additional-cni-plugins-llm78" Oct 29 22:33:00.309689 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.309497 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2ab2aa22-99af-4451-8d33-9e3543c8d9cd-cnibin\") pod \"multus-v4q29\" (UID: \"2ab2aa22-99af-4451-8d33-9e3543c8d9cd\") " pod="openshift-multus/multus-v4q29" Oct 29 22:33:00.309689 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.309536 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2ab2aa22-99af-4451-8d33-9e3543c8d9cd-host-var-lib-cni-multus\") pod \"multus-v4q29\" (UID: \"2ab2aa22-99af-4451-8d33-9e3543c8d9cd\") " pod="openshift-multus/multus-v4q29" Oct 29 22:33:00.309689 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.309564 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/04195c1d-9fb7-479b-8061-f92e3c88aed2-var-lib-openvswitch\") pod \"ovnkube-node-khlhx\" (UID: \"04195c1d-9fb7-479b-8061-f92e3c88aed2\") " pod="openshift-ovn-kubernetes/ovnkube-node-khlhx" Oct 29 22:33:00.309689 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.309611 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/04195c1d-9fb7-479b-8061-f92e3c88aed2-run-openvswitch\") pod \"ovnkube-node-khlhx\" (UID: \"04195c1d-9fb7-479b-8061-f92e3c88aed2\") " pod="openshift-ovn-kubernetes/ovnkube-node-khlhx" Oct 29 22:33:00.309689 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.309636 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tlxfr\" (UniqueName: \"kubernetes.io/projected/04195c1d-9fb7-479b-8061-f92e3c88aed2-kube-api-access-tlxfr\") pod \"ovnkube-node-khlhx\" (UID: \"04195c1d-9fb7-479b-8061-f92e3c88aed2\") " pod="openshift-ovn-kubernetes/ovnkube-node-khlhx" Oct 29 22:33:00.309689 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.309674 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8a972cc8-dce5-4f7c-b5d5-cb24ccf73bad-system-cni-dir\") pod \"multus-additional-cni-plugins-llm78\" (UID: \"8a972cc8-dce5-4f7c-b5d5-cb24ccf73bad\") " pod="openshift-multus/multus-additional-cni-plugins-llm78" Oct 29 22:33:00.309974 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.309717 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/8a972cc8-dce5-4f7c-b5d5-cb24ccf73bad-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-llm78\" (UID: \"8a972cc8-dce5-4f7c-b5d5-cb24ccf73bad\") " pod="openshift-multus/multus-additional-cni-plugins-llm78" Oct 29 22:33:00.309974 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.309741 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f190a1a0-6f6a-44d0-aeaa-b053cff4bd9c-lib-modules\") pod \"tuned-jtlmz\" (UID: \"f190a1a0-6f6a-44d0-aeaa-b053cff4bd9c\") " pod="openshift-cluster-node-tuning-operator/tuned-jtlmz" Oct 29 22:33:00.309974 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.309774 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7fmsg\" (UniqueName: \"kubernetes.io/projected/0e1c2279-b6d4-44e0-84e6-94eb1206d2a0-kube-api-access-7fmsg\") pod \"node-resolver-dpsd9\" (UID: \"0e1c2279-b6d4-44e0-84e6-94eb1206d2a0\") " pod="openshift-dns/node-resolver-dpsd9" Oct 29 22:33:00.309974 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.309797 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2ab2aa22-99af-4451-8d33-9e3543c8d9cd-cni-binary-copy\") pod \"multus-v4q29\" (UID: \"2ab2aa22-99af-4451-8d33-9e3543c8d9cd\") " pod="openshift-multus/multus-v4q29" Oct 29 22:33:00.309974 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.309822 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4nczb\" (UniqueName: \"kubernetes.io/projected/f8f92bfb-5cef-412e-a6c1-4bd793c59fe7-kube-api-access-4nczb\") pod \"network-metrics-daemon-hzlmg\" (UID: \"f8f92bfb-5cef-412e-a6c1-4bd793c59fe7\") " pod="openshift-multus/network-metrics-daemon-hzlmg" Oct 29 22:33:00.309974 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.309851 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqxx5\" (UniqueName: \"kubernetes.io/projected/b23027ca-0748-4125-9bc8-9d149a3418e0-kube-api-access-jqxx5\") pod \"aws-ebs-csi-driver-node-c96pg\" (UID: \"b23027ca-0748-4125-9bc8-9d149a3418e0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c96pg" Oct 29 22:33:00.309974 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.309911 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/04195c1d-9fb7-479b-8061-f92e3c88aed2-log-socket\") pod \"ovnkube-node-khlhx\" (UID: \"04195c1d-9fb7-479b-8061-f92e3c88aed2\") " pod="openshift-ovn-kubernetes/ovnkube-node-khlhx" Oct 29 22:33:00.309974 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.309938 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/04195c1d-9fb7-479b-8061-f92e3c88aed2-host-run-ovn-kubernetes\") pod \"ovnkube-node-khlhx\" (UID: \"04195c1d-9fb7-479b-8061-f92e3c88aed2\") " pod="openshift-ovn-kubernetes/ovnkube-node-khlhx" Oct 29 22:33:00.309974 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.309962 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/04195c1d-9fb7-479b-8061-f92e3c88aed2-env-overrides\") pod \"ovnkube-node-khlhx\" (UID: \"04195c1d-9fb7-479b-8061-f92e3c88aed2\") " pod="openshift-ovn-kubernetes/ovnkube-node-khlhx" Oct 29 22:33:00.310418 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.309987 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6f50ee57-a710-4325-bd45-4f861452ede4-kubelet-config\") pod \"global-pull-secret-syncer-bp46g\" (UID: \"6f50ee57-a710-4325-bd45-4f861452ede4\") " pod="kube-system/global-pull-secret-syncer-bp46g" Oct 29 22:33:00.310418 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.310016 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2ab2aa22-99af-4451-8d33-9e3543c8d9cd-hostroot\") pod \"multus-v4q29\" (UID: \"2ab2aa22-99af-4451-8d33-9e3543c8d9cd\") " pod="openshift-multus/multus-v4q29" Oct 29 22:33:00.310418 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.310038 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2ab2aa22-99af-4451-8d33-9e3543c8d9cd-multus-conf-dir\") pod \"multus-v4q29\" (UID: \"2ab2aa22-99af-4451-8d33-9e3543c8d9cd\") " pod="openshift-multus/multus-v4q29" Oct 29 22:33:00.310418 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.310064 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6f50ee57-a710-4325-bd45-4f861452ede4-original-pull-secret\") pod \"global-pull-secret-syncer-bp46g\" (UID: \"6f50ee57-a710-4325-bd45-4f861452ede4\") " pod="kube-system/global-pull-secret-syncer-bp46g" Oct 29 22:33:00.310418 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.310088 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f190a1a0-6f6a-44d0-aeaa-b053cff4bd9c-etc-sysconfig\") pod \"tuned-jtlmz\" (UID: \"f190a1a0-6f6a-44d0-aeaa-b053cff4bd9c\") " pod="openshift-cluster-node-tuning-operator/tuned-jtlmz" Oct 29 22:33:00.310418 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.310115 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2ab2aa22-99af-4451-8d33-9e3543c8d9cd-multus-socket-dir-parent\") pod \"multus-v4q29\" (UID: \"2ab2aa22-99af-4451-8d33-9e3543c8d9cd\") " pod="openshift-multus/multus-v4q29" Oct 29 22:33:00.310418 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.310151 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/04195c1d-9fb7-479b-8061-f92e3c88aed2-systemd-units\") pod \"ovnkube-node-khlhx\" (UID: \"04195c1d-9fb7-479b-8061-f92e3c88aed2\") " pod="openshift-ovn-kubernetes/ovnkube-node-khlhx" Oct 29 22:33:00.310418 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.310174 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/04195c1d-9fb7-479b-8061-f92e3c88aed2-etc-openvswitch\") pod \"ovnkube-node-khlhx\" (UID: \"04195c1d-9fb7-479b-8061-f92e3c88aed2\") " pod="openshift-ovn-kubernetes/ovnkube-node-khlhx" Oct 29 22:33:00.310418 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.310231 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/04195c1d-9fb7-479b-8061-f92e3c88aed2-ovnkube-config\") pod \"ovnkube-node-khlhx\" (UID: \"04195c1d-9fb7-479b-8061-f92e3c88aed2\") " pod="openshift-ovn-kubernetes/ovnkube-node-khlhx" Oct 29 22:33:00.310418 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.310273 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/04195c1d-9fb7-479b-8061-f92e3c88aed2-host-cni-bin\") pod \"ovnkube-node-khlhx\" (UID: \"04195c1d-9fb7-479b-8061-f92e3c88aed2\") " pod="openshift-ovn-kubernetes/ovnkube-node-khlhx" Oct 29 22:33:00.310418 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.310291 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/04195c1d-9fb7-479b-8061-f92e3c88aed2-etc-openvswitch\") pod \"ovnkube-node-khlhx\" (UID: \"04195c1d-9fb7-479b-8061-f92e3c88aed2\") " pod="openshift-ovn-kubernetes/ovnkube-node-khlhx" Oct 29 22:33:00.310418 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.310309 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/04195c1d-9fb7-479b-8061-f92e3c88aed2-ovn-node-metrics-cert\") pod \"ovnkube-node-khlhx\" (UID: \"04195c1d-9fb7-479b-8061-f92e3c88aed2\") " pod="openshift-ovn-kubernetes/ovnkube-node-khlhx" Oct 29 22:33:00.310418 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.310325 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/04195c1d-9fb7-479b-8061-f92e3c88aed2-var-lib-openvswitch\") pod \"ovnkube-node-khlhx\" (UID: \"04195c1d-9fb7-479b-8061-f92e3c88aed2\") " pod="openshift-ovn-kubernetes/ovnkube-node-khlhx" Oct 29 22:33:00.310418 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.310354 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jxgn\" (UniqueName: \"kubernetes.io/projected/04ff31d0-a870-4527-a0c3-285737359f95-kube-api-access-8jxgn\") pod \"iptables-alerter-wpzm2\" (UID: \"04ff31d0-a870-4527-a0c3-285737359f95\") " pod="openshift-network-operator/iptables-alerter-wpzm2" Oct 29 22:33:00.310418 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.310362 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/04195c1d-9fb7-479b-8061-f92e3c88aed2-run-openvswitch\") pod \"ovnkube-node-khlhx\" (UID: \"04195c1d-9fb7-479b-8061-f92e3c88aed2\") " pod="openshift-ovn-kubernetes/ovnkube-node-khlhx" Oct 29 22:33:00.310418 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.310381 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvgj5\" (UniqueName: \"kubernetes.io/projected/8a972cc8-dce5-4f7c-b5d5-cb24ccf73bad-kube-api-access-jvgj5\") pod \"multus-additional-cni-plugins-llm78\" (UID: \"8a972cc8-dce5-4f7c-b5d5-cb24ccf73bad\") " pod="openshift-multus/multus-additional-cni-plugins-llm78" Oct 29 22:33:00.310418 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.310397 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/04195c1d-9fb7-479b-8061-f92e3c88aed2-log-socket\") pod \"ovnkube-node-khlhx\" (UID: \"04195c1d-9fb7-479b-8061-f92e3c88aed2\") " pod="openshift-ovn-kubernetes/ovnkube-node-khlhx" Oct 29 22:33:00.311190 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.310409 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2d2d7446-0741-4144-9b19-608854f8a04e-agent-certs\") pod \"konnectivity-agent-54f9x\" (UID: \"2d2d7446-0741-4144-9b19-608854f8a04e\") " pod="kube-system/konnectivity-agent-54f9x" Oct 29 22:33:00.311190 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.310418 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/04195c1d-9fb7-479b-8061-f92e3c88aed2-host-run-ovn-kubernetes\") pod \"ovnkube-node-khlhx\" (UID: \"04195c1d-9fb7-479b-8061-f92e3c88aed2\") " pod="openshift-ovn-kubernetes/ovnkube-node-khlhx" Oct 29 22:33:00.311190 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.310451 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6f50ee57-a710-4325-bd45-4f861452ede4-dbus\") pod \"global-pull-secret-syncer-bp46g\" (UID: \"6f50ee57-a710-4325-bd45-4f861452ede4\") " pod="kube-system/global-pull-secret-syncer-bp46g" Oct 29 22:33:00.311190 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.310477 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8a972cc8-dce5-4f7c-b5d5-cb24ccf73bad-tuning-conf-dir\") pod \"multus-additional-cni-plugins-llm78\" (UID: \"8a972cc8-dce5-4f7c-b5d5-cb24ccf73bad\") " pod="openshift-multus/multus-additional-cni-plugins-llm78" Oct 29 22:33:00.311190 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.310576 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2d2d7446-0741-4144-9b19-608854f8a04e-konnectivity-ca\") pod \"konnectivity-agent-54f9x\" (UID: \"2d2d7446-0741-4144-9b19-608854f8a04e\") " pod="kube-system/konnectivity-agent-54f9x" Oct 29 22:33:00.311190 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.310599 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2ab2aa22-99af-4451-8d33-9e3543c8d9cd-multus-cni-dir\") pod \"multus-v4q29\" (UID: \"2ab2aa22-99af-4451-8d33-9e3543c8d9cd\") " pod="openshift-multus/multus-v4q29" Oct 29 22:33:00.311190 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.310615 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2ab2aa22-99af-4451-8d33-9e3543c8d9cd-host-var-lib-kubelet\") pod \"multus-v4q29\" (UID: \"2ab2aa22-99af-4451-8d33-9e3543c8d9cd\") " pod="openshift-multus/multus-v4q29" Oct 29 22:33:00.311190 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.310634 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/04195c1d-9fb7-479b-8061-f92e3c88aed2-host-cni-netd\") pod \"ovnkube-node-khlhx\" (UID: \"04195c1d-9fb7-479b-8061-f92e3c88aed2\") " pod="openshift-ovn-kubernetes/ovnkube-node-khlhx" Oct 29 22:33:00.311190 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.310652 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/04ff31d0-a870-4527-a0c3-285737359f95-host-slash\") pod \"iptables-alerter-wpzm2\" (UID: \"04ff31d0-a870-4527-a0c3-285737359f95\") " pod="openshift-network-operator/iptables-alerter-wpzm2" Oct 29 22:33:00.311190 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.310667 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b23027ca-0748-4125-9bc8-9d149a3418e0-etc-selinux\") pod \"aws-ebs-csi-driver-node-c96pg\" (UID: \"b23027ca-0748-4125-9bc8-9d149a3418e0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c96pg" Oct 29 22:33:00.311190 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.310682 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f190a1a0-6f6a-44d0-aeaa-b053cff4bd9c-etc-sysctl-conf\") pod \"tuned-jtlmz\" (UID: \"f190a1a0-6f6a-44d0-aeaa-b053cff4bd9c\") " pod="openshift-cluster-node-tuning-operator/tuned-jtlmz" Oct 29 22:33:00.311190 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.310692 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/04195c1d-9fb7-479b-8061-f92e3c88aed2-env-overrides\") pod \"ovnkube-node-khlhx\" (UID: \"04195c1d-9fb7-479b-8061-f92e3c88aed2\") " pod="openshift-ovn-kubernetes/ovnkube-node-khlhx" Oct 29 22:33:00.311190 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.310697 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f190a1a0-6f6a-44d0-aeaa-b053cff4bd9c-host\") pod \"tuned-jtlmz\" (UID: \"f190a1a0-6f6a-44d0-aeaa-b053cff4bd9c\") " pod="openshift-cluster-node-tuning-operator/tuned-jtlmz" Oct 29 22:33:00.311190 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.310733 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/04195c1d-9fb7-479b-8061-f92e3c88aed2-systemd-units\") pod \"ovnkube-node-khlhx\" (UID: \"04195c1d-9fb7-479b-8061-f92e3c88aed2\") " pod="openshift-ovn-kubernetes/ovnkube-node-khlhx" Oct 29 22:33:00.311190 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.310750 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f190a1a0-6f6a-44d0-aeaa-b053cff4bd9c-etc-tuned\") pod \"tuned-jtlmz\" (UID: \"f190a1a0-6f6a-44d0-aeaa-b053cff4bd9c\") " pod="openshift-cluster-node-tuning-operator/tuned-jtlmz" Oct 29 22:33:00.311190 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.310760 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/04195c1d-9fb7-479b-8061-f92e3c88aed2-host-cni-bin\") pod \"ovnkube-node-khlhx\" (UID: \"04195c1d-9fb7-479b-8061-f92e3c88aed2\") " pod="openshift-ovn-kubernetes/ovnkube-node-khlhx" Oct 29 22:33:00.311190 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.310764 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0e1c2279-b6d4-44e0-84e6-94eb1206d2a0-hosts-file\") pod \"node-resolver-dpsd9\" (UID: \"0e1c2279-b6d4-44e0-84e6-94eb1206d2a0\") " pod="openshift-dns/node-resolver-dpsd9" Oct 29 22:33:00.312063 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.310789 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2ab2aa22-99af-4451-8d33-9e3543c8d9cd-host-run-netns\") pod \"multus-v4q29\" (UID: \"2ab2aa22-99af-4451-8d33-9e3543c8d9cd\") " pod="openshift-multus/multus-v4q29" Oct 29 22:33:00.312063 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.310777 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/04195c1d-9fb7-479b-8061-f92e3c88aed2-host-cni-netd\") pod \"ovnkube-node-khlhx\" (UID: \"04195c1d-9fb7-479b-8061-f92e3c88aed2\") " pod="openshift-ovn-kubernetes/ovnkube-node-khlhx" Oct 29 22:33:00.312063 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.310806 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/04195c1d-9fb7-479b-8061-f92e3c88aed2-run-systemd\") pod \"ovnkube-node-khlhx\" (UID: \"04195c1d-9fb7-479b-8061-f92e3c88aed2\") " pod="openshift-ovn-kubernetes/ovnkube-node-khlhx" Oct 29 22:33:00.312063 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.310832 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/04195c1d-9fb7-479b-8061-f92e3c88aed2-run-systemd\") pod \"ovnkube-node-khlhx\" (UID: \"04195c1d-9fb7-479b-8061-f92e3c88aed2\") " pod="openshift-ovn-kubernetes/ovnkube-node-khlhx" Oct 29 22:33:00.312063 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.310848 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/04195c1d-9fb7-479b-8061-f92e3c88aed2-run-ovn\") pod \"ovnkube-node-khlhx\" (UID: \"04195c1d-9fb7-479b-8061-f92e3c88aed2\") " pod="openshift-ovn-kubernetes/ovnkube-node-khlhx" Oct 29 22:33:00.312063 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.310879 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2ab2aa22-99af-4451-8d33-9e3543c8d9cd-host-var-lib-cni-bin\") pod \"multus-v4q29\" (UID: \"2ab2aa22-99af-4451-8d33-9e3543c8d9cd\") " pod="openshift-multus/multus-v4q29" Oct 29 22:33:00.312063 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.310887 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/04195c1d-9fb7-479b-8061-f92e3c88aed2-run-ovn\") pod \"ovnkube-node-khlhx\" (UID: \"04195c1d-9fb7-479b-8061-f92e3c88aed2\") " pod="openshift-ovn-kubernetes/ovnkube-node-khlhx" Oct 29 22:33:00.312063 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.310896 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2ab2aa22-99af-4451-8d33-9e3543c8d9cd-etc-kubernetes\") pod \"multus-v4q29\" (UID: \"2ab2aa22-99af-4451-8d33-9e3543c8d9cd\") " pod="openshift-multus/multus-v4q29" Oct 29 22:33:00.312063 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.310919 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/04195c1d-9fb7-479b-8061-f92e3c88aed2-host-kubelet\") pod \"ovnkube-node-khlhx\" (UID: \"04195c1d-9fb7-479b-8061-f92e3c88aed2\") " pod="openshift-ovn-kubernetes/ovnkube-node-khlhx" Oct 29 22:33:00.312063 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.310940 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/04195c1d-9fb7-479b-8061-f92e3c88aed2-host-kubelet\") pod \"ovnkube-node-khlhx\" (UID: \"04195c1d-9fb7-479b-8061-f92e3c88aed2\") " pod="openshift-ovn-kubernetes/ovnkube-node-khlhx" Oct 29 22:33:00.312063 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.310951 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f190a1a0-6f6a-44d0-aeaa-b053cff4bd9c-etc-kubernetes\") pod \"tuned-jtlmz\" (UID: \"f190a1a0-6f6a-44d0-aeaa-b053cff4bd9c\") " pod="openshift-cluster-node-tuning-operator/tuned-jtlmz" Oct 29 22:33:00.312063 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.310984 2577 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 29 22:33:00.312063 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.310991 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f190a1a0-6f6a-44d0-aeaa-b053cff4bd9c-var-lib-kubelet\") pod \"tuned-jtlmz\" (UID: \"f190a1a0-6f6a-44d0-aeaa-b053cff4bd9c\") " pod="openshift-cluster-node-tuning-operator/tuned-jtlmz" Oct 29 22:33:00.312063 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.311031 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f190a1a0-6f6a-44d0-aeaa-b053cff4bd9c-tmp\") pod \"tuned-jtlmz\" (UID: \"f190a1a0-6f6a-44d0-aeaa-b053cff4bd9c\") " pod="openshift-cluster-node-tuning-operator/tuned-jtlmz" Oct 29 22:33:00.312063 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.311056 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mxpd\" (UniqueName: \"kubernetes.io/projected/f190a1a0-6f6a-44d0-aeaa-b053cff4bd9c-kube-api-access-4mxpd\") pod \"tuned-jtlmz\" (UID: \"f190a1a0-6f6a-44d0-aeaa-b053cff4bd9c\") " pod="openshift-cluster-node-tuning-operator/tuned-jtlmz" Oct 29 22:33:00.312063 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.311102 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/04195c1d-9fb7-479b-8061-f92e3c88aed2-ovnkube-script-lib\") pod \"ovnkube-node-khlhx\" (UID: \"04195c1d-9fb7-479b-8061-f92e3c88aed2\") " pod="openshift-ovn-kubernetes/ovnkube-node-khlhx" Oct 29 22:33:00.312063 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.311128 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/93dbae94-f92d-4157-a89b-d3c984a56013-host\") pod \"node-ca-pcjl7\" (UID: \"93dbae94-f92d-4157-a89b-d3c984a56013\") " pod="openshift-image-registry/node-ca-pcjl7" Oct 29 22:33:00.312063 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.311153 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8a972cc8-dce5-4f7c-b5d5-cb24ccf73bad-cni-binary-copy\") pod \"multus-additional-cni-plugins-llm78\" (UID: \"8a972cc8-dce5-4f7c-b5d5-cb24ccf73bad\") " pod="openshift-multus/multus-additional-cni-plugins-llm78" Oct 29 22:33:00.312808 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.311209 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/8a972cc8-dce5-4f7c-b5d5-cb24ccf73bad-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-llm78\" (UID: \"8a972cc8-dce5-4f7c-b5d5-cb24ccf73bad\") " pod="openshift-multus/multus-additional-cni-plugins-llm78" Oct 29 22:33:00.312808 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.311238 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f190a1a0-6f6a-44d0-aeaa-b053cff4bd9c-etc-sysctl-d\") pod \"tuned-jtlmz\" (UID: \"f190a1a0-6f6a-44d0-aeaa-b053cff4bd9c\") " pod="openshift-cluster-node-tuning-operator/tuned-jtlmz" Oct 29 22:33:00.312808 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.311280 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2ab2aa22-99af-4451-8d33-9e3543c8d9cd-system-cni-dir\") pod \"multus-v4q29\" (UID: \"2ab2aa22-99af-4451-8d33-9e3543c8d9cd\") " pod="openshift-multus/multus-v4q29" Oct 29 22:33:00.312808 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.311306 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2ab2aa22-99af-4451-8d33-9e3543c8d9cd-multus-daemon-config\") pod \"multus-v4q29\" (UID: \"2ab2aa22-99af-4451-8d33-9e3543c8d9cd\") " pod="openshift-multus/multus-v4q29" Oct 29 22:33:00.312808 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.311353 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f8f92bfb-5cef-412e-a6c1-4bd793c59fe7-metrics-certs\") pod \"network-metrics-daemon-hzlmg\" (UID: \"f8f92bfb-5cef-412e-a6c1-4bd793c59fe7\") " pod="openshift-multus/network-metrics-daemon-hzlmg" Oct 29 22:33:00.312808 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.311380 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b23027ca-0748-4125-9bc8-9d149a3418e0-registration-dir\") pod \"aws-ebs-csi-driver-node-c96pg\" (UID: \"b23027ca-0748-4125-9bc8-9d149a3418e0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c96pg" Oct 29 22:33:00.312808 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.311460 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f190a1a0-6f6a-44d0-aeaa-b053cff4bd9c-etc-systemd\") pod \"tuned-jtlmz\" (UID: \"f190a1a0-6f6a-44d0-aeaa-b053cff4bd9c\") " pod="openshift-cluster-node-tuning-operator/tuned-jtlmz" Oct 29 22:33:00.312808 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.311485 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f190a1a0-6f6a-44d0-aeaa-b053cff4bd9c-run\") pod \"tuned-jtlmz\" (UID: \"f190a1a0-6f6a-44d0-aeaa-b053cff4bd9c\") " pod="openshift-cluster-node-tuning-operator/tuned-jtlmz" Oct 29 22:33:00.312808 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.311525 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2ab2aa22-99af-4451-8d33-9e3543c8d9cd-host-run-k8s-cni-cncf-io\") pod \"multus-v4q29\" (UID: \"2ab2aa22-99af-4451-8d33-9e3543c8d9cd\") " pod="openshift-multus/multus-v4q29" Oct 29 22:33:00.312808 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.311530 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/04195c1d-9fb7-479b-8061-f92e3c88aed2-ovnkube-script-lib\") pod \"ovnkube-node-khlhx\" (UID: \"04195c1d-9fb7-479b-8061-f92e3c88aed2\") " pod="openshift-ovn-kubernetes/ovnkube-node-khlhx" Oct 29 22:33:00.312808 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.311562 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2ab2aa22-99af-4451-8d33-9e3543c8d9cd-host-run-multus-certs\") pod \"multus-v4q29\" (UID: \"2ab2aa22-99af-4451-8d33-9e3543c8d9cd\") " pod="openshift-multus/multus-v4q29" Oct 29 22:33:00.312808 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.311608 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9v2r9\" (UniqueName: \"kubernetes.io/projected/2ab2aa22-99af-4451-8d33-9e3543c8d9cd-kube-api-access-9v2r9\") pod \"multus-v4q29\" (UID: \"2ab2aa22-99af-4451-8d33-9e3543c8d9cd\") " pod="openshift-multus/multus-v4q29" Oct 29 22:33:00.312808 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.311635 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9srv\" (UniqueName: \"kubernetes.io/projected/a4e1c52c-1eeb-4617-a0c9-d052521360d3-kube-api-access-l9srv\") pod \"network-check-target-7jv5g\" (UID: \"a4e1c52c-1eeb-4617-a0c9-d052521360d3\") " pod="openshift-network-diagnostics/network-check-target-7jv5g" Oct 29 22:33:00.312808 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.311656 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b23027ca-0748-4125-9bc8-9d149a3418e0-sys-fs\") pod \"aws-ebs-csi-driver-node-c96pg\" (UID: \"b23027ca-0748-4125-9bc8-9d149a3418e0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c96pg" Oct 29 22:33:00.312808 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.311696 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f190a1a0-6f6a-44d0-aeaa-b053cff4bd9c-sys\") pod \"tuned-jtlmz\" (UID: \"f190a1a0-6f6a-44d0-aeaa-b053cff4bd9c\") " pod="openshift-cluster-node-tuning-operator/tuned-jtlmz" Oct 29 22:33:00.312808 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.311723 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e1c2279-b6d4-44e0-84e6-94eb1206d2a0-tmp-dir\") pod \"node-resolver-dpsd9\" (UID: \"0e1c2279-b6d4-44e0-84e6-94eb1206d2a0\") " pod="openshift-dns/node-resolver-dpsd9" Oct 29 22:33:00.315540 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.315513 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/04195c1d-9fb7-479b-8061-f92e3c88aed2-ovn-node-metrics-cert\") pod \"ovnkube-node-khlhx\" (UID: \"04195c1d-9fb7-479b-8061-f92e3c88aed2\") " pod="openshift-ovn-kubernetes/ovnkube-node-khlhx" Oct 29 22:33:00.315940 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.315879 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 29 22:33:00.315940 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.315888 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 29 22:33:00.316130 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.315985 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 29 22:33:00.316130 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.316005 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 29 22:33:00.316130 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.316078 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 29 22:33:00.316130 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.316094 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 29 22:33:00.316345 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.316189 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 29 22:33:00.316345 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.316189 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 29 22:33:00.316425 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.316388 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 29 22:33:00.316803 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.316608 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 29 22:33:00.316803 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.316671 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 29 22:33:00.316803 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.316684 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 29 22:33:00.316803 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.316699 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 29 22:33:00.316803 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.316759 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 29 22:33:00.316803 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.316766 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 29 22:33:00.316803 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.316619 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 29 22:33:00.317096 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.316829 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 29 22:33:00.329061 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.329019 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-74.ec2.internal" event={"ID":"ef9ac0a87cdfc5ee9b8eaa2797bc55b1","Type":"ContainerStarted","Data":"2c31fee02392017ba8b41b846564c28cb85ccd960603212936086d59f108fd8a"} Oct 29 22:33:00.330408 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.330384 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-74.ec2.internal" event={"ID":"4c81f7639feb5d07aad0278bf28e49a4","Type":"ContainerStarted","Data":"39746dc1a72c981c5e9698f1f0c72d7c28b892b2793847e6fcf516fb3f1c727f"} Oct 29 22:33:00.332307 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.332288 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tlxfr\" (UniqueName: \"kubernetes.io/projected/04195c1d-9fb7-479b-8061-f92e3c88aed2-kube-api-access-tlxfr\") pod \"ovnkube-node-khlhx\" (UID: \"04195c1d-9fb7-479b-8061-f92e3c88aed2\") " pod="openshift-ovn-kubernetes/ovnkube-node-khlhx" Oct 29 22:33:00.412489 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.412460 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f190a1a0-6f6a-44d0-aeaa-b053cff4bd9c-lib-modules\") pod \"tuned-jtlmz\" (UID: \"f190a1a0-6f6a-44d0-aeaa-b053cff4bd9c\") " pod="openshift-cluster-node-tuning-operator/tuned-jtlmz" Oct 29 22:33:00.412641 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.412494 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7fmsg\" (UniqueName: \"kubernetes.io/projected/0e1c2279-b6d4-44e0-84e6-94eb1206d2a0-kube-api-access-7fmsg\") pod \"node-resolver-dpsd9\" (UID: \"0e1c2279-b6d4-44e0-84e6-94eb1206d2a0\") " pod="openshift-dns/node-resolver-dpsd9" Oct 29 22:33:00.412641 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.412517 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2ab2aa22-99af-4451-8d33-9e3543c8d9cd-cni-binary-copy\") pod \"multus-v4q29\" (UID: \"2ab2aa22-99af-4451-8d33-9e3543c8d9cd\") " pod="openshift-multus/multus-v4q29" Oct 29 22:33:00.412641 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.412538 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4nczb\" (UniqueName: \"kubernetes.io/projected/f8f92bfb-5cef-412e-a6c1-4bd793c59fe7-kube-api-access-4nczb\") pod \"network-metrics-daemon-hzlmg\" (UID: \"f8f92bfb-5cef-412e-a6c1-4bd793c59fe7\") " pod="openshift-multus/network-metrics-daemon-hzlmg" Oct 29 22:33:00.412641 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.412560 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqxx5\" (UniqueName: \"kubernetes.io/projected/b23027ca-0748-4125-9bc8-9d149a3418e0-kube-api-access-jqxx5\") pod \"aws-ebs-csi-driver-node-c96pg\" (UID: \"b23027ca-0748-4125-9bc8-9d149a3418e0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c96pg" Oct 29 22:33:00.412641 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.412593 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6f50ee57-a710-4325-bd45-4f861452ede4-kubelet-config\") pod \"global-pull-secret-syncer-bp46g\" (UID: \"6f50ee57-a710-4325-bd45-4f861452ede4\") " pod="kube-system/global-pull-secret-syncer-bp46g" Oct 29 22:33:00.412884 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.412640 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2ab2aa22-99af-4451-8d33-9e3543c8d9cd-hostroot\") pod \"multus-v4q29\" (UID: \"2ab2aa22-99af-4451-8d33-9e3543c8d9cd\") " pod="openshift-multus/multus-v4q29" Oct 29 22:33:00.412884 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.412663 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2ab2aa22-99af-4451-8d33-9e3543c8d9cd-multus-conf-dir\") pod \"multus-v4q29\" (UID: \"2ab2aa22-99af-4451-8d33-9e3543c8d9cd\") " pod="openshift-multus/multus-v4q29" Oct 29 22:33:00.412884 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.412688 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6f50ee57-a710-4325-bd45-4f861452ede4-original-pull-secret\") pod \"global-pull-secret-syncer-bp46g\" (UID: \"6f50ee57-a710-4325-bd45-4f861452ede4\") " pod="kube-system/global-pull-secret-syncer-bp46g" Oct 29 22:33:00.412884 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.412710 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f190a1a0-6f6a-44d0-aeaa-b053cff4bd9c-etc-sysconfig\") pod \"tuned-jtlmz\" (UID: \"f190a1a0-6f6a-44d0-aeaa-b053cff4bd9c\") " pod="openshift-cluster-node-tuning-operator/tuned-jtlmz" Oct 29 22:33:00.412884 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.412743 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2ab2aa22-99af-4451-8d33-9e3543c8d9cd-multus-socket-dir-parent\") pod \"multus-v4q29\" (UID: \"2ab2aa22-99af-4451-8d33-9e3543c8d9cd\") " pod="openshift-multus/multus-v4q29" Oct 29 22:33:00.412884 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.412771 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jxgn\" (UniqueName: \"kubernetes.io/projected/04ff31d0-a870-4527-a0c3-285737359f95-kube-api-access-8jxgn\") pod \"iptables-alerter-wpzm2\" (UID: \"04ff31d0-a870-4527-a0c3-285737359f95\") " pod="openshift-network-operator/iptables-alerter-wpzm2" Oct 29 22:33:00.412884 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.412799 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvgj5\" (UniqueName: \"kubernetes.io/projected/8a972cc8-dce5-4f7c-b5d5-cb24ccf73bad-kube-api-access-jvgj5\") pod \"multus-additional-cni-plugins-llm78\" (UID: \"8a972cc8-dce5-4f7c-b5d5-cb24ccf73bad\") " pod="openshift-multus/multus-additional-cni-plugins-llm78" Oct 29 22:33:00.412884 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.412829 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2d2d7446-0741-4144-9b19-608854f8a04e-agent-certs\") pod \"konnectivity-agent-54f9x\" (UID: \"2d2d7446-0741-4144-9b19-608854f8a04e\") " pod="kube-system/konnectivity-agent-54f9x" Oct 29 22:33:00.412884 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.412856 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6f50ee57-a710-4325-bd45-4f861452ede4-dbus\") pod \"global-pull-secret-syncer-bp46g\" (UID: \"6f50ee57-a710-4325-bd45-4f861452ede4\") " pod="kube-system/global-pull-secret-syncer-bp46g" Oct 29 22:33:00.412884 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.412879 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8a972cc8-dce5-4f7c-b5d5-cb24ccf73bad-tuning-conf-dir\") pod \"multus-additional-cni-plugins-llm78\" (UID: \"8a972cc8-dce5-4f7c-b5d5-cb24ccf73bad\") " pod="openshift-multus/multus-additional-cni-plugins-llm78" Oct 29 22:33:00.413388 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.412901 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2d2d7446-0741-4144-9b19-608854f8a04e-konnectivity-ca\") pod \"konnectivity-agent-54f9x\" (UID: \"2d2d7446-0741-4144-9b19-608854f8a04e\") " pod="kube-system/konnectivity-agent-54f9x" Oct 29 22:33:00.413388 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.412924 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2ab2aa22-99af-4451-8d33-9e3543c8d9cd-multus-cni-dir\") pod \"multus-v4q29\" (UID: \"2ab2aa22-99af-4451-8d33-9e3543c8d9cd\") " pod="openshift-multus/multus-v4q29" Oct 29 22:33:00.413388 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.412946 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2ab2aa22-99af-4451-8d33-9e3543c8d9cd-host-var-lib-kubelet\") pod \"multus-v4q29\" (UID: \"2ab2aa22-99af-4451-8d33-9e3543c8d9cd\") " pod="openshift-multus/multus-v4q29" Oct 29 22:33:00.413388 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.412971 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/04ff31d0-a870-4527-a0c3-285737359f95-host-slash\") pod \"iptables-alerter-wpzm2\" (UID: \"04ff31d0-a870-4527-a0c3-285737359f95\") " pod="openshift-network-operator/iptables-alerter-wpzm2" Oct 29 22:33:00.413388 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.413006 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b23027ca-0748-4125-9bc8-9d149a3418e0-etc-selinux\") pod \"aws-ebs-csi-driver-node-c96pg\" (UID: \"b23027ca-0748-4125-9bc8-9d149a3418e0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c96pg" Oct 29 22:33:00.413388 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.413035 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f190a1a0-6f6a-44d0-aeaa-b053cff4bd9c-etc-sysctl-conf\") pod \"tuned-jtlmz\" (UID: \"f190a1a0-6f6a-44d0-aeaa-b053cff4bd9c\") " pod="openshift-cluster-node-tuning-operator/tuned-jtlmz" Oct 29 22:33:00.413683 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.413514 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f190a1a0-6f6a-44d0-aeaa-b053cff4bd9c-host\") pod \"tuned-jtlmz\" (UID: \"f190a1a0-6f6a-44d0-aeaa-b053cff4bd9c\") " pod="openshift-cluster-node-tuning-operator/tuned-jtlmz" Oct 29 22:33:00.413683 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.413545 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f190a1a0-6f6a-44d0-aeaa-b053cff4bd9c-etc-tuned\") pod \"tuned-jtlmz\" (UID: \"f190a1a0-6f6a-44d0-aeaa-b053cff4bd9c\") " pod="openshift-cluster-node-tuning-operator/tuned-jtlmz" Oct 29 22:33:00.413683 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.413569 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0e1c2279-b6d4-44e0-84e6-94eb1206d2a0-hosts-file\") pod \"node-resolver-dpsd9\" (UID: \"0e1c2279-b6d4-44e0-84e6-94eb1206d2a0\") " pod="openshift-dns/node-resolver-dpsd9" Oct 29 22:33:00.413683 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.413593 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2ab2aa22-99af-4451-8d33-9e3543c8d9cd-host-run-netns\") pod \"multus-v4q29\" (UID: \"2ab2aa22-99af-4451-8d33-9e3543c8d9cd\") " pod="openshift-multus/multus-v4q29" Oct 29 22:33:00.413683 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.413620 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2ab2aa22-99af-4451-8d33-9e3543c8d9cd-host-var-lib-cni-bin\") pod \"multus-v4q29\" (UID: \"2ab2aa22-99af-4451-8d33-9e3543c8d9cd\") " pod="openshift-multus/multus-v4q29" Oct 29 22:33:00.413683 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.413643 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2ab2aa22-99af-4451-8d33-9e3543c8d9cd-etc-kubernetes\") pod \"multus-v4q29\" (UID: \"2ab2aa22-99af-4451-8d33-9e3543c8d9cd\") " pod="openshift-multus/multus-v4q29" Oct 29 22:33:00.413683 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.413668 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f190a1a0-6f6a-44d0-aeaa-b053cff4bd9c-etc-kubernetes\") pod \"tuned-jtlmz\" (UID: \"f190a1a0-6f6a-44d0-aeaa-b053cff4bd9c\") " pod="openshift-cluster-node-tuning-operator/tuned-jtlmz" Oct 29 22:33:00.414012 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.413690 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f190a1a0-6f6a-44d0-aeaa-b053cff4bd9c-var-lib-kubelet\") pod \"tuned-jtlmz\" (UID: \"f190a1a0-6f6a-44d0-aeaa-b053cff4bd9c\") " pod="openshift-cluster-node-tuning-operator/tuned-jtlmz" Oct 29 22:33:00.414012 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.413712 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f190a1a0-6f6a-44d0-aeaa-b053cff4bd9c-tmp\") pod \"tuned-jtlmz\" (UID: \"f190a1a0-6f6a-44d0-aeaa-b053cff4bd9c\") " pod="openshift-cluster-node-tuning-operator/tuned-jtlmz" Oct 29 22:33:00.414012 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.413735 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mxpd\" (UniqueName: \"kubernetes.io/projected/f190a1a0-6f6a-44d0-aeaa-b053cff4bd9c-kube-api-access-4mxpd\") pod \"tuned-jtlmz\" (UID: \"f190a1a0-6f6a-44d0-aeaa-b053cff4bd9c\") " pod="openshift-cluster-node-tuning-operator/tuned-jtlmz" Oct 29 22:33:00.414012 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.413761 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/93dbae94-f92d-4157-a89b-d3c984a56013-host\") pod \"node-ca-pcjl7\" (UID: \"93dbae94-f92d-4157-a89b-d3c984a56013\") " pod="openshift-image-registry/node-ca-pcjl7" Oct 29 22:33:00.414012 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.413787 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8a972cc8-dce5-4f7c-b5d5-cb24ccf73bad-cni-binary-copy\") pod \"multus-additional-cni-plugins-llm78\" (UID: \"8a972cc8-dce5-4f7c-b5d5-cb24ccf73bad\") " pod="openshift-multus/multus-additional-cni-plugins-llm78" Oct 29 22:33:00.414012 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.413812 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/8a972cc8-dce5-4f7c-b5d5-cb24ccf73bad-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-llm78\" (UID: \"8a972cc8-dce5-4f7c-b5d5-cb24ccf73bad\") " pod="openshift-multus/multus-additional-cni-plugins-llm78" Oct 29 22:33:00.414012 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.413840 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f190a1a0-6f6a-44d0-aeaa-b053cff4bd9c-etc-sysctl-d\") pod \"tuned-jtlmz\" (UID: \"f190a1a0-6f6a-44d0-aeaa-b053cff4bd9c\") " pod="openshift-cluster-node-tuning-operator/tuned-jtlmz" Oct 29 22:33:00.414012 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.413872 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2ab2aa22-99af-4451-8d33-9e3543c8d9cd-system-cni-dir\") pod \"multus-v4q29\" (UID: \"2ab2aa22-99af-4451-8d33-9e3543c8d9cd\") " pod="openshift-multus/multus-v4q29" Oct 29 22:33:00.414012 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.413896 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2ab2aa22-99af-4451-8d33-9e3543c8d9cd-multus-daemon-config\") pod \"multus-v4q29\" (UID: \"2ab2aa22-99af-4451-8d33-9e3543c8d9cd\") " pod="openshift-multus/multus-v4q29" Oct 29 22:33:00.414012 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.413920 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f8f92bfb-5cef-412e-a6c1-4bd793c59fe7-metrics-certs\") pod \"network-metrics-daemon-hzlmg\" (UID: \"f8f92bfb-5cef-412e-a6c1-4bd793c59fe7\") " pod="openshift-multus/network-metrics-daemon-hzlmg" Oct 29 22:33:00.414012 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.413946 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b23027ca-0748-4125-9bc8-9d149a3418e0-registration-dir\") pod \"aws-ebs-csi-driver-node-c96pg\" (UID: \"b23027ca-0748-4125-9bc8-9d149a3418e0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c96pg" Oct 29 22:33:00.414012 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.413967 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f190a1a0-6f6a-44d0-aeaa-b053cff4bd9c-etc-systemd\") pod \"tuned-jtlmz\" (UID: \"f190a1a0-6f6a-44d0-aeaa-b053cff4bd9c\") " pod="openshift-cluster-node-tuning-operator/tuned-jtlmz" Oct 29 22:33:00.414012 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.413992 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f190a1a0-6f6a-44d0-aeaa-b053cff4bd9c-run\") pod \"tuned-jtlmz\" (UID: \"f190a1a0-6f6a-44d0-aeaa-b053cff4bd9c\") " pod="openshift-cluster-node-tuning-operator/tuned-jtlmz" Oct 29 22:33:00.414624 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.414014 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2ab2aa22-99af-4451-8d33-9e3543c8d9cd-host-run-k8s-cni-cncf-io\") pod \"multus-v4q29\" (UID: \"2ab2aa22-99af-4451-8d33-9e3543c8d9cd\") " pod="openshift-multus/multus-v4q29" Oct 29 22:33:00.414624 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.414039 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2ab2aa22-99af-4451-8d33-9e3543c8d9cd-host-run-multus-certs\") pod \"multus-v4q29\" (UID: \"2ab2aa22-99af-4451-8d33-9e3543c8d9cd\") " pod="openshift-multus/multus-v4q29" Oct 29 22:33:00.414624 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.414063 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9v2r9\" (UniqueName: \"kubernetes.io/projected/2ab2aa22-99af-4451-8d33-9e3543c8d9cd-kube-api-access-9v2r9\") pod \"multus-v4q29\" (UID: \"2ab2aa22-99af-4451-8d33-9e3543c8d9cd\") " pod="openshift-multus/multus-v4q29" Oct 29 22:33:00.414624 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.414088 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9srv\" (UniqueName: \"kubernetes.io/projected/a4e1c52c-1eeb-4617-a0c9-d052521360d3-kube-api-access-l9srv\") pod \"network-check-target-7jv5g\" (UID: \"a4e1c52c-1eeb-4617-a0c9-d052521360d3\") " pod="openshift-network-diagnostics/network-check-target-7jv5g" Oct 29 22:33:00.414624 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.414111 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b23027ca-0748-4125-9bc8-9d149a3418e0-sys-fs\") pod \"aws-ebs-csi-driver-node-c96pg\" (UID: \"b23027ca-0748-4125-9bc8-9d149a3418e0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c96pg" Oct 29 22:33:00.414624 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.414133 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f190a1a0-6f6a-44d0-aeaa-b053cff4bd9c-sys\") pod \"tuned-jtlmz\" (UID: \"f190a1a0-6f6a-44d0-aeaa-b053cff4bd9c\") " pod="openshift-cluster-node-tuning-operator/tuned-jtlmz" Oct 29 22:33:00.414624 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.414155 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e1c2279-b6d4-44e0-84e6-94eb1206d2a0-tmp-dir\") pod \"node-resolver-dpsd9\" (UID: \"0e1c2279-b6d4-44e0-84e6-94eb1206d2a0\") " pod="openshift-dns/node-resolver-dpsd9" Oct 29 22:33:00.414624 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.414189 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b23027ca-0748-4125-9bc8-9d149a3418e0-kubelet-dir\") pod \"aws-ebs-csi-driver-node-c96pg\" (UID: \"b23027ca-0748-4125-9bc8-9d149a3418e0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c96pg" Oct 29 22:33:00.414624 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.414218 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/04ff31d0-a870-4527-a0c3-285737359f95-iptables-alerter-script\") pod \"iptables-alerter-wpzm2\" (UID: \"04ff31d0-a870-4527-a0c3-285737359f95\") " pod="openshift-network-operator/iptables-alerter-wpzm2" Oct 29 22:33:00.414624 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.414243 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b23027ca-0748-4125-9bc8-9d149a3418e0-device-dir\") pod \"aws-ebs-csi-driver-node-c96pg\" (UID: \"b23027ca-0748-4125-9bc8-9d149a3418e0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c96pg" Oct 29 22:33:00.414624 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.414264 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/93dbae94-f92d-4157-a89b-d3c984a56013-serviceca\") pod \"node-ca-pcjl7\" (UID: \"93dbae94-f92d-4157-a89b-d3c984a56013\") " pod="openshift-image-registry/node-ca-pcjl7" Oct 29 22:33:00.414624 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.414290 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wf78m\" (UniqueName: \"kubernetes.io/projected/93dbae94-f92d-4157-a89b-d3c984a56013-kube-api-access-wf78m\") pod \"node-ca-pcjl7\" (UID: \"93dbae94-f92d-4157-a89b-d3c984a56013\") " pod="openshift-image-registry/node-ca-pcjl7" Oct 29 22:33:00.414624 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.414319 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b23027ca-0748-4125-9bc8-9d149a3418e0-socket-dir\") pod \"aws-ebs-csi-driver-node-c96pg\" (UID: \"b23027ca-0748-4125-9bc8-9d149a3418e0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c96pg" Oct 29 22:33:00.414624 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.414360 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f190a1a0-6f6a-44d0-aeaa-b053cff4bd9c-etc-modprobe-d\") pod \"tuned-jtlmz\" (UID: \"f190a1a0-6f6a-44d0-aeaa-b053cff4bd9c\") " pod="openshift-cluster-node-tuning-operator/tuned-jtlmz" Oct 29 22:33:00.414624 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.414383 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8a972cc8-dce5-4f7c-b5d5-cb24ccf73bad-cnibin\") pod \"multus-additional-cni-plugins-llm78\" (UID: \"8a972cc8-dce5-4f7c-b5d5-cb24ccf73bad\") " pod="openshift-multus/multus-additional-cni-plugins-llm78" Oct 29 22:33:00.414624 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.414406 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2ab2aa22-99af-4451-8d33-9e3543c8d9cd-cnibin\") pod \"multus-v4q29\" (UID: \"2ab2aa22-99af-4451-8d33-9e3543c8d9cd\") " pod="openshift-multus/multus-v4q29" Oct 29 22:33:00.414624 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.414429 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2ab2aa22-99af-4451-8d33-9e3543c8d9cd-host-var-lib-cni-multus\") pod \"multus-v4q29\" (UID: \"2ab2aa22-99af-4451-8d33-9e3543c8d9cd\") " pod="openshift-multus/multus-v4q29" Oct 29 22:33:00.415402 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.414459 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8a972cc8-dce5-4f7c-b5d5-cb24ccf73bad-system-cni-dir\") pod \"multus-additional-cni-plugins-llm78\" (UID: \"8a972cc8-dce5-4f7c-b5d5-cb24ccf73bad\") " pod="openshift-multus/multus-additional-cni-plugins-llm78" Oct 29 22:33:00.415402 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.414484 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/8a972cc8-dce5-4f7c-b5d5-cb24ccf73bad-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-llm78\" (UID: \"8a972cc8-dce5-4f7c-b5d5-cb24ccf73bad\") " pod="openshift-multus/multus-additional-cni-plugins-llm78" Oct 29 22:33:00.415402 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.415081 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/8a972cc8-dce5-4f7c-b5d5-cb24ccf73bad-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-llm78\" (UID: \"8a972cc8-dce5-4f7c-b5d5-cb24ccf73bad\") " pod="openshift-multus/multus-additional-cni-plugins-llm78" Oct 29 22:33:00.415402 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.415226 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f190a1a0-6f6a-44d0-aeaa-b053cff4bd9c-lib-modules\") pod \"tuned-jtlmz\" (UID: \"f190a1a0-6f6a-44d0-aeaa-b053cff4bd9c\") " pod="openshift-cluster-node-tuning-operator/tuned-jtlmz" Oct 29 22:33:00.415596 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.415464 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f190a1a0-6f6a-44d0-aeaa-b053cff4bd9c-host\") pod \"tuned-jtlmz\" (UID: \"f190a1a0-6f6a-44d0-aeaa-b053cff4bd9c\") " pod="openshift-cluster-node-tuning-operator/tuned-jtlmz" Oct 29 22:33:00.415596 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.415528 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8a972cc8-dce5-4f7c-b5d5-cb24ccf73bad-tuning-conf-dir\") pod \"multus-additional-cni-plugins-llm78\" (UID: \"8a972cc8-dce5-4f7c-b5d5-cb24ccf73bad\") " pod="openshift-multus/multus-additional-cni-plugins-llm78" Oct 29 22:33:00.415717 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.415697 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/93dbae94-f92d-4157-a89b-d3c984a56013-host\") pod \"node-ca-pcjl7\" (UID: \"93dbae94-f92d-4157-a89b-d3c984a56013\") " pod="openshift-image-registry/node-ca-pcjl7" Oct 29 22:33:00.417220 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.416003 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b23027ca-0748-4125-9bc8-9d149a3418e0-sys-fs\") pod \"aws-ebs-csi-driver-node-c96pg\" (UID: \"b23027ca-0748-4125-9bc8-9d149a3418e0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c96pg" Oct 29 22:33:00.417220 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.416084 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2ab2aa22-99af-4451-8d33-9e3543c8d9cd-system-cni-dir\") pod \"multus-v4q29\" (UID: \"2ab2aa22-99af-4451-8d33-9e3543c8d9cd\") " pod="openshift-multus/multus-v4q29" Oct 29 22:33:00.417220 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.416268 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f190a1a0-6f6a-44d0-aeaa-b053cff4bd9c-sys\") pod \"tuned-jtlmz\" (UID: \"f190a1a0-6f6a-44d0-aeaa-b053cff4bd9c\") " pod="openshift-cluster-node-tuning-operator/tuned-jtlmz" Oct 29 22:33:00.417220 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.416314 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2ab2aa22-99af-4451-8d33-9e3543c8d9cd-multus-cni-dir\") pod \"multus-v4q29\" (UID: \"2ab2aa22-99af-4451-8d33-9e3543c8d9cd\") " pod="openshift-multus/multus-v4q29" Oct 29 22:33:00.417220 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.416384 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2ab2aa22-99af-4451-8d33-9e3543c8d9cd-host-var-lib-kubelet\") pod \"multus-v4q29\" (UID: \"2ab2aa22-99af-4451-8d33-9e3543c8d9cd\") " pod="openshift-multus/multus-v4q29" Oct 29 22:33:00.417220 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.416425 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/04ff31d0-a870-4527-a0c3-285737359f95-host-slash\") pod \"iptables-alerter-wpzm2\" (UID: \"04ff31d0-a870-4527-a0c3-285737359f95\") " pod="openshift-network-operator/iptables-alerter-wpzm2" Oct 29 22:33:00.417220 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.416467 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2ab2aa22-99af-4451-8d33-9e3543c8d9cd-multus-socket-dir-parent\") pod \"multus-v4q29\" (UID: \"2ab2aa22-99af-4451-8d33-9e3543c8d9cd\") " pod="openshift-multus/multus-v4q29" Oct 29 22:33:00.417220 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.416513 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b23027ca-0748-4125-9bc8-9d149a3418e0-etc-selinux\") pod \"aws-ebs-csi-driver-node-c96pg\" (UID: \"b23027ca-0748-4125-9bc8-9d149a3418e0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c96pg" Oct 29 22:33:00.417220 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.416519 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6f50ee57-a710-4325-bd45-4f861452ede4-kubelet-config\") pod \"global-pull-secret-syncer-bp46g\" (UID: \"6f50ee57-a710-4325-bd45-4f861452ede4\") " pod="kube-system/global-pull-secret-syncer-bp46g" Oct 29 22:33:00.417220 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.416565 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2ab2aa22-99af-4451-8d33-9e3543c8d9cd-hostroot\") pod \"multus-v4q29\" (UID: \"2ab2aa22-99af-4451-8d33-9e3543c8d9cd\") " pod="openshift-multus/multus-v4q29" Oct 29 22:33:00.417220 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.416609 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2ab2aa22-99af-4451-8d33-9e3543c8d9cd-multus-conf-dir\") pod \"multus-v4q29\" (UID: \"2ab2aa22-99af-4451-8d33-9e3543c8d9cd\") " pod="openshift-multus/multus-v4q29" Oct 29 22:33:00.417220 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.416658 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2d2d7446-0741-4144-9b19-608854f8a04e-konnectivity-ca\") pod \"konnectivity-agent-54f9x\" (UID: \"2d2d7446-0741-4144-9b19-608854f8a04e\") " pod="kube-system/konnectivity-agent-54f9x" Oct 29 22:33:00.417220 ip-10-0-140-74 kubenswrapper[2577]: E1029 22:33:00.416723 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 29 22:33:00.417220 ip-10-0-140-74 kubenswrapper[2577]: E1029 22:33:00.416732 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 29 22:33:00.417220 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.416723 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b23027ca-0748-4125-9bc8-9d149a3418e0-kubelet-dir\") pod \"aws-ebs-csi-driver-node-c96pg\" (UID: \"b23027ca-0748-4125-9bc8-9d149a3418e0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c96pg" Oct 29 22:33:00.417220 ip-10-0-140-74 kubenswrapper[2577]: E1029 22:33:00.416790 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6f50ee57-a710-4325-bd45-4f861452ede4-original-pull-secret podName:6f50ee57-a710-4325-bd45-4f861452ede4 nodeName:}" failed. No retries permitted until 2026-10-29 22:33:00.916773521 +0000 UTC m=+3.073961203 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6f50ee57-a710-4325-bd45-4f861452ede4-original-pull-secret") pod "global-pull-secret-syncer-bp46g" (UID: "6f50ee57-a710-4325-bd45-4f861452ede4") : object "kube-system"/"original-pull-secret" not registered Oct 29 22:33:00.417220 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.416831 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f190a1a0-6f6a-44d0-aeaa-b053cff4bd9c-etc-sysconfig\") pod \"tuned-jtlmz\" (UID: \"f190a1a0-6f6a-44d0-aeaa-b053cff4bd9c\") " pod="openshift-cluster-node-tuning-operator/tuned-jtlmz" Oct 29 22:33:00.417220 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.416831 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f190a1a0-6f6a-44d0-aeaa-b053cff4bd9c-etc-sysctl-conf\") pod \"tuned-jtlmz\" (UID: \"f190a1a0-6f6a-44d0-aeaa-b053cff4bd9c\") " pod="openshift-cluster-node-tuning-operator/tuned-jtlmz" Oct 29 22:33:00.418010 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.416828 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b23027ca-0748-4125-9bc8-9d149a3418e0-registration-dir\") pod \"aws-ebs-csi-driver-node-c96pg\" (UID: \"b23027ca-0748-4125-9bc8-9d149a3418e0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c96pg" Oct 29 22:33:00.418010 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.416893 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f190a1a0-6f6a-44d0-aeaa-b053cff4bd9c-etc-systemd\") pod \"tuned-jtlmz\" (UID: \"f190a1a0-6f6a-44d0-aeaa-b053cff4bd9c\") " pod="openshift-cluster-node-tuning-operator/tuned-jtlmz" Oct 29 22:33:00.418010 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.416938 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f190a1a0-6f6a-44d0-aeaa-b053cff4bd9c-run\") pod \"tuned-jtlmz\" (UID: \"f190a1a0-6f6a-44d0-aeaa-b053cff4bd9c\") " pod="openshift-cluster-node-tuning-operator/tuned-jtlmz" Oct 29 22:33:00.418010 ip-10-0-140-74 kubenswrapper[2577]: E1029 22:33:00.416971 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f8f92bfb-5cef-412e-a6c1-4bd793c59fe7-metrics-certs podName:f8f92bfb-5cef-412e-a6c1-4bd793c59fe7 nodeName:}" failed. No retries permitted until 2026-10-29 22:33:00.916956814 +0000 UTC m=+3.074144474 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f8f92bfb-5cef-412e-a6c1-4bd793c59fe7-metrics-certs") pod "network-metrics-daemon-hzlmg" (UID: "f8f92bfb-5cef-412e-a6c1-4bd793c59fe7") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 29 22:33:00.418010 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.416989 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2ab2aa22-99af-4451-8d33-9e3543c8d9cd-cni-binary-copy\") pod \"multus-v4q29\" (UID: \"2ab2aa22-99af-4451-8d33-9e3543c8d9cd\") " pod="openshift-multus/multus-v4q29" Oct 29 22:33:00.418010 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.416992 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2ab2aa22-99af-4451-8d33-9e3543c8d9cd-host-run-k8s-cni-cncf-io\") pod \"multus-v4q29\" (UID: \"2ab2aa22-99af-4451-8d33-9e3543c8d9cd\") " pod="openshift-multus/multus-v4q29" Oct 29 22:33:00.418010 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.417034 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2ab2aa22-99af-4451-8d33-9e3543c8d9cd-host-run-multus-certs\") pod \"multus-v4q29\" (UID: \"2ab2aa22-99af-4451-8d33-9e3543c8d9cd\") " pod="openshift-multus/multus-v4q29" Oct 29 22:33:00.418010 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.417052 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b23027ca-0748-4125-9bc8-9d149a3418e0-device-dir\") pod \"aws-ebs-csi-driver-node-c96pg\" (UID: \"b23027ca-0748-4125-9bc8-9d149a3418e0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c96pg" Oct 29 22:33:00.421069 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.417184 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8a972cc8-dce5-4f7c-b5d5-cb24ccf73bad-cni-binary-copy\") pod \"multus-additional-cni-plugins-llm78\" (UID: \"8a972cc8-dce5-4f7c-b5d5-cb24ccf73bad\") " pod="openshift-multus/multus-additional-cni-plugins-llm78" Oct 29 22:33:00.421569 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.421547 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0e1c2279-b6d4-44e0-84e6-94eb1206d2a0-hosts-file\") pod \"node-resolver-dpsd9\" (UID: \"0e1c2279-b6d4-44e0-84e6-94eb1206d2a0\") " pod="openshift-dns/node-resolver-dpsd9" Oct 29 22:33:00.421647 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.421633 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2ab2aa22-99af-4451-8d33-9e3543c8d9cd-host-run-netns\") pod \"multus-v4q29\" (UID: \"2ab2aa22-99af-4451-8d33-9e3543c8d9cd\") " pod="openshift-multus/multus-v4q29" Oct 29 22:33:00.421711 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.421695 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2ab2aa22-99af-4451-8d33-9e3543c8d9cd-host-var-lib-cni-bin\") pod \"multus-v4q29\" (UID: \"2ab2aa22-99af-4451-8d33-9e3543c8d9cd\") " pod="openshift-multus/multus-v4q29" Oct 29 22:33:00.421778 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.421754 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2ab2aa22-99af-4451-8d33-9e3543c8d9cd-etc-kubernetes\") pod \"multus-v4q29\" (UID: \"2ab2aa22-99af-4451-8d33-9e3543c8d9cd\") " pod="openshift-multus/multus-v4q29" Oct 29 22:33:00.421825 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.421811 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f190a1a0-6f6a-44d0-aeaa-b053cff4bd9c-etc-kubernetes\") pod \"tuned-jtlmz\" (UID: \"f190a1a0-6f6a-44d0-aeaa-b053cff4bd9c\") " pod="openshift-cluster-node-tuning-operator/tuned-jtlmz" Oct 29 22:33:00.422016 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.421994 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/04ff31d0-a870-4527-a0c3-285737359f95-iptables-alerter-script\") pod \"iptables-alerter-wpzm2\" (UID: \"04ff31d0-a870-4527-a0c3-285737359f95\") " pod="openshift-network-operator/iptables-alerter-wpzm2" Oct 29 22:33:00.424630 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.422101 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e1c2279-b6d4-44e0-84e6-94eb1206d2a0-tmp-dir\") pod \"node-resolver-dpsd9\" (UID: \"0e1c2279-b6d4-44e0-84e6-94eb1206d2a0\") " pod="openshift-dns/node-resolver-dpsd9" Oct 29 22:33:00.424630 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.422482 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2d2d7446-0741-4144-9b19-608854f8a04e-agent-certs\") pod \"konnectivity-agent-54f9x\" (UID: \"2d2d7446-0741-4144-9b19-608854f8a04e\") " pod="kube-system/konnectivity-agent-54f9x" Oct 29 22:33:00.424630 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.422788 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/93dbae94-f92d-4157-a89b-d3c984a56013-serviceca\") pod \"node-ca-pcjl7\" (UID: \"93dbae94-f92d-4157-a89b-d3c984a56013\") " pod="openshift-image-registry/node-ca-pcjl7" Oct 29 22:33:00.424630 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.422929 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f190a1a0-6f6a-44d0-aeaa-b053cff4bd9c-tmp\") pod \"tuned-jtlmz\" (UID: \"f190a1a0-6f6a-44d0-aeaa-b053cff4bd9c\") " pod="openshift-cluster-node-tuning-operator/tuned-jtlmz" Oct 29 22:33:00.424630 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.423228 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2ab2aa22-99af-4451-8d33-9e3543c8d9cd-multus-daemon-config\") pod \"multus-v4q29\" (UID: \"2ab2aa22-99af-4451-8d33-9e3543c8d9cd\") " pod="openshift-multus/multus-v4q29" Oct 29 22:33:00.424630 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.423524 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f190a1a0-6f6a-44d0-aeaa-b053cff4bd9c-var-lib-kubelet\") pod \"tuned-jtlmz\" (UID: \"f190a1a0-6f6a-44d0-aeaa-b053cff4bd9c\") " pod="openshift-cluster-node-tuning-operator/tuned-jtlmz" Oct 29 22:33:00.424630 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.423743 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f190a1a0-6f6a-44d0-aeaa-b053cff4bd9c-etc-sysctl-d\") pod \"tuned-jtlmz\" (UID: \"f190a1a0-6f6a-44d0-aeaa-b053cff4bd9c\") " pod="openshift-cluster-node-tuning-operator/tuned-jtlmz" Oct 29 22:33:00.424630 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.423796 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6f50ee57-a710-4325-bd45-4f861452ede4-dbus\") pod \"global-pull-secret-syncer-bp46g\" (UID: \"6f50ee57-a710-4325-bd45-4f861452ede4\") " pod="kube-system/global-pull-secret-syncer-bp46g" Oct 29 22:33:00.424630 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.424044 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8a972cc8-dce5-4f7c-b5d5-cb24ccf73bad-cnibin\") pod \"multus-additional-cni-plugins-llm78\" (UID: \"8a972cc8-dce5-4f7c-b5d5-cb24ccf73bad\") " pod="openshift-multus/multus-additional-cni-plugins-llm78" Oct 29 22:33:00.424630 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.424368 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f190a1a0-6f6a-44d0-aeaa-b053cff4bd9c-etc-modprobe-d\") pod \"tuned-jtlmz\" (UID: \"f190a1a0-6f6a-44d0-aeaa-b053cff4bd9c\") " pod="openshift-cluster-node-tuning-operator/tuned-jtlmz" Oct 29 22:33:00.424630 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.424440 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2ab2aa22-99af-4451-8d33-9e3543c8d9cd-cnibin\") pod \"multus-v4q29\" (UID: \"2ab2aa22-99af-4451-8d33-9e3543c8d9cd\") " pod="openshift-multus/multus-v4q29" Oct 29 22:33:00.424630 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.424468 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b23027ca-0748-4125-9bc8-9d149a3418e0-socket-dir\") pod \"aws-ebs-csi-driver-node-c96pg\" (UID: \"b23027ca-0748-4125-9bc8-9d149a3418e0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c96pg" Oct 29 22:33:00.424630 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.424502 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2ab2aa22-99af-4451-8d33-9e3543c8d9cd-host-var-lib-cni-multus\") pod \"multus-v4q29\" (UID: \"2ab2aa22-99af-4451-8d33-9e3543c8d9cd\") " pod="openshift-multus/multus-v4q29" Oct 29 22:33:00.424630 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.424551 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8a972cc8-dce5-4f7c-b5d5-cb24ccf73bad-system-cni-dir\") pod \"multus-additional-cni-plugins-llm78\" (UID: \"8a972cc8-dce5-4f7c-b5d5-cb24ccf73bad\") " pod="openshift-multus/multus-additional-cni-plugins-llm78" Oct 29 22:33:00.425377 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.424754 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/8a972cc8-dce5-4f7c-b5d5-cb24ccf73bad-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-llm78\" (UID: \"8a972cc8-dce5-4f7c-b5d5-cb24ccf73bad\") " pod="openshift-multus/multus-additional-cni-plugins-llm78" Oct 29 22:33:00.426980 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.425726 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f190a1a0-6f6a-44d0-aeaa-b053cff4bd9c-etc-tuned\") pod \"tuned-jtlmz\" (UID: \"f190a1a0-6f6a-44d0-aeaa-b053cff4bd9c\") " pod="openshift-cluster-node-tuning-operator/tuned-jtlmz" Oct 29 22:33:00.436246 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.436215 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mxpd\" (UniqueName: \"kubernetes.io/projected/f190a1a0-6f6a-44d0-aeaa-b053cff4bd9c-kube-api-access-4mxpd\") pod \"tuned-jtlmz\" (UID: \"f190a1a0-6f6a-44d0-aeaa-b053cff4bd9c\") " pod="openshift-cluster-node-tuning-operator/tuned-jtlmz" Oct 29 22:33:00.436579 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.436559 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wf78m\" (UniqueName: \"kubernetes.io/projected/93dbae94-f92d-4157-a89b-d3c984a56013-kube-api-access-wf78m\") pod \"node-ca-pcjl7\" (UID: \"93dbae94-f92d-4157-a89b-d3c984a56013\") " pod="openshift-image-registry/node-ca-pcjl7" Oct 29 22:33:00.437102 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.437079 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7fmsg\" (UniqueName: \"kubernetes.io/projected/0e1c2279-b6d4-44e0-84e6-94eb1206d2a0-kube-api-access-7fmsg\") pod \"node-resolver-dpsd9\" (UID: \"0e1c2279-b6d4-44e0-84e6-94eb1206d2a0\") " pod="openshift-dns/node-resolver-dpsd9" Oct 29 22:33:00.438611 ip-10-0-140-74 kubenswrapper[2577]: E1029 22:33:00.438588 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 29 22:33:00.438700 ip-10-0-140-74 kubenswrapper[2577]: E1029 22:33:00.438619 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 29 22:33:00.438700 ip-10-0-140-74 kubenswrapper[2577]: E1029 22:33:00.438633 2577 projected.go:196] Error preparing data for projected volume kube-api-access-l9srv for pod openshift-network-diagnostics/network-check-target-7jv5g: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 29 22:33:00.438700 ip-10-0-140-74 kubenswrapper[2577]: E1029 22:33:00.438694 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a4e1c52c-1eeb-4617-a0c9-d052521360d3-kube-api-access-l9srv podName:a4e1c52c-1eeb-4617-a0c9-d052521360d3 nodeName:}" failed. No retries permitted until 2026-10-29 22:33:00.938678606 +0000 UTC m=+3.095866289 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-l9srv" (UniqueName: "kubernetes.io/projected/a4e1c52c-1eeb-4617-a0c9-d052521360d3-kube-api-access-l9srv") pod "network-check-target-7jv5g" (UID: "a4e1c52c-1eeb-4617-a0c9-d052521360d3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 29 22:33:00.439998 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.439955 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jxgn\" (UniqueName: \"kubernetes.io/projected/04ff31d0-a870-4527-a0c3-285737359f95-kube-api-access-8jxgn\") pod \"iptables-alerter-wpzm2\" (UID: \"04ff31d0-a870-4527-a0c3-285737359f95\") " pod="openshift-network-operator/iptables-alerter-wpzm2" Oct 29 22:33:00.440504 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.440480 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvgj5\" (UniqueName: \"kubernetes.io/projected/8a972cc8-dce5-4f7c-b5d5-cb24ccf73bad-kube-api-access-jvgj5\") pod \"multus-additional-cni-plugins-llm78\" (UID: \"8a972cc8-dce5-4f7c-b5d5-cb24ccf73bad\") " pod="openshift-multus/multus-additional-cni-plugins-llm78" Oct 29 22:33:00.440644 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.440626 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqxx5\" (UniqueName: \"kubernetes.io/projected/b23027ca-0748-4125-9bc8-9d149a3418e0-kube-api-access-jqxx5\") pod \"aws-ebs-csi-driver-node-c96pg\" (UID: \"b23027ca-0748-4125-9bc8-9d149a3418e0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c96pg" Oct 29 22:33:00.440985 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.440963 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4nczb\" (UniqueName: \"kubernetes.io/projected/f8f92bfb-5cef-412e-a6c1-4bd793c59fe7-kube-api-access-4nczb\") pod \"network-metrics-daemon-hzlmg\" (UID: \"f8f92bfb-5cef-412e-a6c1-4bd793c59fe7\") " pod="openshift-multus/network-metrics-daemon-hzlmg" Oct 29 22:33:00.442383 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.442311 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9v2r9\" (UniqueName: \"kubernetes.io/projected/2ab2aa22-99af-4451-8d33-9e3543c8d9cd-kube-api-access-9v2r9\") pod \"multus-v4q29\" (UID: \"2ab2aa22-99af-4451-8d33-9e3543c8d9cd\") " pod="openshift-multus/multus-v4q29" Oct 29 22:33:00.490958 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.490934 2577 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 29 22:33:00.531698 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.531674 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-jtlmz" Oct 29 22:33:00.537347 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.537161 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-khlhx" Oct 29 22:33:00.540928 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.540908 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-54f9x" Oct 29 22:33:00.541982 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.541962 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-wpzm2" Oct 29 22:33:00.544078 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:33:00.544042 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf190a1a0_6f6a_44d0_aeaa_b053cff4bd9c.slice/crio-6b78f8cb1ddfe4814034df662b443fbd7598a6c8ab7f700ecf16975165b76448 WatchSource:0}: Error finding container 6b78f8cb1ddfe4814034df662b443fbd7598a6c8ab7f700ecf16975165b76448: Status 404 returned error can't find the container with id 6b78f8cb1ddfe4814034df662b443fbd7598a6c8ab7f700ecf16975165b76448 Oct 29 22:33:00.548746 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:33:00.548721 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04195c1d_9fb7_479b_8061_f92e3c88aed2.slice/crio-906939625d9827aaf66a0763c523390560b22eb094cee8e38b2583671e2bffa2 WatchSource:0}: Error finding container 906939625d9827aaf66a0763c523390560b22eb094cee8e38b2583671e2bffa2: Status 404 returned error can't find the container with id 906939625d9827aaf66a0763c523390560b22eb094cee8e38b2583671e2bffa2 Oct 29 22:33:00.555428 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.554407 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c96pg" Oct 29 22:33:00.555428 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:33:00.554622 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d2d7446_0741_4144_9b19_608854f8a04e.slice/crio-532a2188ba13d70f4724205518db4794b66779cf1496ec8df5fcd266cd31d6db WatchSource:0}: Error finding container 532a2188ba13d70f4724205518db4794b66779cf1496ec8df5fcd266cd31d6db: Status 404 returned error can't find the container with id 532a2188ba13d70f4724205518db4794b66779cf1496ec8df5fcd266cd31d6db Oct 29 22:33:00.555620 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:33:00.555435 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04ff31d0_a870_4527_a0c3_285737359f95.slice/crio-b86a3f547def9a615d8d32a9c5db979d9fc77c413b9595f6b4fca5be95b62f4d WatchSource:0}: Error finding container b86a3f547def9a615d8d32a9c5db979d9fc77c413b9595f6b4fca5be95b62f4d: Status 404 returned error can't find the container with id b86a3f547def9a615d8d32a9c5db979d9fc77c413b9595f6b4fca5be95b62f4d Oct 29 22:33:00.558857 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.558840 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-pcjl7" Oct 29 22:33:00.575391 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:33:00.575355 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb23027ca_0748_4125_9bc8_9d149a3418e0.slice/crio-6f79e832218106c064e6731373a1925dd3862c0b2ad6865dfe89b297a3ff15c2 WatchSource:0}: Error finding container 6f79e832218106c064e6731373a1925dd3862c0b2ad6865dfe89b297a3ff15c2: Status 404 returned error can't find the container with id 6f79e832218106c064e6731373a1925dd3862c0b2ad6865dfe89b297a3ff15c2 Oct 29 22:33:00.576233 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:33:00.576215 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93dbae94_f92d_4157_a89b_d3c984a56013.slice/crio-556b333946dd2fef6c2935b79d83a3334ac5de4f63fb2b425f3b36f57a4abca1 WatchSource:0}: Error finding container 556b333946dd2fef6c2935b79d83a3334ac5de4f63fb2b425f3b36f57a4abca1: Status 404 returned error can't find the container with id 556b333946dd2fef6c2935b79d83a3334ac5de4f63fb2b425f3b36f57a4abca1 Oct 29 22:33:00.579222 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.578741 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-dpsd9" Oct 29 22:33:00.581927 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.581907 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-v4q29" Oct 29 22:33:00.583127 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.583110 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-llm78" Oct 29 22:33:00.595757 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:33:00.595737 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a972cc8_dce5_4f7c_b5d5_cb24ccf73bad.slice/crio-a19be32c77f0d44cd8abc3ba5e326985413ea3d9dcd29dada2ca1c237c96f2c2 WatchSource:0}: Error finding container a19be32c77f0d44cd8abc3ba5e326985413ea3d9dcd29dada2ca1c237c96f2c2: Status 404 returned error can't find the container with id a19be32c77f0d44cd8abc3ba5e326985413ea3d9dcd29dada2ca1c237c96f2c2 Oct 29 22:33:00.918688 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.918133 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6f50ee57-a710-4325-bd45-4f861452ede4-original-pull-secret\") pod \"global-pull-secret-syncer-bp46g\" (UID: \"6f50ee57-a710-4325-bd45-4f861452ede4\") " pod="kube-system/global-pull-secret-syncer-bp46g" Oct 29 22:33:00.918688 ip-10-0-140-74 kubenswrapper[2577]: E1029 22:33:00.918297 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 29 22:33:00.918688 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:00.918389 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f8f92bfb-5cef-412e-a6c1-4bd793c59fe7-metrics-certs\") pod \"network-metrics-daemon-hzlmg\" (UID: \"f8f92bfb-5cef-412e-a6c1-4bd793c59fe7\") " pod="openshift-multus/network-metrics-daemon-hzlmg" Oct 29 22:33:00.918688 ip-10-0-140-74 kubenswrapper[2577]: E1029 22:33:00.918407 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6f50ee57-a710-4325-bd45-4f861452ede4-original-pull-secret podName:6f50ee57-a710-4325-bd45-4f861452ede4 nodeName:}" failed. No retries permitted until 2026-10-29 22:33:01.918391838 +0000 UTC m=+4.075579513 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6f50ee57-a710-4325-bd45-4f861452ede4-original-pull-secret") pod "global-pull-secret-syncer-bp46g" (UID: "6f50ee57-a710-4325-bd45-4f861452ede4") : object "kube-system"/"original-pull-secret" not registered Oct 29 22:33:00.918688 ip-10-0-140-74 kubenswrapper[2577]: E1029 22:33:00.918479 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 29 22:33:00.918688 ip-10-0-140-74 kubenswrapper[2577]: E1029 22:33:00.918541 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f8f92bfb-5cef-412e-a6c1-4bd793c59fe7-metrics-certs podName:f8f92bfb-5cef-412e-a6c1-4bd793c59fe7 nodeName:}" failed. No retries permitted until 2026-10-29 22:33:01.918526539 +0000 UTC m=+4.075714207 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f8f92bfb-5cef-412e-a6c1-4bd793c59fe7-metrics-certs") pod "network-metrics-daemon-hzlmg" (UID: "f8f92bfb-5cef-412e-a6c1-4bd793c59fe7") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 29 22:33:01.019452 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:01.018807 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9srv\" (UniqueName: \"kubernetes.io/projected/a4e1c52c-1eeb-4617-a0c9-d052521360d3-kube-api-access-l9srv\") pod \"network-check-target-7jv5g\" (UID: \"a4e1c52c-1eeb-4617-a0c9-d052521360d3\") " pod="openshift-network-diagnostics/network-check-target-7jv5g" Oct 29 22:33:01.019452 ip-10-0-140-74 kubenswrapper[2577]: E1029 22:33:01.018980 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 29 22:33:01.019452 ip-10-0-140-74 kubenswrapper[2577]: E1029 22:33:01.018998 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 29 22:33:01.019452 ip-10-0-140-74 kubenswrapper[2577]: E1029 22:33:01.019010 2577 projected.go:196] Error preparing data for projected volume kube-api-access-l9srv for pod openshift-network-diagnostics/network-check-target-7jv5g: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 29 22:33:01.019452 ip-10-0-140-74 kubenswrapper[2577]: E1029 22:33:01.019060 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a4e1c52c-1eeb-4617-a0c9-d052521360d3-kube-api-access-l9srv podName:a4e1c52c-1eeb-4617-a0c9-d052521360d3 nodeName:}" failed. No retries permitted until 2026-10-29 22:33:02.019043536 +0000 UTC m=+4.176231198 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-l9srv" (UniqueName: "kubernetes.io/projected/a4e1c52c-1eeb-4617-a0c9-d052521360d3-kube-api-access-l9srv") pod "network-check-target-7jv5g" (UID: "a4e1c52c-1eeb-4617-a0c9-d052521360d3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 29 22:33:01.268942 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:01.268875 2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-28 22:28:00 +0000 UTC" deadline="2028-04-15 20:34:30.27361742 +0000 UTC" Oct 29 22:33:01.268942 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:01.268939 2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12814h1m29.004682694s" Oct 29 22:33:01.331425 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:01.324940 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bp46g" Oct 29 22:33:01.331425 ip-10-0-140-74 kubenswrapper[2577]: E1029 22:33:01.325054 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bp46g" podUID="6f50ee57-a710-4325-bd45-4f861452ede4" Oct 29 22:33:01.356347 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:01.351848 2577 generic.go:356] "Generic (PLEG): container finished" podID="4c81f7639feb5d07aad0278bf28e49a4" containerID="cc1013fca8c05a300d6b735484f7ea18f02fa49a4188949ccb6a61dadee5670d" exitCode=0 Oct 29 22:33:01.356347 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:01.351930 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-74.ec2.internal" event={"ID":"4c81f7639feb5d07aad0278bf28e49a4","Type":"ContainerDied","Data":"cc1013fca8c05a300d6b735484f7ea18f02fa49a4188949ccb6a61dadee5670d"} Oct 29 22:33:01.377956 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:01.374591 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-llm78" event={"ID":"8a972cc8-dce5-4f7c-b5d5-cb24ccf73bad","Type":"ContainerStarted","Data":"a19be32c77f0d44cd8abc3ba5e326985413ea3d9dcd29dada2ca1c237c96f2c2"} Oct 29 22:33:01.388160 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:01.388131 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-dpsd9" event={"ID":"0e1c2279-b6d4-44e0-84e6-94eb1206d2a0","Type":"ContainerStarted","Data":"74ff81a79115f8387b397a72e407635fe9068a1bdfda4852af15c18dcc2b2181"} Oct 29 22:33:01.407056 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:01.407029 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-pcjl7" event={"ID":"93dbae94-f92d-4157-a89b-d3c984a56013","Type":"ContainerStarted","Data":"556b333946dd2fef6c2935b79d83a3334ac5de4f63fb2b425f3b36f57a4abca1"} Oct 29 22:33:01.415367 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:01.415323 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-wpzm2" event={"ID":"04ff31d0-a870-4527-a0c3-285737359f95","Type":"ContainerStarted","Data":"b86a3f547def9a615d8d32a9c5db979d9fc77c413b9595f6b4fca5be95b62f4d"} Oct 29 22:33:01.422040 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:01.418573 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-khlhx" event={"ID":"04195c1d-9fb7-479b-8061-f92e3c88aed2","Type":"ContainerStarted","Data":"906939625d9827aaf66a0763c523390560b22eb094cee8e38b2583671e2bffa2"} Oct 29 22:33:01.441290 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:01.441263 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-jtlmz" event={"ID":"f190a1a0-6f6a-44d0-aeaa-b053cff4bd9c","Type":"ContainerStarted","Data":"6b78f8cb1ddfe4814034df662b443fbd7598a6c8ab7f700ecf16975165b76448"} Oct 29 22:33:01.445920 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:01.445870 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-74.ec2.internal" event={"ID":"ef9ac0a87cdfc5ee9b8eaa2797bc55b1","Type":"ContainerStarted","Data":"5224a8a015dc5215e49834b8a31d3d856f4db56d46c58670903965e367d96af7"} Oct 29 22:33:01.448580 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:01.448556 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-v4q29" event={"ID":"2ab2aa22-99af-4451-8d33-9e3543c8d9cd","Type":"ContainerStarted","Data":"ef6db9261bebd09853a31da06a9228ae19172ae39bb69d4c5d723a543410dd82"} Oct 29 22:33:01.451008 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:01.450985 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c96pg" event={"ID":"b23027ca-0748-4125-9bc8-9d149a3418e0","Type":"ContainerStarted","Data":"6f79e832218106c064e6731373a1925dd3862c0b2ad6865dfe89b297a3ff15c2"} Oct 29 22:33:01.460794 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:01.460771 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-54f9x" event={"ID":"2d2d7446-0741-4144-9b19-608854f8a04e","Type":"ContainerStarted","Data":"532a2188ba13d70f4724205518db4794b66779cf1496ec8df5fcd266cd31d6db"} Oct 29 22:33:01.482816 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:01.482748 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-74.ec2.internal" podStartSLOduration=2.482732908 podStartE2EDuration="2.482732908s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 22:32:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:33:01.482017469 +0000 UTC m=+3.639205154" watchObservedRunningTime="2026-10-29 22:33:01.482732908 +0000 UTC m=+3.639920593" Oct 29 22:33:01.931643 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:01.931553 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f8f92bfb-5cef-412e-a6c1-4bd793c59fe7-metrics-certs\") pod \"network-metrics-daemon-hzlmg\" (UID: \"f8f92bfb-5cef-412e-a6c1-4bd793c59fe7\") " pod="openshift-multus/network-metrics-daemon-hzlmg" Oct 29 22:33:01.931808 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:01.931698 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6f50ee57-a710-4325-bd45-4f861452ede4-original-pull-secret\") pod \"global-pull-secret-syncer-bp46g\" (UID: \"6f50ee57-a710-4325-bd45-4f861452ede4\") " pod="kube-system/global-pull-secret-syncer-bp46g" Oct 29 22:33:01.931863 ip-10-0-140-74 kubenswrapper[2577]: E1029 22:33:01.931838 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 29 22:33:01.931916 ip-10-0-140-74 kubenswrapper[2577]: E1029 22:33:01.931899 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6f50ee57-a710-4325-bd45-4f861452ede4-original-pull-secret podName:6f50ee57-a710-4325-bd45-4f861452ede4 nodeName:}" failed. No retries permitted until 2026-10-29 22:33:03.931882345 +0000 UTC m=+6.089070023 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6f50ee57-a710-4325-bd45-4f861452ede4-original-pull-secret") pod "global-pull-secret-syncer-bp46g" (UID: "6f50ee57-a710-4325-bd45-4f861452ede4") : object "kube-system"/"original-pull-secret" not registered Oct 29 22:33:01.932294 ip-10-0-140-74 kubenswrapper[2577]: E1029 22:33:01.932276 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 29 22:33:01.932372 ip-10-0-140-74 kubenswrapper[2577]: E1029 22:33:01.932345 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f8f92bfb-5cef-412e-a6c1-4bd793c59fe7-metrics-certs podName:f8f92bfb-5cef-412e-a6c1-4bd793c59fe7 nodeName:}" failed. No retries permitted until 2026-10-29 22:33:03.93231484 +0000 UTC m=+6.089502514 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f8f92bfb-5cef-412e-a6c1-4bd793c59fe7-metrics-certs") pod "network-metrics-daemon-hzlmg" (UID: "f8f92bfb-5cef-412e-a6c1-4bd793c59fe7") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 29 22:33:02.032367 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:02.032321 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9srv\" (UniqueName: \"kubernetes.io/projected/a4e1c52c-1eeb-4617-a0c9-d052521360d3-kube-api-access-l9srv\") pod \"network-check-target-7jv5g\" (UID: \"a4e1c52c-1eeb-4617-a0c9-d052521360d3\") " pod="openshift-network-diagnostics/network-check-target-7jv5g" Oct 29 22:33:02.032535 ip-10-0-140-74 kubenswrapper[2577]: E1029 22:33:02.032503 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 29 22:33:02.032535 ip-10-0-140-74 kubenswrapper[2577]: E1029 22:33:02.032525 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 29 22:33:02.032535 ip-10-0-140-74 kubenswrapper[2577]: E1029 22:33:02.032537 2577 projected.go:196] Error preparing data for projected volume kube-api-access-l9srv for pod openshift-network-diagnostics/network-check-target-7jv5g: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 29 22:33:02.032679 ip-10-0-140-74 kubenswrapper[2577]: E1029 22:33:02.032592 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a4e1c52c-1eeb-4617-a0c9-d052521360d3-kube-api-access-l9srv podName:a4e1c52c-1eeb-4617-a0c9-d052521360d3 nodeName:}" failed. No retries permitted until 2026-10-29 22:33:04.032574128 +0000 UTC m=+6.189761803 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-l9srv" (UniqueName: "kubernetes.io/projected/a4e1c52c-1eeb-4617-a0c9-d052521360d3-kube-api-access-l9srv") pod "network-check-target-7jv5g" (UID: "a4e1c52c-1eeb-4617-a0c9-d052521360d3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 29 22:33:02.269970 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:02.269862 2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-28 22:28:00 +0000 UTC" deadline="2028-06-29 02:39:50.648739052 +0000 UTC" Oct 29 22:33:02.269970 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:02.269924 2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14596h6m48.378819029s" Oct 29 22:33:02.330381 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:02.329483 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7jv5g" Oct 29 22:33:02.330381 ip-10-0-140-74 kubenswrapper[2577]: E1029 22:33:02.329613 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7jv5g" podUID="a4e1c52c-1eeb-4617-a0c9-d052521360d3" Oct 29 22:33:02.330381 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:02.330113 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hzlmg" Oct 29 22:33:02.330381 ip-10-0-140-74 kubenswrapper[2577]: E1029 22:33:02.330224 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hzlmg" podUID="f8f92bfb-5cef-412e-a6c1-4bd793c59fe7" Oct 29 22:33:02.473215 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:02.473098 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-74.ec2.internal" event={"ID":"4c81f7639feb5d07aad0278bf28e49a4","Type":"ContainerStarted","Data":"6757406edfe64af5cec51cc522da581f0e82635378a89ce2ec008829acfa9837"} Oct 29 22:33:03.324792 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:03.324763 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bp46g" Oct 29 22:33:03.325153 ip-10-0-140-74 kubenswrapper[2577]: E1029 22:33:03.324885 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bp46g" podUID="6f50ee57-a710-4325-bd45-4f861452ede4" Oct 29 22:33:03.496276 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:03.494997 2577 generic.go:356] "Generic (PLEG): container finished" podID="8a972cc8-dce5-4f7c-b5d5-cb24ccf73bad" containerID="93cfdb05efe755447ad1112baf02fe611d673a0725f410122248e2005c6ec90e" exitCode=0 Oct 29 22:33:03.496276 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:03.496018 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-llm78" event={"ID":"8a972cc8-dce5-4f7c-b5d5-cb24ccf73bad","Type":"ContainerDied","Data":"93cfdb05efe755447ad1112baf02fe611d673a0725f410122248e2005c6ec90e"} Oct 29 22:33:03.559893 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:03.558904 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-74.ec2.internal" podStartSLOduration=4.558885432 podStartE2EDuration="4.558885432s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 22:32:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:33:02.509690478 +0000 UTC m=+4.666878162" watchObservedRunningTime="2026-10-29 22:33:03.558885432 +0000 UTC m=+5.716073117" Oct 29 22:33:03.950317 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:03.950271 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6f50ee57-a710-4325-bd45-4f861452ede4-original-pull-secret\") pod \"global-pull-secret-syncer-bp46g\" (UID: \"6f50ee57-a710-4325-bd45-4f861452ede4\") " pod="kube-system/global-pull-secret-syncer-bp46g" Oct 29 22:33:03.950475 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:03.950358 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f8f92bfb-5cef-412e-a6c1-4bd793c59fe7-metrics-certs\") pod \"network-metrics-daemon-hzlmg\" (UID: \"f8f92bfb-5cef-412e-a6c1-4bd793c59fe7\") " pod="openshift-multus/network-metrics-daemon-hzlmg" Oct 29 22:33:03.950531 ip-10-0-140-74 kubenswrapper[2577]: E1029 22:33:03.950511 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 29 22:33:03.950586 ip-10-0-140-74 kubenswrapper[2577]: E1029 22:33:03.950574 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f8f92bfb-5cef-412e-a6c1-4bd793c59fe7-metrics-certs podName:f8f92bfb-5cef-412e-a6c1-4bd793c59fe7 nodeName:}" failed. No retries permitted until 2026-10-29 22:33:07.950554295 +0000 UTC m=+10.107741961 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f8f92bfb-5cef-412e-a6c1-4bd793c59fe7-metrics-certs") pod "network-metrics-daemon-hzlmg" (UID: "f8f92bfb-5cef-412e-a6c1-4bd793c59fe7") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 29 22:33:03.950987 ip-10-0-140-74 kubenswrapper[2577]: E1029 22:33:03.950970 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 29 22:33:03.951065 ip-10-0-140-74 kubenswrapper[2577]: E1029 22:33:03.951020 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6f50ee57-a710-4325-bd45-4f861452ede4-original-pull-secret podName:6f50ee57-a710-4325-bd45-4f861452ede4 nodeName:}" failed. No retries permitted until 2026-10-29 22:33:07.951006313 +0000 UTC m=+10.108193990 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6f50ee57-a710-4325-bd45-4f861452ede4-original-pull-secret") pod "global-pull-secret-syncer-bp46g" (UID: "6f50ee57-a710-4325-bd45-4f861452ede4") : object "kube-system"/"original-pull-secret" not registered Oct 29 22:33:04.052067 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:04.051456 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9srv\" (UniqueName: \"kubernetes.io/projected/a4e1c52c-1eeb-4617-a0c9-d052521360d3-kube-api-access-l9srv\") pod \"network-check-target-7jv5g\" (UID: \"a4e1c52c-1eeb-4617-a0c9-d052521360d3\") " pod="openshift-network-diagnostics/network-check-target-7jv5g" Oct 29 22:33:04.052067 ip-10-0-140-74 kubenswrapper[2577]: E1029 22:33:04.051656 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 29 22:33:04.052067 ip-10-0-140-74 kubenswrapper[2577]: E1029 22:33:04.051677 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 29 22:33:04.052067 ip-10-0-140-74 kubenswrapper[2577]: E1029 22:33:04.051689 2577 projected.go:196] Error preparing data for projected volume kube-api-access-l9srv for pod openshift-network-diagnostics/network-check-target-7jv5g: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 29 22:33:04.052067 ip-10-0-140-74 kubenswrapper[2577]: E1029 22:33:04.051743 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a4e1c52c-1eeb-4617-a0c9-d052521360d3-kube-api-access-l9srv podName:a4e1c52c-1eeb-4617-a0c9-d052521360d3 nodeName:}" failed. No retries permitted until 2026-10-29 22:33:08.051724487 +0000 UTC m=+10.208912170 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-l9srv" (UniqueName: "kubernetes.io/projected/a4e1c52c-1eeb-4617-a0c9-d052521360d3-kube-api-access-l9srv") pod "network-check-target-7jv5g" (UID: "a4e1c52c-1eeb-4617-a0c9-d052521360d3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 29 22:33:04.326927 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:04.326830 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7jv5g" Oct 29 22:33:04.327472 ip-10-0-140-74 kubenswrapper[2577]: E1029 22:33:04.326959 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7jv5g" podUID="a4e1c52c-1eeb-4617-a0c9-d052521360d3" Oct 29 22:33:04.327472 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:04.327020 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hzlmg" Oct 29 22:33:04.327472 ip-10-0-140-74 kubenswrapper[2577]: E1029 22:33:04.327096 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hzlmg" podUID="f8f92bfb-5cef-412e-a6c1-4bd793c59fe7" Oct 29 22:33:04.502852 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:04.502818 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-54f9x" event={"ID":"2d2d7446-0741-4144-9b19-608854f8a04e","Type":"ContainerStarted","Data":"a6c85cb1040f7beddc4c84b999a539a89006d67258e2e4bf4438ad2f3413d0db"} Oct 29 22:33:05.324303 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:05.324273 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bp46g" Oct 29 22:33:05.324539 ip-10-0-140-74 kubenswrapper[2577]: E1029 22:33:05.324418 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bp46g" podUID="6f50ee57-a710-4325-bd45-4f861452ede4" Oct 29 22:33:06.325170 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:06.325103 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7jv5g" Oct 29 22:33:06.325533 ip-10-0-140-74 kubenswrapper[2577]: E1029 22:33:06.325225 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7jv5g" podUID="a4e1c52c-1eeb-4617-a0c9-d052521360d3" Oct 29 22:33:06.325630 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:06.325613 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hzlmg" Oct 29 22:33:06.325745 ip-10-0-140-74 kubenswrapper[2577]: E1029 22:33:06.325725 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hzlmg" podUID="f8f92bfb-5cef-412e-a6c1-4bd793c59fe7" Oct 29 22:33:06.518188 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:06.518151 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c96pg" event={"ID":"b23027ca-0748-4125-9bc8-9d149a3418e0","Type":"ContainerStarted","Data":"e3641cf7a1cb1a0e3348264ac28284a1b42fbe4eff870c85608e53bf3e7f069a"} Oct 29 22:33:06.520122 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:06.520097 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-pcjl7" event={"ID":"93dbae94-f92d-4157-a89b-d3c984a56013","Type":"ContainerStarted","Data":"33a8d2aaa33804b217cf132683524d326dfb59baeea430e3f23581b8b58c4bd2"} Oct 29 22:33:06.555092 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:06.555046 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-54f9x" podStartSLOduration=5.246705103 podStartE2EDuration="8.555029463s" totalImagesPullingTime="3.30832436s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:32:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:33:04.53934504 +0000 UTC m=+6.696532720" watchObservedRunningTime="2026-10-29 22:33:06.555029463 +0000 UTC m=+8.712217147" Oct 29 22:33:07.325241 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:07.325170 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bp46g" Oct 29 22:33:07.325716 ip-10-0-140-74 kubenswrapper[2577]: E1029 22:33:07.325304 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bp46g" podUID="6f50ee57-a710-4325-bd45-4f861452ede4" Oct 29 22:33:07.757488 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:07.756223 2577 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 29 22:33:07.982163 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:07.982130 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6f50ee57-a710-4325-bd45-4f861452ede4-original-pull-secret\") pod \"global-pull-secret-syncer-bp46g\" (UID: \"6f50ee57-a710-4325-bd45-4f861452ede4\") " pod="kube-system/global-pull-secret-syncer-bp46g" Oct 29 22:33:07.982271 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:07.982201 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f8f92bfb-5cef-412e-a6c1-4bd793c59fe7-metrics-certs\") pod \"network-metrics-daemon-hzlmg\" (UID: \"f8f92bfb-5cef-412e-a6c1-4bd793c59fe7\") " pod="openshift-multus/network-metrics-daemon-hzlmg" Oct 29 22:33:07.982352 ip-10-0-140-74 kubenswrapper[2577]: E1029 22:33:07.982325 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 29 22:33:07.982442 ip-10-0-140-74 kubenswrapper[2577]: E1029 22:33:07.982402 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f8f92bfb-5cef-412e-a6c1-4bd793c59fe7-metrics-certs podName:f8f92bfb-5cef-412e-a6c1-4bd793c59fe7 nodeName:}" failed. No retries permitted until 2026-10-29 22:33:15.982383935 +0000 UTC m=+18.139571610 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f8f92bfb-5cef-412e-a6c1-4bd793c59fe7-metrics-certs") pod "network-metrics-daemon-hzlmg" (UID: "f8f92bfb-5cef-412e-a6c1-4bd793c59fe7") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 29 22:33:07.982492 ip-10-0-140-74 kubenswrapper[2577]: E1029 22:33:07.982475 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 29 22:33:07.982544 ip-10-0-140-74 kubenswrapper[2577]: E1029 22:33:07.982534 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6f50ee57-a710-4325-bd45-4f861452ede4-original-pull-secret podName:6f50ee57-a710-4325-bd45-4f861452ede4 nodeName:}" failed. No retries permitted until 2026-10-29 22:33:15.982522967 +0000 UTC m=+18.139710642 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6f50ee57-a710-4325-bd45-4f861452ede4-original-pull-secret") pod "global-pull-secret-syncer-bp46g" (UID: "6f50ee57-a710-4325-bd45-4f861452ede4") : object "kube-system"/"original-pull-secret" not registered Oct 29 22:33:08.083197 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:08.083156 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9srv\" (UniqueName: \"kubernetes.io/projected/a4e1c52c-1eeb-4617-a0c9-d052521360d3-kube-api-access-l9srv\") pod \"network-check-target-7jv5g\" (UID: \"a4e1c52c-1eeb-4617-a0c9-d052521360d3\") " pod="openshift-network-diagnostics/network-check-target-7jv5g" Oct 29 22:33:08.083350 ip-10-0-140-74 kubenswrapper[2577]: E1029 22:33:08.083302 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 29 22:33:08.083350 ip-10-0-140-74 kubenswrapper[2577]: E1029 22:33:08.083319 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 29 22:33:08.083350 ip-10-0-140-74 kubenswrapper[2577]: E1029 22:33:08.083345 2577 projected.go:196] Error preparing data for projected volume kube-api-access-l9srv for pod openshift-network-diagnostics/network-check-target-7jv5g: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 29 22:33:08.083523 ip-10-0-140-74 kubenswrapper[2577]: E1029 22:33:08.083408 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a4e1c52c-1eeb-4617-a0c9-d052521360d3-kube-api-access-l9srv podName:a4e1c52c-1eeb-4617-a0c9-d052521360d3 nodeName:}" failed. No retries permitted until 2026-10-29 22:33:16.083391077 +0000 UTC m=+18.240578751 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-l9srv" (UniqueName: "kubernetes.io/projected/a4e1c52c-1eeb-4617-a0c9-d052521360d3-kube-api-access-l9srv") pod "network-check-target-7jv5g" (UID: "a4e1c52c-1eeb-4617-a0c9-d052521360d3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 29 22:33:08.314959 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:08.314232 2577 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-54f9x" Oct 29 22:33:08.315378 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:08.315325 2577 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-54f9x" Oct 29 22:33:08.324387 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:08.324365 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7jv5g" Oct 29 22:33:08.324494 ip-10-0-140-74 kubenswrapper[2577]: E1029 22:33:08.324450 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7jv5g" podUID="a4e1c52c-1eeb-4617-a0c9-d052521360d3" Oct 29 22:33:08.324580 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:08.324512 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hzlmg" Oct 29 22:33:08.324642 ip-10-0-140-74 kubenswrapper[2577]: E1029 22:33:08.324622 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hzlmg" podUID="f8f92bfb-5cef-412e-a6c1-4bd793c59fe7" Oct 29 22:33:08.351254 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:08.351196 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-pcjl7" podStartSLOduration=5.489424866 podStartE2EDuration="10.351179499s" totalImagesPullingTime="4.861754633s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 22:32:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:33:06.555497893 +0000 UTC m=+8.712685579" watchObservedRunningTime="2026-10-29 22:33:08.351179499 +0000 UTC m=+10.508367183" Oct 29 22:33:08.419707 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:08.419593 2577 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-29T22:33:07.756236849Z","UUID":"95f27512-3bc0-48c6-9faa-21de057763d8","Handler":null,"Name":"","Endpoint":""} Oct 29 22:33:08.421798 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:08.421749 2577 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 29 22:33:08.421798 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:08.421775 2577 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 29 22:33:08.527535 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:08.527500 2577 generic.go:356] "Generic (PLEG): container finished" podID="8a972cc8-dce5-4f7c-b5d5-cb24ccf73bad" containerID="c9362cd1dc9fc8be11035beb2b37ad66e41e0d36082b49d8e4b5a8db4ed20470" exitCode=0 Oct 29 22:33:08.527720 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:08.527596 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-llm78" event={"ID":"8a972cc8-dce5-4f7c-b5d5-cb24ccf73bad","Type":"ContainerDied","Data":"c9362cd1dc9fc8be11035beb2b37ad66e41e0d36082b49d8e4b5a8db4ed20470"} Oct 29 22:33:08.530467 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:08.530385 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-dpsd9" event={"ID":"0e1c2279-b6d4-44e0-84e6-94eb1206d2a0","Type":"ContainerStarted","Data":"7a8a82a2a17cfbe3816c9ba9400c96ce6c7464169e22ddfc8e38b69132dea497"} Oct 29 22:33:08.532058 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:08.532012 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-v4q29" event={"ID":"2ab2aa22-99af-4451-8d33-9e3543c8d9cd","Type":"ContainerStarted","Data":"28e57de5749d96c3824593d5797bbf235435fcd53eb52733cc68f11b4a20e343"} Oct 29 22:33:08.534770 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:08.534741 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c96pg" event={"ID":"b23027ca-0748-4125-9bc8-9d149a3418e0","Type":"ContainerStarted","Data":"07bde45c29fcbbe1b44df6d20eb5ceb11273a6ff306eef319f81497f81d76059"} Oct 29 22:33:08.604836 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:08.604793 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-dpsd9" podStartSLOduration=2.887330622 podStartE2EDuration="10.604781204s" totalImagesPullingTime="7.717450582s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 22:32:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:33:08.604540386 +0000 UTC m=+10.761728070" watchObservedRunningTime="2026-10-29 22:33:08.604781204 +0000 UTC m=+10.761968887" Oct 29 22:33:08.643034 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:08.642979 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-v4q29" podStartSLOduration=3.297342633 podStartE2EDuration="10.642960303s" totalImagesPullingTime="7.34561767s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 22:32:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:33:08.642105808 +0000 UTC m=+10.799293492" watchObservedRunningTime="2026-10-29 22:33:08.642960303 +0000 UTC m=+10.800147987" Oct 29 22:33:09.324294 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:09.324206 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bp46g" Oct 29 22:33:09.324459 ip-10-0-140-74 kubenswrapper[2577]: E1029 22:33:09.324314 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bp46g" podUID="6f50ee57-a710-4325-bd45-4f861452ede4" Oct 29 22:33:09.539325 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:09.539288 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c96pg" event={"ID":"b23027ca-0748-4125-9bc8-9d149a3418e0","Type":"ContainerStarted","Data":"2bf528a87632738e75f967dd9d941a1c7bf09dce60aed76bf36fae2d02db2e38"} Oct 29 22:33:09.542095 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:09.542063 2577 generic.go:356] "Generic (PLEG): container finished" podID="8a972cc8-dce5-4f7c-b5d5-cb24ccf73bad" containerID="509bf01b64666e56e89300f5782fa80e720d319c48f512372f2ff6dc7aae771d" exitCode=0 Oct 29 22:33:09.542211 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:09.542174 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-llm78" event={"ID":"8a972cc8-dce5-4f7c-b5d5-cb24ccf73bad","Type":"ContainerDied","Data":"509bf01b64666e56e89300f5782fa80e720d319c48f512372f2ff6dc7aae771d"} Oct 29 22:33:09.579043 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:09.578999 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c96pg" podStartSLOduration=3.834402076 podStartE2EDuration="11.578987109s" totalImagesPullingTime="7.744585033s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 22:32:58 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:33:09.578653258 +0000 UTC m=+11.735840965" watchObservedRunningTime="2026-10-29 22:33:09.578987109 +0000 UTC m=+11.736174835" Oct 29 22:33:10.325917 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:10.325395 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hzlmg" Oct 29 22:33:10.325917 ip-10-0-140-74 kubenswrapper[2577]: E1029 22:33:10.325536 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hzlmg" podUID="f8f92bfb-5cef-412e-a6c1-4bd793c59fe7" Oct 29 22:33:10.325917 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:10.325402 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7jv5g" Oct 29 22:33:10.325917 ip-10-0-140-74 kubenswrapper[2577]: E1029 22:33:10.325886 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7jv5g" podUID="a4e1c52c-1eeb-4617-a0c9-d052521360d3" Oct 29 22:33:10.545651 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:10.545593 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-wpzm2" event={"ID":"04ff31d0-a870-4527-a0c3-285737359f95","Type":"ContainerStarted","Data":"32ac7255d0a1fa835a2aab00f66a47fa4e38cbc3287f29bd53359a346fe8d8ad"} Oct 29 22:33:10.580918 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:10.580875 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-wpzm2" podStartSLOduration=4.9743759260000004 podStartE2EDuration="12.580861554s" totalImagesPullingTime="7.606485628s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 22:32:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:33:10.580413064 +0000 UTC m=+12.737600746" watchObservedRunningTime="2026-10-29 22:33:10.580861554 +0000 UTC m=+12.738049237" Oct 29 22:33:11.325135 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:11.325097 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bp46g" Oct 29 22:33:11.325301 ip-10-0-140-74 kubenswrapper[2577]: E1029 22:33:11.325217 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bp46g" podUID="6f50ee57-a710-4325-bd45-4f861452ede4" Oct 29 22:33:11.550779 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:11.550748 2577 generic.go:356] "Generic (PLEG): container finished" podID="8a972cc8-dce5-4f7c-b5d5-cb24ccf73bad" containerID="63808e2ca3fe89606c5aec28d215f23b4bd9e746d7bdb945fa9c7792d2f18cbc" exitCode=0 Oct 29 22:33:11.551436 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:11.550838 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-llm78" event={"ID":"8a972cc8-dce5-4f7c-b5d5-cb24ccf73bad","Type":"ContainerDied","Data":"63808e2ca3fe89606c5aec28d215f23b4bd9e746d7bdb945fa9c7792d2f18cbc"} Oct 29 22:33:11.552525 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:11.552500 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-jtlmz" event={"ID":"f190a1a0-6f6a-44d0-aeaa-b053cff4bd9c","Type":"ContainerStarted","Data":"f9a75848a7a8fad280cdb013a0d303c3e6ad6ce66e37189fe311e197dd531607"} Oct 29 22:33:11.650023 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:11.649918 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-jtlmz" podStartSLOduration=3.700678955 podStartE2EDuration="13.649902431s" totalImagesPullingTime="9.949223476s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 22:32:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:33:11.649373757 +0000 UTC m=+13.806561441" watchObservedRunningTime="2026-10-29 22:33:11.649902431 +0000 UTC m=+13.807090114" Oct 29 22:33:12.325018 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:12.324986 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hzlmg" Oct 29 22:33:12.325181 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:12.324987 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7jv5g" Oct 29 22:33:12.325181 ip-10-0-140-74 kubenswrapper[2577]: E1029 22:33:12.325137 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hzlmg" podUID="f8f92bfb-5cef-412e-a6c1-4bd793c59fe7" Oct 29 22:33:12.325268 ip-10-0-140-74 kubenswrapper[2577]: E1029 22:33:12.325195 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7jv5g" podUID="a4e1c52c-1eeb-4617-a0c9-d052521360d3" Oct 29 22:33:13.325405 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:13.325213 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bp46g" Oct 29 22:33:13.325781 ip-10-0-140-74 kubenswrapper[2577]: E1029 22:33:13.325504 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bp46g" podUID="6f50ee57-a710-4325-bd45-4f861452ede4" Oct 29 22:33:13.560174 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:13.560083 2577 generic.go:356] "Generic (PLEG): container finished" podID="8a972cc8-dce5-4f7c-b5d5-cb24ccf73bad" containerID="8f8ee81ce02909c8bc39a0050b5296d599e92f270670edf383497f06f78ee26d" exitCode=0 Oct 29 22:33:13.560174 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:13.560144 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-llm78" event={"ID":"8a972cc8-dce5-4f7c-b5d5-cb24ccf73bad","Type":"ContainerDied","Data":"8f8ee81ce02909c8bc39a0050b5296d599e92f270670edf383497f06f78ee26d"} Oct 29 22:33:14.324825 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:14.324794 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7jv5g" Oct 29 22:33:14.324973 ip-10-0-140-74 kubenswrapper[2577]: E1029 22:33:14.324887 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7jv5g" podUID="a4e1c52c-1eeb-4617-a0c9-d052521360d3" Oct 29 22:33:14.324973 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:14.324965 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hzlmg" Oct 29 22:33:14.325071 ip-10-0-140-74 kubenswrapper[2577]: E1029 22:33:14.325055 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hzlmg" podUID="f8f92bfb-5cef-412e-a6c1-4bd793c59fe7" Oct 29 22:33:14.564581 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:14.564547 2577 generic.go:356] "Generic (PLEG): container finished" podID="8a972cc8-dce5-4f7c-b5d5-cb24ccf73bad" containerID="fca31e6edd11c11236b1a8598967db8b22f22b67bd81bf0a2579bccd8f0785f8" exitCode=0 Oct 29 22:33:14.564938 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:14.564606 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-llm78" event={"ID":"8a972cc8-dce5-4f7c-b5d5-cb24ccf73bad","Type":"ContainerDied","Data":"fca31e6edd11c11236b1a8598967db8b22f22b67bd81bf0a2579bccd8f0785f8"} Oct 29 22:33:15.325101 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:15.324890 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bp46g" Oct 29 22:33:15.325254 ip-10-0-140-74 kubenswrapper[2577]: E1029 22:33:15.325121 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bp46g" podUID="6f50ee57-a710-4325-bd45-4f861452ede4" Oct 29 22:33:15.569455 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:15.569417 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-khlhx" event={"ID":"04195c1d-9fb7-479b-8061-f92e3c88aed2","Type":"ContainerStarted","Data":"5d4d491be807f12a86cd941b5424a0f121a711d1fc9431b610d3cb46fbd8fee2"} Oct 29 22:33:15.569866 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:15.569492 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-khlhx" event={"ID":"04195c1d-9fb7-479b-8061-f92e3c88aed2","Type":"ContainerStarted","Data":"a19b50be067d0265b5404081199bf802e460dbc3f8633b2f5b5dea49158f96d3"} Oct 29 22:33:15.569866 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:15.569542 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-khlhx" event={"ID":"04195c1d-9fb7-479b-8061-f92e3c88aed2","Type":"ContainerStarted","Data":"b36c27e3882fcce6200532beabde07cd0f36fda776b65fe226baacd469415c79"} Oct 29 22:33:15.569866 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:15.569573 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-khlhx" event={"ID":"04195c1d-9fb7-479b-8061-f92e3c88aed2","Type":"ContainerStarted","Data":"7f0f3ef1cfa4dd91a55d7593866fe7a6e6b34e7ff1d67d1c5da3ae3c4eae8433"} Oct 29 22:33:15.569866 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:15.569600 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-khlhx" event={"ID":"04195c1d-9fb7-479b-8061-f92e3c88aed2","Type":"ContainerStarted","Data":"f8af61f034d707a182371d07077081b9ca0beca7e5e6676708303429a858a2e5"} Oct 29 22:33:15.569866 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:15.569644 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-khlhx" event={"ID":"04195c1d-9fb7-479b-8061-f92e3c88aed2","Type":"ContainerStarted","Data":"762fdad213495d7d5c28d95c3e622c5eda75ba5ca5fe738144c93120454eed9c"} Oct 29 22:33:15.572602 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:15.572576 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-llm78" event={"ID":"8a972cc8-dce5-4f7c-b5d5-cb24ccf73bad","Type":"ContainerStarted","Data":"996727ae8d43dced534b2e2e697f696dc33ff367b8fdd7067f6d708b56977d53"} Oct 29 22:33:15.624603 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:15.624523 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-llm78" podStartSLOduration=7.573374196 podStartE2EDuration="17.624511029s" totalImagesPullingTime="10.051136833s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 22:32:58 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:33:15.624031462 +0000 UTC m=+17.781219151" watchObservedRunningTime="2026-10-29 22:33:15.624511029 +0000 UTC m=+17.781698711" Oct 29 22:33:16.054932 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:16.054855 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6f50ee57-a710-4325-bd45-4f861452ede4-original-pull-secret\") pod \"global-pull-secret-syncer-bp46g\" (UID: \"6f50ee57-a710-4325-bd45-4f861452ede4\") " pod="kube-system/global-pull-secret-syncer-bp46g" Oct 29 22:33:16.054932 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:16.054892 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f8f92bfb-5cef-412e-a6c1-4bd793c59fe7-metrics-certs\") pod \"network-metrics-daemon-hzlmg\" (UID: \"f8f92bfb-5cef-412e-a6c1-4bd793c59fe7\") " pod="openshift-multus/network-metrics-daemon-hzlmg" Oct 29 22:33:16.055136 ip-10-0-140-74 kubenswrapper[2577]: E1029 22:33:16.055014 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 29 22:33:16.055136 ip-10-0-140-74 kubenswrapper[2577]: E1029 22:33:16.055045 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 29 22:33:16.055136 ip-10-0-140-74 kubenswrapper[2577]: E1029 22:33:16.055082 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6f50ee57-a710-4325-bd45-4f861452ede4-original-pull-secret podName:6f50ee57-a710-4325-bd45-4f861452ede4 nodeName:}" failed. No retries permitted until 2026-10-29 22:33:32.055063458 +0000 UTC m=+34.212251138 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6f50ee57-a710-4325-bd45-4f861452ede4-original-pull-secret") pod "global-pull-secret-syncer-bp46g" (UID: "6f50ee57-a710-4325-bd45-4f861452ede4") : object "kube-system"/"original-pull-secret" not registered Oct 29 22:33:16.055136 ip-10-0-140-74 kubenswrapper[2577]: E1029 22:33:16.055110 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f8f92bfb-5cef-412e-a6c1-4bd793c59fe7-metrics-certs podName:f8f92bfb-5cef-412e-a6c1-4bd793c59fe7 nodeName:}" failed. No retries permitted until 2026-10-29 22:33:32.055100442 +0000 UTC m=+34.212288110 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f8f92bfb-5cef-412e-a6c1-4bd793c59fe7-metrics-certs") pod "network-metrics-daemon-hzlmg" (UID: "f8f92bfb-5cef-412e-a6c1-4bd793c59fe7") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 29 22:33:16.155502 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:16.155477 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9srv\" (UniqueName: \"kubernetes.io/projected/a4e1c52c-1eeb-4617-a0c9-d052521360d3-kube-api-access-l9srv\") pod \"network-check-target-7jv5g\" (UID: \"a4e1c52c-1eeb-4617-a0c9-d052521360d3\") " pod="openshift-network-diagnostics/network-check-target-7jv5g" Oct 29 22:33:16.155630 ip-10-0-140-74 kubenswrapper[2577]: E1029 22:33:16.155595 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 29 22:33:16.155630 ip-10-0-140-74 kubenswrapper[2577]: E1029 22:33:16.155607 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 29 22:33:16.155630 ip-10-0-140-74 kubenswrapper[2577]: E1029 22:33:16.155622 2577 projected.go:196] Error preparing data for projected volume kube-api-access-l9srv for pod openshift-network-diagnostics/network-check-target-7jv5g: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 29 22:33:16.155718 ip-10-0-140-74 kubenswrapper[2577]: E1029 22:33:16.155663 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a4e1c52c-1eeb-4617-a0c9-d052521360d3-kube-api-access-l9srv podName:a4e1c52c-1eeb-4617-a0c9-d052521360d3 nodeName:}" failed. No retries permitted until 2026-10-29 22:33:32.155651988 +0000 UTC m=+34.312839648 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-l9srv" (UniqueName: "kubernetes.io/projected/a4e1c52c-1eeb-4617-a0c9-d052521360d3-kube-api-access-l9srv") pod "network-check-target-7jv5g" (UID: "a4e1c52c-1eeb-4617-a0c9-d052521360d3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 29 22:33:16.160586 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:16.160555 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-54f9x" Oct 29 22:33:16.160703 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:16.160685 2577 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 29 22:33:16.161049 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:16.161038 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-54f9x" Oct 29 22:33:16.324588 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:16.324510 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hzlmg" Oct 29 22:33:16.324725 ip-10-0-140-74 kubenswrapper[2577]: E1029 22:33:16.324664 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hzlmg" podUID="f8f92bfb-5cef-412e-a6c1-4bd793c59fe7" Oct 29 22:33:16.324725 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:16.324705 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7jv5g" Oct 29 22:33:16.324837 ip-10-0-140-74 kubenswrapper[2577]: E1029 22:33:16.324777 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7jv5g" podUID="a4e1c52c-1eeb-4617-a0c9-d052521360d3" Oct 29 22:33:17.324714 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:17.324683 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bp46g" Oct 29 22:33:17.325153 ip-10-0-140-74 kubenswrapper[2577]: E1029 22:33:17.325124 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bp46g" podUID="6f50ee57-a710-4325-bd45-4f861452ede4" Oct 29 22:33:17.580087 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:17.580059 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-khlhx" event={"ID":"04195c1d-9fb7-479b-8061-f92e3c88aed2","Type":"ContainerStarted","Data":"67b842a34ebe9a6234f0da1835e74c9e7a5f9c0a8ea19ece28ca63145a893da5"} Oct 29 22:33:18.326182 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:18.326149 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hzlmg" Oct 29 22:33:18.326603 ip-10-0-140-74 kubenswrapper[2577]: E1029 22:33:18.326268 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hzlmg" podUID="f8f92bfb-5cef-412e-a6c1-4bd793c59fe7" Oct 29 22:33:18.326603 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:18.326315 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7jv5g" Oct 29 22:33:18.326603 ip-10-0-140-74 kubenswrapper[2577]: E1029 22:33:18.326407 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7jv5g" podUID="a4e1c52c-1eeb-4617-a0c9-d052521360d3" Oct 29 22:33:19.324766 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:19.324731 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bp46g" Oct 29 22:33:19.324905 ip-10-0-140-74 kubenswrapper[2577]: E1029 22:33:19.324833 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bp46g" podUID="6f50ee57-a710-4325-bd45-4f861452ede4" Oct 29 22:33:19.589553 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:19.589470 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-khlhx" event={"ID":"04195c1d-9fb7-479b-8061-f92e3c88aed2","Type":"ContainerStarted","Data":"fdafb8f58d087cf7668af4f6e5742918118fedc3274d30529ea3a226d141e028"} Oct 29 22:33:19.590349 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:19.589770 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-khlhx" Oct 29 22:33:19.590349 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:19.589823 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-khlhx" Oct 29 22:33:19.604152 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:19.604123 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-khlhx" Oct 29 22:33:19.666974 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:19.666932 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-khlhx" podStartSLOduration=7.466883882 podStartE2EDuration="21.666921853s" totalImagesPullingTime="14.200037971s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 22:32:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:33:19.666413903 +0000 UTC m=+21.823601585" watchObservedRunningTime="2026-10-29 22:33:19.666921853 +0000 UTC m=+21.824109533" Oct 29 22:33:20.324760 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:20.324733 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7jv5g" Oct 29 22:33:20.324924 ip-10-0-140-74 kubenswrapper[2577]: E1029 22:33:20.324863 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7jv5g" podUID="a4e1c52c-1eeb-4617-a0c9-d052521360d3" Oct 29 22:33:20.324924 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:20.324916 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hzlmg" Oct 29 22:33:20.325014 ip-10-0-140-74 kubenswrapper[2577]: E1029 22:33:20.324996 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hzlmg" podUID="f8f92bfb-5cef-412e-a6c1-4bd793c59fe7" Oct 29 22:33:20.591773 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:20.591699 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-khlhx" Oct 29 22:33:20.605753 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:20.605732 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-khlhx" Oct 29 22:33:21.212998 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:21.212778 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-bp46g"] Oct 29 22:33:21.213196 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:21.213103 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bp46g" Oct 29 22:33:21.213247 ip-10-0-140-74 kubenswrapper[2577]: E1029 22:33:21.213196 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bp46g" podUID="6f50ee57-a710-4325-bd45-4f861452ede4" Oct 29 22:33:21.217164 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:21.217140 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-7jv5g"] Oct 29 22:33:21.217286 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:21.217227 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7jv5g" Oct 29 22:33:21.217327 ip-10-0-140-74 kubenswrapper[2577]: E1029 22:33:21.217309 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7jv5g" podUID="a4e1c52c-1eeb-4617-a0c9-d052521360d3" Oct 29 22:33:21.219381 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:21.219349 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-hzlmg"] Oct 29 22:33:21.219495 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:21.219425 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hzlmg" Oct 29 22:33:21.219556 ip-10-0-140-74 kubenswrapper[2577]: E1029 22:33:21.219499 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hzlmg" podUID="f8f92bfb-5cef-412e-a6c1-4bd793c59fe7" Oct 29 22:33:23.325038 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:23.325008 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bp46g" Oct 29 22:33:23.325480 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:23.325122 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7jv5g" Oct 29 22:33:23.325480 ip-10-0-140-74 kubenswrapper[2577]: E1029 22:33:23.325134 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bp46g" podUID="6f50ee57-a710-4325-bd45-4f861452ede4" Oct 29 22:33:23.325480 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:23.325158 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hzlmg" Oct 29 22:33:23.325480 ip-10-0-140-74 kubenswrapper[2577]: E1029 22:33:23.325251 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7jv5g" podUID="a4e1c52c-1eeb-4617-a0c9-d052521360d3" Oct 29 22:33:23.325480 ip-10-0-140-74 kubenswrapper[2577]: E1029 22:33:23.325364 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hzlmg" podUID="f8f92bfb-5cef-412e-a6c1-4bd793c59fe7" Oct 29 22:33:25.325229 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:25.325192 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bp46g" Oct 29 22:33:25.325614 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:25.325314 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7jv5g" Oct 29 22:33:25.325614 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:25.325327 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hzlmg" Oct 29 22:33:25.325614 ip-10-0-140-74 kubenswrapper[2577]: E1029 22:33:25.325342 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bp46g" podUID="6f50ee57-a710-4325-bd45-4f861452ede4" Oct 29 22:33:25.325614 ip-10-0-140-74 kubenswrapper[2577]: E1029 22:33:25.325407 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7jv5g" podUID="a4e1c52c-1eeb-4617-a0c9-d052521360d3" Oct 29 22:33:25.325614 ip-10-0-140-74 kubenswrapper[2577]: E1029 22:33:25.325481 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hzlmg" podUID="f8f92bfb-5cef-412e-a6c1-4bd793c59fe7" Oct 29 22:33:27.324806 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.324785 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bp46g" Oct 29 22:33:27.325090 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.324804 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hzlmg" Oct 29 22:33:27.325090 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.324804 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7jv5g" Oct 29 22:33:27.325090 ip-10-0-140-74 kubenswrapper[2577]: E1029 22:33:27.324872 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bp46g" podUID="6f50ee57-a710-4325-bd45-4f861452ede4" Oct 29 22:33:27.325090 ip-10-0-140-74 kubenswrapper[2577]: E1029 22:33:27.325004 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hzlmg" podUID="f8f92bfb-5cef-412e-a6c1-4bd793c59fe7" Oct 29 22:33:27.325090 ip-10-0-140-74 kubenswrapper[2577]: E1029 22:33:27.325079 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7jv5g" podUID="a4e1c52c-1eeb-4617-a0c9-d052521360d3" Oct 29 22:33:27.572881 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.572807 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-74.ec2.internal" event="NodeReady" Oct 29 22:33:27.573031 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.572928 2577 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Oct 29 22:33:27.622365 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.622318 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-6rr4f"] Oct 29 22:33:27.627826 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.627796 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-5fjlz"] Oct 29 22:33:27.627959 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.627921 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-6rr4f" Oct 29 22:33:27.630555 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.630536 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-jgvr6"] Oct 29 22:33:27.630661 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.630629 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-5fjlz" Oct 29 22:33:27.632843 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.632827 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-7c4986d449-b7bwd"] Oct 29 22:33:27.632952 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.632936 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-jgvr6" Oct 29 22:33:27.636421 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.636403 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-5d8d7cc579-8cwjp"] Oct 29 22:33:27.636596 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.636574 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-b7bwd" Oct 29 22:33:27.639909 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.639890 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-78db97464f-6ljzh"] Oct 29 22:33:27.640034 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.640013 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-5d8d7cc579-8cwjp" Oct 29 22:33:27.642619 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.642599 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-mrbgh"] Oct 29 22:33:27.642727 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.642710 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-78db97464f-6ljzh" Oct 29 22:33:27.645095 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.645078 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-w5d4m"] Oct 29 22:33:27.645188 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.645176 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-mrbgh" Oct 29 22:33:27.646677 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.646658 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Oct 29 22:33:27.646832 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.646815 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Oct 29 22:33:27.646891 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.646816 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Oct 29 22:33:27.648023 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.648010 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Oct 29 22:33:27.648083 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.648042 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-8pd7h\"" Oct 29 22:33:27.648403 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.648387 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-p2qv4"] Oct 29 22:33:27.648494 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.648481 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-w5d4m" Oct 29 22:33:27.651023 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.651005 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-pk97l"] Oct 29 22:33:27.651114 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.651102 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-p2qv4" Oct 29 22:33:27.654933 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.654916 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-vj6md"] Oct 29 22:33:27.655121 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.655101 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-pk97l" Oct 29 22:33:27.658236 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.658215 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-84866f86bd-2v6d7"] Oct 29 22:33:27.658359 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.658345 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-vj6md" Oct 29 22:33:27.664046 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.664028 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-p9ds4"] Oct 29 22:33:27.667320 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.667121 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-84866f86bd-2v6d7" Oct 29 22:33:27.676817 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.674946 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-8c5hr"] Oct 29 22:33:27.676817 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.675168 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-p9ds4" Oct 29 22:33:27.680481 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.680436 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:33:27.681558 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.681539 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-sk5qq"] Oct 29 22:33:27.681678 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.681662 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-8c5hr" Oct 29 22:33:27.685373 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.684537 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-jjz47"] Oct 29 22:33:27.685373 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.684689 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-sk5qq" Oct 29 22:33:27.689593 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.687874 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-czfl9"] Oct 29 22:33:27.689593 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.688605 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-jjz47" Oct 29 22:33:27.691361 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.691325 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-jgvr6"] Oct 29 22:33:27.691447 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.691375 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-5fjlz"] Oct 29 22:33:27.691447 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.691397 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-6rr4f"] Oct 29 22:33:27.691447 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.691415 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-czfl9" Oct 29 22:33:27.691566 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.691417 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-mrbgh"] Oct 29 22:33:27.691626 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.691593 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-78db97464f-6ljzh"] Oct 29 22:33:27.691676 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.691630 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-5d8d7cc579-8cwjp"] Oct 29 22:33:27.691676 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.691651 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-p9ds4"] Oct 29 22:33:27.691676 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.691673 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-pk97l"] Oct 29 22:33:27.691796 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.691691 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-sk5qq"] Oct 29 22:33:27.691796 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.691711 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-vj6md"] Oct 29 22:33:27.691796 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.691735 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-jjz47"] Oct 29 22:33:27.691796 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.691758 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-p2qv4"] Oct 29 22:33:27.691796 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.691781 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-b7bwd"] Oct 29 22:33:27.691933 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.691801 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-czfl9"] Oct 29 22:33:27.691933 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.691819 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-w5d4m"] Oct 29 22:33:27.694937 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.694613 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-8c5hr"] Oct 29 22:33:27.700102 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.700083 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-84866f86bd-2v6d7"] Oct 29 22:33:27.746230 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.746211 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 29 22:33:27.746326 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.746257 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Oct 29 22:33:27.746542 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.746527 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Oct 29 22:33:27.746660 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.746647 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Oct 29 22:33:27.746718 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.746708 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Oct 29 22:33:27.746783 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.746774 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 29 22:33:27.746851 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.746805 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Oct 29 22:33:27.746903 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.746892 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Oct 29 22:33:27.746961 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.746932 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Oct 29 22:33:27.746961 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.746944 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Oct 29 22:33:27.747109 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.747079 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 29 22:33:27.747109 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.747073 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 29 22:33:27.747264 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.747131 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 29 22:33:27.747264 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.747166 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/989956e8-1893-4bbe-a4bf-6a2081509080-installation-pull-secrets\") pod \"image-registry-84866f86bd-2v6d7\" (UID: \"989956e8-1893-4bbe-a4bf-6a2081509080\") " pod="openshift-image-registry/image-registry-84866f86bd-2v6d7" Oct 29 22:33:27.747264 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.747192 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 29 22:33:27.747264 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.747207 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/71d80eec-b5e7-41d0-b58d-8a8d9a6c769c-default-certificate\") pod \"router-default-5d8d7cc579-8cwjp\" (UID: \"71d80eec-b5e7-41d0-b58d-8a8d9a6c769c\") " pod="openshift-ingress/router-default-5d8d7cc579-8cwjp" Oct 29 22:33:27.747264 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.747237 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mkxzs\" (UniqueName: \"kubernetes.io/projected/c635d035-053b-49aa-9559-79c31e8e49ff-kube-api-access-mkxzs\") pod \"service-ca-operator-544d749c8c-pk97l\" (UID: \"c635d035-053b-49aa-9559-79c31e8e49ff\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-pk97l" Oct 29 22:33:27.747531 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.747277 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fp2lh\" (UniqueName: \"kubernetes.io/projected/5296979a-7927-4ee2-84b9-a93780a2ad22-kube-api-access-fp2lh\") pod \"image-registry-78db97464f-6ljzh\" (UID: \"5296979a-7927-4ee2-84b9-a93780a2ad22\") " pod="openshift-image-registry/image-registry-78db97464f-6ljzh" Oct 29 22:33:27.747531 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.747291 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/989956e8-1893-4bbe-a4bf-6a2081509080-bound-sa-token\") pod \"image-registry-84866f86bd-2v6d7\" (UID: \"989956e8-1893-4bbe-a4bf-6a2081509080\") " pod="openshift-image-registry/image-registry-84866f86bd-2v6d7" Oct 29 22:33:27.747531 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.747360 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a53078b9-a6a8-423c-b852-34eececa7ab1-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-p2qv4\" (UID: \"a53078b9-a6a8-423c-b852-34eececa7ab1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-p2qv4" Oct 29 22:33:27.747531 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.747366 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Oct 29 22:33:27.747531 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.747416 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 29 22:33:27.747531 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.747447 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 29 22:33:27.747531 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.747491 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c635d035-053b-49aa-9559-79c31e8e49ff-config\") pod \"service-ca-operator-544d749c8c-pk97l\" (UID: \"c635d035-053b-49aa-9559-79c31e8e49ff\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-pk97l" Oct 29 22:33:27.747531 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.747521 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Oct 29 22:33:27.747531 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.747532 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5296979a-7927-4ee2-84b9-a93780a2ad22-registry-tls\") pod \"image-registry-78db97464f-6ljzh\" (UID: \"5296979a-7927-4ee2-84b9-a93780a2ad22\") " pod="openshift-image-registry/image-registry-78db97464f-6ljzh" Oct 29 22:33:27.747922 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.747572 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Oct 29 22:33:27.747922 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.747575 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Oct 29 22:33:27.747922 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.747658 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Oct 29 22:33:27.747922 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.747572 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5296979a-7927-4ee2-84b9-a93780a2ad22-bound-sa-token\") pod \"image-registry-78db97464f-6ljzh\" (UID: \"5296979a-7927-4ee2-84b9-a93780a2ad22\") " pod="openshift-image-registry/image-registry-78db97464f-6ljzh" Oct 29 22:33:27.747922 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.747684 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Oct 29 22:33:27.747922 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.747696 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Oct 29 22:33:27.747922 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.747704 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whjbn\" (UniqueName: \"kubernetes.io/projected/4ff372bf-82aa-4007-b74c-df0c779e5271-kube-api-access-whjbn\") pod \"kas-connection-checker-685895c7c9-mrbgh\" (UID: \"4ff372bf-82aa-4007-b74c-df0c779e5271\") " pod="kube-system/kas-connection-checker-685895c7c9-mrbgh" Oct 29 22:33:27.747922 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.747728 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/989956e8-1893-4bbe-a4bf-6a2081509080-ca-trust-extracted\") pod \"image-registry-84866f86bd-2v6d7\" (UID: \"989956e8-1893-4bbe-a4bf-6a2081509080\") " pod="openshift-image-registry/image-registry-84866f86bd-2v6d7" Oct 29 22:33:27.747922 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.747673 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Oct 29 22:33:27.747922 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.747746 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/71d80eec-b5e7-41d0-b58d-8a8d9a6c769c-stats-auth\") pod \"router-default-5d8d7cc579-8cwjp\" (UID: \"71d80eec-b5e7-41d0-b58d-8a8d9a6c769c\") " pod="openshift-ingress/router-default-5d8d7cc579-8cwjp" Oct 29 22:33:27.747922 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.747764 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/aa6a190e-3283-458a-9dfd-2de7cb193fc5-config\") pod \"console-operator-7cd56b8dbc-vj6md\" (UID: \"aa6a190e-3283-458a-9dfd-2de7cb193fc5\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-vj6md" Oct 29 22:33:27.747922 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.747776 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Oct 29 22:33:27.747922 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.747811 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Oct 29 22:33:27.747922 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.747811 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7h8fj\" (UniqueName: \"kubernetes.io/projected/c54ef754-c91e-4bc5-8e83-08523aa09d37-kube-api-access-7h8fj\") pod \"volume-data-source-validator-58c9f5f46d-w5d4m\" (UID: \"c54ef754-c91e-4bc5-8e83-08523aa09d37\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-w5d4m" Oct 29 22:33:27.747922 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.747815 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Oct 29 22:33:27.747922 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.747817 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Oct 29 22:33:27.747922 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.747843 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 29 22:33:27.747922 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.747856 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 29 22:33:27.747922 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.747865 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/65c96dfb-06e9-4bea-bb83-f63b4c62d860-snapshots\") pod \"insights-operator-7c4986d449-b7bwd\" (UID: \"65c96dfb-06e9-4bea-bb83-f63b4c62d860\") " pod="openshift-insights/insights-operator-7c4986d449-b7bwd" Oct 29 22:33:27.747922 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.747877 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 29 22:33:27.747922 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.747880 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 29 22:33:27.747922 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.747891 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-gg4pb\"" Oct 29 22:33:27.747922 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.747897 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 29 22:33:27.747922 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.747903 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Oct 29 22:33:27.747922 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.747912 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Oct 29 22:33:27.747922 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.747916 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjwwn\" (UniqueName: \"kubernetes.io/projected/587893f5-e85a-46a9-829f-545a42c10056-kube-api-access-bjwwn\") pod \"kas-connection-checker-685895c7c9-jgvr6\" (UID: \"587893f5-e85a-46a9-829f-545a42c10056\") " pod="kube-system/kas-connection-checker-685895c7c9-jgvr6" Oct 29 22:33:27.748680 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.747943 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-z2g4l\"" Oct 29 22:33:27.748680 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.747948 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/71d80eec-b5e7-41d0-b58d-8a8d9a6c769c-service-ca-bundle\") pod \"router-default-5d8d7cc579-8cwjp\" (UID: \"71d80eec-b5e7-41d0-b58d-8a8d9a6c769c\") " pod="openshift-ingress/router-default-5d8d7cc579-8cwjp" Oct 29 22:33:27.748680 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.747964 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4xwz\" (UniqueName: \"kubernetes.io/projected/aa6a190e-3283-458a-9dfd-2de7cb193fc5-kube-api-access-c4xwz\") pod \"console-operator-7cd56b8dbc-vj6md\" (UID: \"aa6a190e-3283-458a-9dfd-2de7cb193fc5\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-vj6md" Oct 29 22:33:27.748680 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.748015 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/989956e8-1893-4bbe-a4bf-6a2081509080-registry-certificates\") pod \"image-registry-84866f86bd-2v6d7\" (UID: \"989956e8-1893-4bbe-a4bf-6a2081509080\") " pod="openshift-image-registry/image-registry-84866f86bd-2v6d7" Oct 29 22:33:27.748680 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.748104 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5296979a-7927-4ee2-84b9-a93780a2ad22-registry-certificates\") pod \"image-registry-78db97464f-6ljzh\" (UID: \"5296979a-7927-4ee2-84b9-a93780a2ad22\") " pod="openshift-image-registry/image-registry-78db97464f-6ljzh" Oct 29 22:33:27.748680 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.748157 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/aa6a190e-3283-458a-9dfd-2de7cb193fc5-trusted-ca\") pod \"console-operator-7cd56b8dbc-vj6md\" (UID: \"aa6a190e-3283-458a-9dfd-2de7cb193fc5\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-vj6md" Oct 29 22:33:27.748680 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.748176 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/65c96dfb-06e9-4bea-bb83-f63b4c62d860-tmp\") pod \"insights-operator-7c4986d449-b7bwd\" (UID: \"65c96dfb-06e9-4bea-bb83-f63b4c62d860\") " pod="openshift-insights/insights-operator-7c4986d449-b7bwd" Oct 29 22:33:27.748680 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.748192 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gfk2t\" (UniqueName: \"kubernetes.io/projected/989956e8-1893-4bbe-a4bf-6a2081509080-kube-api-access-gfk2t\") pod \"image-registry-84866f86bd-2v6d7\" (UID: \"989956e8-1893-4bbe-a4bf-6a2081509080\") " pod="openshift-image-registry/image-registry-84866f86bd-2v6d7" Oct 29 22:33:27.748680 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.748213 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-698ds\" (UniqueName: \"kubernetes.io/projected/a53078b9-a6a8-423c-b852-34eececa7ab1-kube-api-access-698ds\") pod \"cluster-samples-operator-6ddb6c8b94-p2qv4\" (UID: \"a53078b9-a6a8-423c-b852-34eececa7ab1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-p2qv4" Oct 29 22:33:27.748680 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.748278 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/65c96dfb-06e9-4bea-bb83-f63b4c62d860-serving-cert\") pod \"insights-operator-7c4986d449-b7bwd\" (UID: \"65c96dfb-06e9-4bea-bb83-f63b4c62d860\") " pod="openshift-insights/insights-operator-7c4986d449-b7bwd" Oct 29 22:33:27.748680 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.748301 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/2accd775-2173-4bfe-a413-7d243bd0820f-nginx-conf\") pod \"networking-console-plugin-86c478bb88-6rr4f\" (UID: \"2accd775-2173-4bfe-a413-7d243bd0820f\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-6rr4f" Oct 29 22:33:27.748680 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.748322 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/aa6a190e-3283-458a-9dfd-2de7cb193fc5-serving-cert\") pod \"console-operator-7cd56b8dbc-vj6md\" (UID: \"aa6a190e-3283-458a-9dfd-2de7cb193fc5\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-vj6md" Oct 29 22:33:27.748680 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.748384 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/989956e8-1893-4bbe-a4bf-6a2081509080-image-registry-private-configuration\") pod \"image-registry-84866f86bd-2v6d7\" (UID: \"989956e8-1893-4bbe-a4bf-6a2081509080\") " pod="openshift-image-registry/image-registry-84866f86bd-2v6d7" Oct 29 22:33:27.748680 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.748447 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/a53078b9-a6a8-423c-b852-34eececa7ab1-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-p2qv4\" (UID: \"a53078b9-a6a8-423c-b852-34eececa7ab1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-p2qv4" Oct 29 22:33:27.748680 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.748485 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5296979a-7927-4ee2-84b9-a93780a2ad22-installation-pull-secrets\") pod \"image-registry-78db97464f-6ljzh\" (UID: \"5296979a-7927-4ee2-84b9-a93780a2ad22\") " pod="openshift-image-registry/image-registry-78db97464f-6ljzh" Oct 29 22:33:27.749080 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.748521 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2accd775-2173-4bfe-a413-7d243bd0820f-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-6rr4f\" (UID: \"2accd775-2173-4bfe-a413-7d243bd0820f\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-6rr4f" Oct 29 22:33:27.749080 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.748628 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/65c96dfb-06e9-4bea-bb83-f63b4c62d860-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-b7bwd\" (UID: \"65c96dfb-06e9-4bea-bb83-f63b4c62d860\") " pod="openshift-insights/insights-operator-7c4986d449-b7bwd" Oct 29 22:33:27.749080 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.748658 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5296979a-7927-4ee2-84b9-a93780a2ad22-trusted-ca\") pod \"image-registry-78db97464f-6ljzh\" (UID: \"5296979a-7927-4ee2-84b9-a93780a2ad22\") " pod="openshift-image-registry/image-registry-78db97464f-6ljzh" Oct 29 22:33:27.749080 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.748674 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/65c96dfb-06e9-4bea-bb83-f63b4c62d860-service-ca-bundle\") pod \"insights-operator-7c4986d449-b7bwd\" (UID: \"65c96dfb-06e9-4bea-bb83-f63b4c62d860\") " pod="openshift-insights/insights-operator-7c4986d449-b7bwd" Oct 29 22:33:27.749080 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.748690 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2tdnl\" (UniqueName: \"kubernetes.io/projected/71d80eec-b5e7-41d0-b58d-8a8d9a6c769c-kube-api-access-2tdnl\") pod \"router-default-5d8d7cc579-8cwjp\" (UID: \"71d80eec-b5e7-41d0-b58d-8a8d9a6c769c\") " pod="openshift-ingress/router-default-5d8d7cc579-8cwjp" Oct 29 22:33:27.749080 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.748733 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71d80eec-b5e7-41d0-b58d-8a8d9a6c769c-metrics-certs\") pod \"router-default-5d8d7cc579-8cwjp\" (UID: \"71d80eec-b5e7-41d0-b58d-8a8d9a6c769c\") " pod="openshift-ingress/router-default-5d8d7cc579-8cwjp" Oct 29 22:33:27.749080 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.748779 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/989956e8-1893-4bbe-a4bf-6a2081509080-registry-tls\") pod \"image-registry-84866f86bd-2v6d7\" (UID: \"989956e8-1893-4bbe-a4bf-6a2081509080\") " pod="openshift-image-registry/image-registry-84866f86bd-2v6d7" Oct 29 22:33:27.749080 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.748804 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/989956e8-1893-4bbe-a4bf-6a2081509080-trusted-ca\") pod \"image-registry-84866f86bd-2v6d7\" (UID: \"989956e8-1893-4bbe-a4bf-6a2081509080\") " pod="openshift-image-registry/image-registry-84866f86bd-2v6d7" Oct 29 22:33:27.749080 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.748847 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2tw5\" (UniqueName: \"kubernetes.io/projected/65c96dfb-06e9-4bea-bb83-f63b4c62d860-kube-api-access-z2tw5\") pod \"insights-operator-7c4986d449-b7bwd\" (UID: \"65c96dfb-06e9-4bea-bb83-f63b4c62d860\") " pod="openshift-insights/insights-operator-7c4986d449-b7bwd" Oct 29 22:33:27.749080 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.748887 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c635d035-053b-49aa-9559-79c31e8e49ff-serving-cert\") pod \"service-ca-operator-544d749c8c-pk97l\" (UID: \"c635d035-053b-49aa-9559-79c31e8e49ff\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-pk97l" Oct 29 22:33:27.749080 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.748904 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5296979a-7927-4ee2-84b9-a93780a2ad22-image-registry-private-configuration\") pod \"image-registry-78db97464f-6ljzh\" (UID: \"5296979a-7927-4ee2-84b9-a93780a2ad22\") " pod="openshift-image-registry/image-registry-78db97464f-6ljzh" Oct 29 22:33:27.749080 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.748922 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5296979a-7927-4ee2-84b9-a93780a2ad22-ca-trust-extracted\") pod \"image-registry-78db97464f-6ljzh\" (UID: \"5296979a-7927-4ee2-84b9-a93780a2ad22\") " pod="openshift-image-registry/image-registry-78db97464f-6ljzh" Oct 29 22:33:27.749080 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.748981 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vv2vp\" (UniqueName: \"kubernetes.io/projected/79ad3230-163e-447d-9f04-a072479ff957-kube-api-access-vv2vp\") pod \"kas-connection-checker-685895c7c9-5fjlz\" (UID: \"79ad3230-163e-447d-9f04-a072479ff957\") " pod="kube-system/kas-connection-checker-685895c7c9-5fjlz" Oct 29 22:33:27.756807 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.756775 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Oct 29 22:33:27.756807 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.756796 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Oct 29 22:33:27.757155 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.757140 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-6b58k\"" Oct 29 22:33:27.757210 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.757153 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Oct 29 22:33:27.757210 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.757173 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Oct 29 22:33:27.757210 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.757177 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-647gd\"" Oct 29 22:33:27.757210 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.757189 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Oct 29 22:33:27.757210 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.757208 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-5rnmj\"" Oct 29 22:33:27.757552 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.757535 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-sllfs\"" Oct 29 22:33:27.757628 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.757562 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-xzwzj\"" Oct 29 22:33:27.757628 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.757617 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-s24md\"" Oct 29 22:33:27.757715 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.757678 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-nr2md\"" Oct 29 22:33:27.757715 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.757685 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-7zghq\"" Oct 29 22:33:27.757715 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.757695 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Oct 29 22:33:27.757797 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.757724 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-wwqh2\"" Oct 29 22:33:27.759693 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.758675 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Oct 29 22:33:27.763323 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.763300 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-6b9sj\"" Oct 29 22:33:27.768274 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.768258 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Oct 29 22:33:27.768662 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.768646 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Oct 29 22:33:27.768710 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.768648 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Oct 29 22:33:27.849638 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.849571 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8d5d\" (UniqueName: \"kubernetes.io/projected/dd4e19ce-ef37-44e5-8b57-9dafc590685c-kube-api-access-l8d5d\") pod \"network-check-source-6f78d59f99-p9ds4\" (UID: \"dd4e19ce-ef37-44e5-8b57-9dafc590685c\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-p9ds4" Oct 29 22:33:27.849771 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.849649 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c635d035-053b-49aa-9559-79c31e8e49ff-config\") pod \"service-ca-operator-544d749c8c-pk97l\" (UID: \"c635d035-053b-49aa-9559-79c31e8e49ff\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-pk97l" Oct 29 22:33:27.849771 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.849679 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5296979a-7927-4ee2-84b9-a93780a2ad22-registry-tls\") pod \"image-registry-78db97464f-6ljzh\" (UID: \"5296979a-7927-4ee2-84b9-a93780a2ad22\") " pod="openshift-image-registry/image-registry-78db97464f-6ljzh" Oct 29 22:33:27.849771 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.849695 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5296979a-7927-4ee2-84b9-a93780a2ad22-bound-sa-token\") pod \"image-registry-78db97464f-6ljzh\" (UID: \"5296979a-7927-4ee2-84b9-a93780a2ad22\") " pod="openshift-image-registry/image-registry-78db97464f-6ljzh" Oct 29 22:33:27.849771 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.849713 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whjbn\" (UniqueName: \"kubernetes.io/projected/4ff372bf-82aa-4007-b74c-df0c779e5271-kube-api-access-whjbn\") pod \"kas-connection-checker-685895c7c9-mrbgh\" (UID: \"4ff372bf-82aa-4007-b74c-df0c779e5271\") " pod="kube-system/kas-connection-checker-685895c7c9-mrbgh" Oct 29 22:33:27.849771 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.849738 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6sv4r\" (UniqueName: \"kubernetes.io/projected/641f9ec1-c8e2-4774-9d4f-6a7a2da97b6f-kube-api-access-6sv4r\") pod \"cluster-monitoring-operator-884576c57-8c5hr\" (UID: \"641f9ec1-c8e2-4774-9d4f-6a7a2da97b6f\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-8c5hr" Oct 29 22:33:27.849771 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.849767 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/989956e8-1893-4bbe-a4bf-6a2081509080-ca-trust-extracted\") pod \"image-registry-84866f86bd-2v6d7\" (UID: \"989956e8-1893-4bbe-a4bf-6a2081509080\") " pod="openshift-image-registry/image-registry-84866f86bd-2v6d7" Oct 29 22:33:27.849995 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.849790 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6b34b3b1-84af-45f3-92ee-1ca720e8b263-metrics-tls\") pod \"dns-default-jjz47\" (UID: \"6b34b3b1-84af-45f3-92ee-1ca720e8b263\") " pod="openshift-dns/dns-default-jjz47" Oct 29 22:33:27.849995 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.849817 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/71d80eec-b5e7-41d0-b58d-8a8d9a6c769c-stats-auth\") pod \"router-default-5d8d7cc579-8cwjp\" (UID: \"71d80eec-b5e7-41d0-b58d-8a8d9a6c769c\") " pod="openshift-ingress/router-default-5d8d7cc579-8cwjp" Oct 29 22:33:27.849995 ip-10-0-140-74 kubenswrapper[2577]: E1029 22:33:27.849815 2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 29 22:33:27.849995 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.849843 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/aa6a190e-3283-458a-9dfd-2de7cb193fc5-config\") pod \"console-operator-7cd56b8dbc-vj6md\" (UID: \"aa6a190e-3283-458a-9dfd-2de7cb193fc5\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-vj6md" Oct 29 22:33:27.849995 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.849872 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7h8fj\" (UniqueName: \"kubernetes.io/projected/c54ef754-c91e-4bc5-8e83-08523aa09d37-kube-api-access-7h8fj\") pod \"volume-data-source-validator-58c9f5f46d-w5d4m\" (UID: \"c54ef754-c91e-4bc5-8e83-08523aa09d37\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-w5d4m" Oct 29 22:33:27.849995 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.849896 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4071b0c7-c4f3-419a-8d6e-6acc144fc9a4-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-sk5qq\" (UID: \"4071b0c7-c4f3-419a-8d6e-6acc144fc9a4\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-sk5qq" Oct 29 22:33:27.849995 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.849922 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/65c96dfb-06e9-4bea-bb83-f63b4c62d860-snapshots\") pod \"insights-operator-7c4986d449-b7bwd\" (UID: \"65c96dfb-06e9-4bea-bb83-f63b4c62d860\") " pod="openshift-insights/insights-operator-7c4986d449-b7bwd" Oct 29 22:33:27.849995 ip-10-0-140-74 kubenswrapper[2577]: E1029 22:33:27.849849 2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-78db97464f-6ljzh: secret "image-registry-tls" not found Oct 29 22:33:27.850365 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.850101 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjwwn\" (UniqueName: \"kubernetes.io/projected/587893f5-e85a-46a9-829f-545a42c10056-kube-api-access-bjwwn\") pod \"kas-connection-checker-685895c7c9-jgvr6\" (UID: \"587893f5-e85a-46a9-829f-545a42c10056\") " pod="kube-system/kas-connection-checker-685895c7c9-jgvr6" Oct 29 22:33:27.850365 ip-10-0-140-74 kubenswrapper[2577]: E1029 22:33:27.850187 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5296979a-7927-4ee2-84b9-a93780a2ad22-registry-tls podName:5296979a-7927-4ee2-84b9-a93780a2ad22 nodeName:}" failed. No retries permitted until 2026-10-29 22:33:28.350167453 +0000 UTC m=+30.507355128 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5296979a-7927-4ee2-84b9-a93780a2ad22-registry-tls") pod "image-registry-78db97464f-6ljzh" (UID: "5296979a-7927-4ee2-84b9-a93780a2ad22") : secret "image-registry-tls" not found Oct 29 22:33:27.850365 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.850261 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/71d80eec-b5e7-41d0-b58d-8a8d9a6c769c-service-ca-bundle\") pod \"router-default-5d8d7cc579-8cwjp\" (UID: \"71d80eec-b5e7-41d0-b58d-8a8d9a6c769c\") " pod="openshift-ingress/router-default-5d8d7cc579-8cwjp" Oct 29 22:33:27.850365 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.850345 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c635d035-053b-49aa-9559-79c31e8e49ff-config\") pod \"service-ca-operator-544d749c8c-pk97l\" (UID: \"c635d035-053b-49aa-9559-79c31e8e49ff\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-pk97l" Oct 29 22:33:27.850577 ip-10-0-140-74 kubenswrapper[2577]: E1029 22:33:27.850375 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/71d80eec-b5e7-41d0-b58d-8a8d9a6c769c-service-ca-bundle podName:71d80eec-b5e7-41d0-b58d-8a8d9a6c769c nodeName:}" failed. No retries permitted until 2026-10-29 22:33:28.350360145 +0000 UTC m=+30.507547823 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/71d80eec-b5e7-41d0-b58d-8a8d9a6c769c-service-ca-bundle") pod "router-default-5d8d7cc579-8cwjp" (UID: "71d80eec-b5e7-41d0-b58d-8a8d9a6c769c") : configmap references non-existent config key: service-ca.crt Oct 29 22:33:27.850577 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.850397 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4xwz\" (UniqueName: \"kubernetes.io/projected/aa6a190e-3283-458a-9dfd-2de7cb193fc5-kube-api-access-c4xwz\") pod \"console-operator-7cd56b8dbc-vj6md\" (UID: \"aa6a190e-3283-458a-9dfd-2de7cb193fc5\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-vj6md" Oct 29 22:33:27.850577 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.850429 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/989956e8-1893-4bbe-a4bf-6a2081509080-registry-certificates\") pod \"image-registry-84866f86bd-2v6d7\" (UID: \"989956e8-1893-4bbe-a4bf-6a2081509080\") " pod="openshift-image-registry/image-registry-84866f86bd-2v6d7" Oct 29 22:33:27.850577 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.850460 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5296979a-7927-4ee2-84b9-a93780a2ad22-registry-certificates\") pod \"image-registry-78db97464f-6ljzh\" (UID: \"5296979a-7927-4ee2-84b9-a93780a2ad22\") " pod="openshift-image-registry/image-registry-78db97464f-6ljzh" Oct 29 22:33:27.850577 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.850489 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/aa6a190e-3283-458a-9dfd-2de7cb193fc5-trusted-ca\") pod \"console-operator-7cd56b8dbc-vj6md\" (UID: \"aa6a190e-3283-458a-9dfd-2de7cb193fc5\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-vj6md" Oct 29 22:33:27.850827 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.850698 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/989956e8-1893-4bbe-a4bf-6a2081509080-ca-trust-extracted\") pod \"image-registry-84866f86bd-2v6d7\" (UID: \"989956e8-1893-4bbe-a4bf-6a2081509080\") " pod="openshift-image-registry/image-registry-84866f86bd-2v6d7" Oct 29 22:33:27.850827 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.850740 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/65c96dfb-06e9-4bea-bb83-f63b4c62d860-tmp\") pod \"insights-operator-7c4986d449-b7bwd\" (UID: \"65c96dfb-06e9-4bea-bb83-f63b4c62d860\") " pod="openshift-insights/insights-operator-7c4986d449-b7bwd" Oct 29 22:33:27.850827 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.850765 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfk2t\" (UniqueName: \"kubernetes.io/projected/989956e8-1893-4bbe-a4bf-6a2081509080-kube-api-access-gfk2t\") pod \"image-registry-84866f86bd-2v6d7\" (UID: \"989956e8-1893-4bbe-a4bf-6a2081509080\") " pod="openshift-image-registry/image-registry-84866f86bd-2v6d7" Oct 29 22:33:27.850827 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.850790 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8gw9q\" (UniqueName: \"kubernetes.io/projected/6b34b3b1-84af-45f3-92ee-1ca720e8b263-kube-api-access-8gw9q\") pod \"dns-default-jjz47\" (UID: \"6b34b3b1-84af-45f3-92ee-1ca720e8b263\") " pod="openshift-dns/dns-default-jjz47" Oct 29 22:33:27.850827 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.850816 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-698ds\" (UniqueName: \"kubernetes.io/projected/a53078b9-a6a8-423c-b852-34eececa7ab1-kube-api-access-698ds\") pod \"cluster-samples-operator-6ddb6c8b94-p2qv4\" (UID: \"a53078b9-a6a8-423c-b852-34eececa7ab1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-p2qv4" Oct 29 22:33:27.851073 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.850841 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/641f9ec1-c8e2-4774-9d4f-6a7a2da97b6f-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-8c5hr\" (UID: \"641f9ec1-c8e2-4774-9d4f-6a7a2da97b6f\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-8c5hr" Oct 29 22:33:27.851073 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.850875 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/65c96dfb-06e9-4bea-bb83-f63b4c62d860-serving-cert\") pod \"insights-operator-7c4986d449-b7bwd\" (UID: \"65c96dfb-06e9-4bea-bb83-f63b4c62d860\") " pod="openshift-insights/insights-operator-7c4986d449-b7bwd" Oct 29 22:33:27.851073 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.850898 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/2accd775-2173-4bfe-a413-7d243bd0820f-nginx-conf\") pod \"networking-console-plugin-86c478bb88-6rr4f\" (UID: \"2accd775-2173-4bfe-a413-7d243bd0820f\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-6rr4f" Oct 29 22:33:27.851073 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.850922 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/aa6a190e-3283-458a-9dfd-2de7cb193fc5-serving-cert\") pod \"console-operator-7cd56b8dbc-vj6md\" (UID: \"aa6a190e-3283-458a-9dfd-2de7cb193fc5\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-vj6md" Oct 29 22:33:27.851073 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.850947 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/989956e8-1893-4bbe-a4bf-6a2081509080-image-registry-private-configuration\") pod \"image-registry-84866f86bd-2v6d7\" (UID: \"989956e8-1893-4bbe-a4bf-6a2081509080\") " pod="openshift-image-registry/image-registry-84866f86bd-2v6d7" Oct 29 22:33:27.851073 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.850963 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/65c96dfb-06e9-4bea-bb83-f63b4c62d860-snapshots\") pod \"insights-operator-7c4986d449-b7bwd\" (UID: \"65c96dfb-06e9-4bea-bb83-f63b4c62d860\") " pod="openshift-insights/insights-operator-7c4986d449-b7bwd" Oct 29 22:33:27.851073 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.850974 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/a53078b9-a6a8-423c-b852-34eececa7ab1-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-p2qv4\" (UID: \"a53078b9-a6a8-423c-b852-34eececa7ab1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-p2qv4" Oct 29 22:33:27.851073 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.851006 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/6b34b3b1-84af-45f3-92ee-1ca720e8b263-config-volume\") pod \"dns-default-jjz47\" (UID: \"6b34b3b1-84af-45f3-92ee-1ca720e8b263\") " pod="openshift-dns/dns-default-jjz47" Oct 29 22:33:27.851073 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.851035 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5296979a-7927-4ee2-84b9-a93780a2ad22-installation-pull-secrets\") pod \"image-registry-78db97464f-6ljzh\" (UID: \"5296979a-7927-4ee2-84b9-a93780a2ad22\") " pod="openshift-image-registry/image-registry-78db97464f-6ljzh" Oct 29 22:33:27.851073 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.851047 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/989956e8-1893-4bbe-a4bf-6a2081509080-registry-certificates\") pod \"image-registry-84866f86bd-2v6d7\" (UID: \"989956e8-1893-4bbe-a4bf-6a2081509080\") " pod="openshift-image-registry/image-registry-84866f86bd-2v6d7" Oct 29 22:33:27.851073 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.851054 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5296979a-7927-4ee2-84b9-a93780a2ad22-registry-certificates\") pod \"image-registry-78db97464f-6ljzh\" (UID: \"5296979a-7927-4ee2-84b9-a93780a2ad22\") " pod="openshift-image-registry/image-registry-78db97464f-6ljzh" Oct 29 22:33:27.851073 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.851067 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/aa6a190e-3283-458a-9dfd-2de7cb193fc5-config\") pod \"console-operator-7cd56b8dbc-vj6md\" (UID: \"aa6a190e-3283-458a-9dfd-2de7cb193fc5\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-vj6md" Oct 29 22:33:27.851624 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.851136 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2accd775-2173-4bfe-a413-7d243bd0820f-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-6rr4f\" (UID: \"2accd775-2173-4bfe-a413-7d243bd0820f\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-6rr4f" Oct 29 22:33:27.851624 ip-10-0-140-74 kubenswrapper[2577]: E1029 22:33:27.851354 2577 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 29 22:33:27.851624 ip-10-0-140-74 kubenswrapper[2577]: E1029 22:33:27.851442 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2accd775-2173-4bfe-a413-7d243bd0820f-networking-console-plugin-cert podName:2accd775-2173-4bfe-a413-7d243bd0820f nodeName:}" failed. No retries permitted until 2026-10-29 22:33:28.351424974 +0000 UTC m=+30.508612639 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/2accd775-2173-4bfe-a413-7d243bd0820f-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-6rr4f" (UID: "2accd775-2173-4bfe-a413-7d243bd0820f") : secret "networking-console-plugin-cert" not found Oct 29 22:33:27.851624 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.851462 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/aa6a190e-3283-458a-9dfd-2de7cb193fc5-trusted-ca\") pod \"console-operator-7cd56b8dbc-vj6md\" (UID: \"aa6a190e-3283-458a-9dfd-2de7cb193fc5\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-vj6md" Oct 29 22:33:27.851624 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.851510 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/65c96dfb-06e9-4bea-bb83-f63b4c62d860-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-b7bwd\" (UID: \"65c96dfb-06e9-4bea-bb83-f63b4c62d860\") " pod="openshift-insights/insights-operator-7c4986d449-b7bwd" Oct 29 22:33:27.851878 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.851654 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/a53078b9-a6a8-423c-b852-34eececa7ab1-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-p2qv4\" (UID: \"a53078b9-a6a8-423c-b852-34eececa7ab1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-p2qv4" Oct 29 22:33:27.851878 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.851743 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/65c96dfb-06e9-4bea-bb83-f63b4c62d860-tmp\") pod \"insights-operator-7c4986d449-b7bwd\" (UID: \"65c96dfb-06e9-4bea-bb83-f63b4c62d860\") " pod="openshift-insights/insights-operator-7c4986d449-b7bwd" Oct 29 22:33:27.851878 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.851794 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5296979a-7927-4ee2-84b9-a93780a2ad22-trusted-ca\") pod \"image-registry-78db97464f-6ljzh\" (UID: \"5296979a-7927-4ee2-84b9-a93780a2ad22\") " pod="openshift-image-registry/image-registry-78db97464f-6ljzh" Oct 29 22:33:27.851878 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.851831 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b34b3b1-84af-45f3-92ee-1ca720e8b263-tmp-dir\") pod \"dns-default-jjz47\" (UID: \"6b34b3b1-84af-45f3-92ee-1ca720e8b263\") " pod="openshift-dns/dns-default-jjz47" Oct 29 22:33:27.852068 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.851939 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/2accd775-2173-4bfe-a413-7d243bd0820f-nginx-conf\") pod \"networking-console-plugin-86c478bb88-6rr4f\" (UID: \"2accd775-2173-4bfe-a413-7d243bd0820f\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-6rr4f" Oct 29 22:33:27.852068 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.852002 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/65c96dfb-06e9-4bea-bb83-f63b4c62d860-service-ca-bundle\") pod \"insights-operator-7c4986d449-b7bwd\" (UID: \"65c96dfb-06e9-4bea-bb83-f63b4c62d860\") " pod="openshift-insights/insights-operator-7c4986d449-b7bwd" Oct 29 22:33:27.852068 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.852044 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tdnl\" (UniqueName: \"kubernetes.io/projected/71d80eec-b5e7-41d0-b58d-8a8d9a6c769c-kube-api-access-2tdnl\") pod \"router-default-5d8d7cc579-8cwjp\" (UID: \"71d80eec-b5e7-41d0-b58d-8a8d9a6c769c\") " pod="openshift-ingress/router-default-5d8d7cc579-8cwjp" Oct 29 22:33:27.852297 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.852275 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71d80eec-b5e7-41d0-b58d-8a8d9a6c769c-metrics-certs\") pod \"router-default-5d8d7cc579-8cwjp\" (UID: \"71d80eec-b5e7-41d0-b58d-8a8d9a6c769c\") " pod="openshift-ingress/router-default-5d8d7cc579-8cwjp" Oct 29 22:33:27.852376 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.852313 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/989956e8-1893-4bbe-a4bf-6a2081509080-registry-tls\") pod \"image-registry-84866f86bd-2v6d7\" (UID: \"989956e8-1893-4bbe-a4bf-6a2081509080\") " pod="openshift-image-registry/image-registry-84866f86bd-2v6d7" Oct 29 22:33:27.852376 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.852355 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/989956e8-1893-4bbe-a4bf-6a2081509080-trusted-ca\") pod \"image-registry-84866f86bd-2v6d7\" (UID: \"989956e8-1893-4bbe-a4bf-6a2081509080\") " pod="openshift-image-registry/image-registry-84866f86bd-2v6d7" Oct 29 22:33:27.852486 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.852382 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4071b0c7-c4f3-419a-8d6e-6acc144fc9a4-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-sk5qq\" (UID: \"4071b0c7-c4f3-419a-8d6e-6acc144fc9a4\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-sk5qq" Oct 29 22:33:27.852486 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.852408 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2tw5\" (UniqueName: \"kubernetes.io/projected/65c96dfb-06e9-4bea-bb83-f63b4c62d860-kube-api-access-z2tw5\") pod \"insights-operator-7c4986d449-b7bwd\" (UID: \"65c96dfb-06e9-4bea-bb83-f63b4c62d860\") " pod="openshift-insights/insights-operator-7c4986d449-b7bwd" Oct 29 22:33:27.852486 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.852431 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8406bb9f-3a4e-42f3-a822-f754d9b8e7a8-cert\") pod \"ingress-canary-czfl9\" (UID: \"8406bb9f-3a4e-42f3-a822-f754d9b8e7a8\") " pod="openshift-ingress-canary/ingress-canary-czfl9" Oct 29 22:33:27.852486 ip-10-0-140-74 kubenswrapper[2577]: E1029 22:33:27.852449 2577 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 29 22:33:27.852486 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.852460 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c635d035-053b-49aa-9559-79c31e8e49ff-serving-cert\") pod \"service-ca-operator-544d749c8c-pk97l\" (UID: \"c635d035-053b-49aa-9559-79c31e8e49ff\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-pk97l" Oct 29 22:33:27.852727 ip-10-0-140-74 kubenswrapper[2577]: E1029 22:33:27.852500 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71d80eec-b5e7-41d0-b58d-8a8d9a6c769c-metrics-certs podName:71d80eec-b5e7-41d0-b58d-8a8d9a6c769c nodeName:}" failed. No retries permitted until 2026-10-29 22:33:28.352484952 +0000 UTC m=+30.509672627 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/71d80eec-b5e7-41d0-b58d-8a8d9a6c769c-metrics-certs") pod "router-default-5d8d7cc579-8cwjp" (UID: "71d80eec-b5e7-41d0-b58d-8a8d9a6c769c") : secret "router-metrics-certs-default" not found Oct 29 22:33:27.852727 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.852523 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5296979a-7927-4ee2-84b9-a93780a2ad22-image-registry-private-configuration\") pod \"image-registry-78db97464f-6ljzh\" (UID: \"5296979a-7927-4ee2-84b9-a93780a2ad22\") " pod="openshift-image-registry/image-registry-78db97464f-6ljzh" Oct 29 22:33:27.852727 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.852552 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5296979a-7927-4ee2-84b9-a93780a2ad22-ca-trust-extracted\") pod \"image-registry-78db97464f-6ljzh\" (UID: \"5296979a-7927-4ee2-84b9-a93780a2ad22\") " pod="openshift-image-registry/image-registry-78db97464f-6ljzh" Oct 29 22:33:27.852727 ip-10-0-140-74 kubenswrapper[2577]: E1029 22:33:27.852566 2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 29 22:33:27.852727 ip-10-0-140-74 kubenswrapper[2577]: E1029 22:33:27.852581 2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-84866f86bd-2v6d7: secret "image-registry-tls" not found Oct 29 22:33:27.852727 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.852592 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vv2vp\" (UniqueName: \"kubernetes.io/projected/79ad3230-163e-447d-9f04-a072479ff957-kube-api-access-vv2vp\") pod \"kas-connection-checker-685895c7c9-5fjlz\" (UID: \"79ad3230-163e-447d-9f04-a072479ff957\") " pod="kube-system/kas-connection-checker-685895c7c9-5fjlz" Oct 29 22:33:27.852727 ip-10-0-140-74 kubenswrapper[2577]: E1029 22:33:27.852625 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/989956e8-1893-4bbe-a4bf-6a2081509080-registry-tls podName:989956e8-1893-4bbe-a4bf-6a2081509080 nodeName:}" failed. No retries permitted until 2026-10-29 22:33:28.352608434 +0000 UTC m=+30.509796097 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/989956e8-1893-4bbe-a4bf-6a2081509080-registry-tls") pod "image-registry-84866f86bd-2v6d7" (UID: "989956e8-1893-4bbe-a4bf-6a2081509080") : secret "image-registry-tls" not found Oct 29 22:33:27.852727 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.852624 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5296979a-7927-4ee2-84b9-a93780a2ad22-trusted-ca\") pod \"image-registry-78db97464f-6ljzh\" (UID: \"5296979a-7927-4ee2-84b9-a93780a2ad22\") " pod="openshift-image-registry/image-registry-78db97464f-6ljzh" Oct 29 22:33:27.852727 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.852645 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/989956e8-1893-4bbe-a4bf-6a2081509080-installation-pull-secrets\") pod \"image-registry-84866f86bd-2v6d7\" (UID: \"989956e8-1893-4bbe-a4bf-6a2081509080\") " pod="openshift-image-registry/image-registry-84866f86bd-2v6d7" Oct 29 22:33:27.852727 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.852677 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/641f9ec1-c8e2-4774-9d4f-6a7a2da97b6f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-8c5hr\" (UID: \"641f9ec1-c8e2-4774-9d4f-6a7a2da97b6f\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-8c5hr" Oct 29 22:33:27.852727 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.852714 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/71d80eec-b5e7-41d0-b58d-8a8d9a6c769c-default-certificate\") pod \"router-default-5d8d7cc579-8cwjp\" (UID: \"71d80eec-b5e7-41d0-b58d-8a8d9a6c769c\") " pod="openshift-ingress/router-default-5d8d7cc579-8cwjp" Oct 29 22:33:27.853244 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.852741 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkxzs\" (UniqueName: \"kubernetes.io/projected/c635d035-053b-49aa-9559-79c31e8e49ff-kube-api-access-mkxzs\") pod \"service-ca-operator-544d749c8c-pk97l\" (UID: \"c635d035-053b-49aa-9559-79c31e8e49ff\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-pk97l" Oct 29 22:33:27.853244 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.852769 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fp2lh\" (UniqueName: \"kubernetes.io/projected/5296979a-7927-4ee2-84b9-a93780a2ad22-kube-api-access-fp2lh\") pod \"image-registry-78db97464f-6ljzh\" (UID: \"5296979a-7927-4ee2-84b9-a93780a2ad22\") " pod="openshift-image-registry/image-registry-78db97464f-6ljzh" Oct 29 22:33:27.853244 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.852799 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/65c96dfb-06e9-4bea-bb83-f63b4c62d860-service-ca-bundle\") pod \"insights-operator-7c4986d449-b7bwd\" (UID: \"65c96dfb-06e9-4bea-bb83-f63b4c62d860\") " pod="openshift-insights/insights-operator-7c4986d449-b7bwd" Oct 29 22:33:27.853244 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.852459 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/65c96dfb-06e9-4bea-bb83-f63b4c62d860-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-b7bwd\" (UID: \"65c96dfb-06e9-4bea-bb83-f63b4c62d860\") " pod="openshift-insights/insights-operator-7c4986d449-b7bwd" Oct 29 22:33:27.853244 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.852802 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/989956e8-1893-4bbe-a4bf-6a2081509080-bound-sa-token\") pod \"image-registry-84866f86bd-2v6d7\" (UID: \"989956e8-1893-4bbe-a4bf-6a2081509080\") " pod="openshift-image-registry/image-registry-84866f86bd-2v6d7" Oct 29 22:33:27.853244 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.852869 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a53078b9-a6a8-423c-b852-34eececa7ab1-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-p2qv4\" (UID: \"a53078b9-a6a8-423c-b852-34eececa7ab1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-p2qv4" Oct 29 22:33:27.853244 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.852901 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8hb6\" (UniqueName: \"kubernetes.io/projected/4071b0c7-c4f3-419a-8d6e-6acc144fc9a4-kube-api-access-n8hb6\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-sk5qq\" (UID: \"4071b0c7-c4f3-419a-8d6e-6acc144fc9a4\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-sk5qq" Oct 29 22:33:27.853244 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.852933 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvmbg\" (UniqueName: \"kubernetes.io/projected/8406bb9f-3a4e-42f3-a822-f754d9b8e7a8-kube-api-access-xvmbg\") pod \"ingress-canary-czfl9\" (UID: \"8406bb9f-3a4e-42f3-a822-f754d9b8e7a8\") " pod="openshift-ingress-canary/ingress-canary-czfl9" Oct 29 22:33:27.853244 ip-10-0-140-74 kubenswrapper[2577]: E1029 22:33:27.852947 2577 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 29 22:33:27.853244 ip-10-0-140-74 kubenswrapper[2577]: E1029 22:33:27.852993 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a53078b9-a6a8-423c-b852-34eececa7ab1-samples-operator-tls podName:a53078b9-a6a8-423c-b852-34eececa7ab1 nodeName:}" failed. No retries permitted until 2026-10-29 22:33:28.352978882 +0000 UTC m=+30.510166565 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/a53078b9-a6a8-423c-b852-34eececa7ab1-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-p2qv4" (UID: "a53078b9-a6a8-423c-b852-34eececa7ab1") : secret "samples-operator-tls" not found Oct 29 22:33:27.853720 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.853501 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5296979a-7927-4ee2-84b9-a93780a2ad22-ca-trust-extracted\") pod \"image-registry-78db97464f-6ljzh\" (UID: \"5296979a-7927-4ee2-84b9-a93780a2ad22\") " pod="openshift-image-registry/image-registry-78db97464f-6ljzh" Oct 29 22:33:27.854461 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.854370 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/989956e8-1893-4bbe-a4bf-6a2081509080-trusted-ca\") pod \"image-registry-84866f86bd-2v6d7\" (UID: \"989956e8-1893-4bbe-a4bf-6a2081509080\") " pod="openshift-image-registry/image-registry-84866f86bd-2v6d7" Oct 29 22:33:27.855688 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.855660 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5296979a-7927-4ee2-84b9-a93780a2ad22-installation-pull-secrets\") pod \"image-registry-78db97464f-6ljzh\" (UID: \"5296979a-7927-4ee2-84b9-a93780a2ad22\") " pod="openshift-image-registry/image-registry-78db97464f-6ljzh" Oct 29 22:33:27.855774 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.855663 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/aa6a190e-3283-458a-9dfd-2de7cb193fc5-serving-cert\") pod \"console-operator-7cd56b8dbc-vj6md\" (UID: \"aa6a190e-3283-458a-9dfd-2de7cb193fc5\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-vj6md" Oct 29 22:33:27.855977 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.855954 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c635d035-053b-49aa-9559-79c31e8e49ff-serving-cert\") pod \"service-ca-operator-544d749c8c-pk97l\" (UID: \"c635d035-053b-49aa-9559-79c31e8e49ff\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-pk97l" Oct 29 22:33:27.856061 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.856044 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/989956e8-1893-4bbe-a4bf-6a2081509080-installation-pull-secrets\") pod \"image-registry-84866f86bd-2v6d7\" (UID: \"989956e8-1893-4bbe-a4bf-6a2081509080\") " pod="openshift-image-registry/image-registry-84866f86bd-2v6d7" Oct 29 22:33:27.856388 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.856142 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/71d80eec-b5e7-41d0-b58d-8a8d9a6c769c-stats-auth\") pod \"router-default-5d8d7cc579-8cwjp\" (UID: \"71d80eec-b5e7-41d0-b58d-8a8d9a6c769c\") " pod="openshift-ingress/router-default-5d8d7cc579-8cwjp" Oct 29 22:33:27.856388 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.856261 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/65c96dfb-06e9-4bea-bb83-f63b4c62d860-serving-cert\") pod \"insights-operator-7c4986d449-b7bwd\" (UID: \"65c96dfb-06e9-4bea-bb83-f63b4c62d860\") " pod="openshift-insights/insights-operator-7c4986d449-b7bwd" Oct 29 22:33:27.856649 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.856631 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/71d80eec-b5e7-41d0-b58d-8a8d9a6c769c-default-certificate\") pod \"router-default-5d8d7cc579-8cwjp\" (UID: \"71d80eec-b5e7-41d0-b58d-8a8d9a6c769c\") " pod="openshift-ingress/router-default-5d8d7cc579-8cwjp" Oct 29 22:33:27.856649 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.856640 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/989956e8-1893-4bbe-a4bf-6a2081509080-image-registry-private-configuration\") pod \"image-registry-84866f86bd-2v6d7\" (UID: \"989956e8-1893-4bbe-a4bf-6a2081509080\") " pod="openshift-image-registry/image-registry-84866f86bd-2v6d7" Oct 29 22:33:27.857172 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.857144 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5296979a-7927-4ee2-84b9-a93780a2ad22-image-registry-private-configuration\") pod \"image-registry-78db97464f-6ljzh\" (UID: \"5296979a-7927-4ee2-84b9-a93780a2ad22\") " pod="openshift-image-registry/image-registry-78db97464f-6ljzh" Oct 29 22:33:27.873622 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.873594 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjwwn\" (UniqueName: \"kubernetes.io/projected/587893f5-e85a-46a9-829f-545a42c10056-kube-api-access-bjwwn\") pod \"kas-connection-checker-685895c7c9-jgvr6\" (UID: \"587893f5-e85a-46a9-829f-545a42c10056\") " pod="kube-system/kas-connection-checker-685895c7c9-jgvr6" Oct 29 22:33:27.873622 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.873615 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-whjbn\" (UniqueName: \"kubernetes.io/projected/4ff372bf-82aa-4007-b74c-df0c779e5271-kube-api-access-whjbn\") pod \"kas-connection-checker-685895c7c9-mrbgh\" (UID: \"4ff372bf-82aa-4007-b74c-df0c779e5271\") " pod="kube-system/kas-connection-checker-685895c7c9-mrbgh" Oct 29 22:33:27.873761 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.873715 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5296979a-7927-4ee2-84b9-a93780a2ad22-bound-sa-token\") pod \"image-registry-78db97464f-6ljzh\" (UID: \"5296979a-7927-4ee2-84b9-a93780a2ad22\") " pod="openshift-image-registry/image-registry-78db97464f-6ljzh" Oct 29 22:33:27.874092 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.874077 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4xwz\" (UniqueName: \"kubernetes.io/projected/aa6a190e-3283-458a-9dfd-2de7cb193fc5-kube-api-access-c4xwz\") pod \"console-operator-7cd56b8dbc-vj6md\" (UID: \"aa6a190e-3283-458a-9dfd-2de7cb193fc5\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-vj6md" Oct 29 22:33:27.874207 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.874191 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7h8fj\" (UniqueName: \"kubernetes.io/projected/c54ef754-c91e-4bc5-8e83-08523aa09d37-kube-api-access-7h8fj\") pod \"volume-data-source-validator-58c9f5f46d-w5d4m\" (UID: \"c54ef754-c91e-4bc5-8e83-08523aa09d37\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-w5d4m" Oct 29 22:33:27.880649 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.880623 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfk2t\" (UniqueName: \"kubernetes.io/projected/989956e8-1893-4bbe-a4bf-6a2081509080-kube-api-access-gfk2t\") pod \"image-registry-84866f86bd-2v6d7\" (UID: \"989956e8-1893-4bbe-a4bf-6a2081509080\") " pod="openshift-image-registry/image-registry-84866f86bd-2v6d7" Oct 29 22:33:27.881609 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.881225 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2tw5\" (UniqueName: \"kubernetes.io/projected/65c96dfb-06e9-4bea-bb83-f63b4c62d860-kube-api-access-z2tw5\") pod \"insights-operator-7c4986d449-b7bwd\" (UID: \"65c96dfb-06e9-4bea-bb83-f63b4c62d860\") " pod="openshift-insights/insights-operator-7c4986d449-b7bwd" Oct 29 22:33:27.881609 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.881256 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-698ds\" (UniqueName: \"kubernetes.io/projected/a53078b9-a6a8-423c-b852-34eececa7ab1-kube-api-access-698ds\") pod \"cluster-samples-operator-6ddb6c8b94-p2qv4\" (UID: \"a53078b9-a6a8-423c-b852-34eececa7ab1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-p2qv4" Oct 29 22:33:27.881609 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.881563 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkxzs\" (UniqueName: \"kubernetes.io/projected/c635d035-053b-49aa-9559-79c31e8e49ff-kube-api-access-mkxzs\") pod \"service-ca-operator-544d749c8c-pk97l\" (UID: \"c635d035-053b-49aa-9559-79c31e8e49ff\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-pk97l" Oct 29 22:33:27.881800 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.881719 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fp2lh\" (UniqueName: \"kubernetes.io/projected/5296979a-7927-4ee2-84b9-a93780a2ad22-kube-api-access-fp2lh\") pod \"image-registry-78db97464f-6ljzh\" (UID: \"5296979a-7927-4ee2-84b9-a93780a2ad22\") " pod="openshift-image-registry/image-registry-78db97464f-6ljzh" Oct 29 22:33:27.881800 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.881758 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/989956e8-1893-4bbe-a4bf-6a2081509080-bound-sa-token\") pod \"image-registry-84866f86bd-2v6d7\" (UID: \"989956e8-1893-4bbe-a4bf-6a2081509080\") " pod="openshift-image-registry/image-registry-84866f86bd-2v6d7" Oct 29 22:33:27.883017 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.882999 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2tdnl\" (UniqueName: \"kubernetes.io/projected/71d80eec-b5e7-41d0-b58d-8a8d9a6c769c-kube-api-access-2tdnl\") pod \"router-default-5d8d7cc579-8cwjp\" (UID: \"71d80eec-b5e7-41d0-b58d-8a8d9a6c769c\") " pod="openshift-ingress/router-default-5d8d7cc579-8cwjp" Oct 29 22:33:27.883672 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.883653 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vv2vp\" (UniqueName: \"kubernetes.io/projected/79ad3230-163e-447d-9f04-a072479ff957-kube-api-access-vv2vp\") pod \"kas-connection-checker-685895c7c9-5fjlz\" (UID: \"79ad3230-163e-447d-9f04-a072479ff957\") " pod="kube-system/kas-connection-checker-685895c7c9-5fjlz" Oct 29 22:33:27.885530 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.885507 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:33:27.950068 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.950047 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-5fjlz" Oct 29 22:33:27.953405 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.953387 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8gw9q\" (UniqueName: \"kubernetes.io/projected/6b34b3b1-84af-45f3-92ee-1ca720e8b263-kube-api-access-8gw9q\") pod \"dns-default-jjz47\" (UID: \"6b34b3b1-84af-45f3-92ee-1ca720e8b263\") " pod="openshift-dns/dns-default-jjz47" Oct 29 22:33:27.953465 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.953417 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/641f9ec1-c8e2-4774-9d4f-6a7a2da97b6f-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-8c5hr\" (UID: \"641f9ec1-c8e2-4774-9d4f-6a7a2da97b6f\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-8c5hr" Oct 29 22:33:27.953465 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.953444 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/6b34b3b1-84af-45f3-92ee-1ca720e8b263-config-volume\") pod \"dns-default-jjz47\" (UID: \"6b34b3b1-84af-45f3-92ee-1ca720e8b263\") " pod="openshift-dns/dns-default-jjz47" Oct 29 22:33:27.953602 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.953575 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b34b3b1-84af-45f3-92ee-1ca720e8b263-tmp-dir\") pod \"dns-default-jjz47\" (UID: \"6b34b3b1-84af-45f3-92ee-1ca720e8b263\") " pod="openshift-dns/dns-default-jjz47" Oct 29 22:33:27.953749 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.953729 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4071b0c7-c4f3-419a-8d6e-6acc144fc9a4-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-sk5qq\" (UID: \"4071b0c7-c4f3-419a-8d6e-6acc144fc9a4\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-sk5qq" Oct 29 22:33:27.953787 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.953761 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8406bb9f-3a4e-42f3-a822-f754d9b8e7a8-cert\") pod \"ingress-canary-czfl9\" (UID: \"8406bb9f-3a4e-42f3-a822-f754d9b8e7a8\") " pod="openshift-ingress-canary/ingress-canary-czfl9" Oct 29 22:33:27.953834 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.953799 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/641f9ec1-c8e2-4774-9d4f-6a7a2da97b6f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-8c5hr\" (UID: \"641f9ec1-c8e2-4774-9d4f-6a7a2da97b6f\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-8c5hr" Oct 29 22:33:27.953883 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.953850 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8hb6\" (UniqueName: \"kubernetes.io/projected/4071b0c7-c4f3-419a-8d6e-6acc144fc9a4-kube-api-access-n8hb6\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-sk5qq\" (UID: \"4071b0c7-c4f3-419a-8d6e-6acc144fc9a4\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-sk5qq" Oct 29 22:33:27.953883 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.953870 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b34b3b1-84af-45f3-92ee-1ca720e8b263-tmp-dir\") pod \"dns-default-jjz47\" (UID: \"6b34b3b1-84af-45f3-92ee-1ca720e8b263\") " pod="openshift-dns/dns-default-jjz47" Oct 29 22:33:27.953975 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.953875 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvmbg\" (UniqueName: \"kubernetes.io/projected/8406bb9f-3a4e-42f3-a822-f754d9b8e7a8-kube-api-access-xvmbg\") pod \"ingress-canary-czfl9\" (UID: \"8406bb9f-3a4e-42f3-a822-f754d9b8e7a8\") " pod="openshift-ingress-canary/ingress-canary-czfl9" Oct 29 22:33:27.953975 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.953939 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8d5d\" (UniqueName: \"kubernetes.io/projected/dd4e19ce-ef37-44e5-8b57-9dafc590685c-kube-api-access-l8d5d\") pod \"network-check-source-6f78d59f99-p9ds4\" (UID: \"dd4e19ce-ef37-44e5-8b57-9dafc590685c\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-p9ds4" Oct 29 22:33:27.953975 ip-10-0-140-74 kubenswrapper[2577]: E1029 22:33:27.953964 2577 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 29 22:33:27.954078 ip-10-0-140-74 kubenswrapper[2577]: E1029 22:33:27.954020 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/641f9ec1-c8e2-4774-9d4f-6a7a2da97b6f-cluster-monitoring-operator-tls podName:641f9ec1-c8e2-4774-9d4f-6a7a2da97b6f nodeName:}" failed. No retries permitted until 2026-10-29 22:33:28.454004369 +0000 UTC m=+30.611192048 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/641f9ec1-c8e2-4774-9d4f-6a7a2da97b6f-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-8c5hr" (UID: "641f9ec1-c8e2-4774-9d4f-6a7a2da97b6f") : secret "cluster-monitoring-operator-tls" not found Oct 29 22:33:27.954078 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.953968 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6sv4r\" (UniqueName: \"kubernetes.io/projected/641f9ec1-c8e2-4774-9d4f-6a7a2da97b6f-kube-api-access-6sv4r\") pod \"cluster-monitoring-operator-884576c57-8c5hr\" (UID: \"641f9ec1-c8e2-4774-9d4f-6a7a2da97b6f\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-8c5hr" Oct 29 22:33:27.954078 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.954059 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6b34b3b1-84af-45f3-92ee-1ca720e8b263-metrics-tls\") pod \"dns-default-jjz47\" (UID: \"6b34b3b1-84af-45f3-92ee-1ca720e8b263\") " pod="openshift-dns/dns-default-jjz47" Oct 29 22:33:27.954179 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.954098 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4071b0c7-c4f3-419a-8d6e-6acc144fc9a4-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-sk5qq\" (UID: \"4071b0c7-c4f3-419a-8d6e-6acc144fc9a4\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-sk5qq" Oct 29 22:33:27.954283 ip-10-0-140-74 kubenswrapper[2577]: E1029 22:33:27.954265 2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 29 22:33:27.954357 ip-10-0-140-74 kubenswrapper[2577]: E1029 22:33:27.954323 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b34b3b1-84af-45f3-92ee-1ca720e8b263-metrics-tls podName:6b34b3b1-84af-45f3-92ee-1ca720e8b263 nodeName:}" failed. No retries permitted until 2026-10-29 22:33:28.454308596 +0000 UTC m=+30.611496269 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6b34b3b1-84af-45f3-92ee-1ca720e8b263-metrics-tls") pod "dns-default-jjz47" (UID: "6b34b3b1-84af-45f3-92ee-1ca720e8b263") : secret "dns-default-metrics-tls" not found Oct 29 22:33:27.954449 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.954434 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/641f9ec1-c8e2-4774-9d4f-6a7a2da97b6f-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-8c5hr\" (UID: \"641f9ec1-c8e2-4774-9d4f-6a7a2da97b6f\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-8c5hr" Oct 29 22:33:27.954604 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.954583 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4071b0c7-c4f3-419a-8d6e-6acc144fc9a4-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-sk5qq\" (UID: \"4071b0c7-c4f3-419a-8d6e-6acc144fc9a4\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-sk5qq" Oct 29 22:33:27.954696 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.954678 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/6b34b3b1-84af-45f3-92ee-1ca720e8b263-config-volume\") pod \"dns-default-jjz47\" (UID: \"6b34b3b1-84af-45f3-92ee-1ca720e8b263\") " pod="openshift-dns/dns-default-jjz47" Oct 29 22:33:27.956280 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.956260 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4071b0c7-c4f3-419a-8d6e-6acc144fc9a4-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-sk5qq\" (UID: \"4071b0c7-c4f3-419a-8d6e-6acc144fc9a4\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-sk5qq" Oct 29 22:33:27.956526 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.956508 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8406bb9f-3a4e-42f3-a822-f754d9b8e7a8-cert\") pod \"ingress-canary-czfl9\" (UID: \"8406bb9f-3a4e-42f3-a822-f754d9b8e7a8\") " pod="openshift-ingress-canary/ingress-canary-czfl9" Oct 29 22:33:27.957745 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.957729 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-jgvr6" Oct 29 22:33:27.975304 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.975277 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8d5d\" (UniqueName: \"kubernetes.io/projected/dd4e19ce-ef37-44e5-8b57-9dafc590685c-kube-api-access-l8d5d\") pod \"network-check-source-6f78d59f99-p9ds4\" (UID: \"dd4e19ce-ef37-44e5-8b57-9dafc590685c\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-p9ds4" Oct 29 22:33:27.975304 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.975296 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8hb6\" (UniqueName: \"kubernetes.io/projected/4071b0c7-c4f3-419a-8d6e-6acc144fc9a4-kube-api-access-n8hb6\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-sk5qq\" (UID: \"4071b0c7-c4f3-419a-8d6e-6acc144fc9a4\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-sk5qq" Oct 29 22:33:27.975444 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.975320 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8gw9q\" (UniqueName: \"kubernetes.io/projected/6b34b3b1-84af-45f3-92ee-1ca720e8b263-kube-api-access-8gw9q\") pod \"dns-default-jjz47\" (UID: \"6b34b3b1-84af-45f3-92ee-1ca720e8b263\") " pod="openshift-dns/dns-default-jjz47" Oct 29 22:33:27.975879 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.975859 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvmbg\" (UniqueName: \"kubernetes.io/projected/8406bb9f-3a4e-42f3-a822-f754d9b8e7a8-kube-api-access-xvmbg\") pod \"ingress-canary-czfl9\" (UID: \"8406bb9f-3a4e-42f3-a822-f754d9b8e7a8\") " pod="openshift-ingress-canary/ingress-canary-czfl9" Oct 29 22:33:27.975915 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.975859 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6sv4r\" (UniqueName: \"kubernetes.io/projected/641f9ec1-c8e2-4774-9d4f-6a7a2da97b6f-kube-api-access-6sv4r\") pod \"cluster-monitoring-operator-884576c57-8c5hr\" (UID: \"641f9ec1-c8e2-4774-9d4f-6a7a2da97b6f\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-8c5hr" Oct 29 22:33:27.977483 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:27.977468 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-b7bwd" Oct 29 22:33:28.004465 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:28.004449 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-mrbgh" Oct 29 22:33:28.010603 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:28.010587 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-w5d4m" Oct 29 22:33:28.023247 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:28.023230 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-pk97l" Oct 29 22:33:28.029162 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:28.029140 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-vj6md" Oct 29 22:33:28.041197 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:28.041174 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-p9ds4" Oct 29 22:33:28.055706 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:28.055674 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-sk5qq" Oct 29 22:33:28.069609 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:28.069584 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-czfl9" Oct 29 22:33:28.332351 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:28.332307 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-jgvr6"] Oct 29 22:33:28.335506 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:28.335477 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-5fjlz"] Oct 29 22:33:28.363106 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:28.363046 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a53078b9-a6a8-423c-b852-34eececa7ab1-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-p2qv4\" (UID: \"a53078b9-a6a8-423c-b852-34eececa7ab1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-p2qv4" Oct 29 22:33:28.363198 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:28.363107 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5296979a-7927-4ee2-84b9-a93780a2ad22-registry-tls\") pod \"image-registry-78db97464f-6ljzh\" (UID: \"5296979a-7927-4ee2-84b9-a93780a2ad22\") " pod="openshift-image-registry/image-registry-78db97464f-6ljzh" Oct 29 22:33:28.363198 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:28.363177 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/71d80eec-b5e7-41d0-b58d-8a8d9a6c769c-service-ca-bundle\") pod \"router-default-5d8d7cc579-8cwjp\" (UID: \"71d80eec-b5e7-41d0-b58d-8a8d9a6c769c\") " pod="openshift-ingress/router-default-5d8d7cc579-8cwjp" Oct 29 22:33:28.363302 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:28.363235 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2accd775-2173-4bfe-a413-7d243bd0820f-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-6rr4f\" (UID: \"2accd775-2173-4bfe-a413-7d243bd0820f\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-6rr4f" Oct 29 22:33:28.363302 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:28.363279 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71d80eec-b5e7-41d0-b58d-8a8d9a6c769c-metrics-certs\") pod \"router-default-5d8d7cc579-8cwjp\" (UID: \"71d80eec-b5e7-41d0-b58d-8a8d9a6c769c\") " pod="openshift-ingress/router-default-5d8d7cc579-8cwjp" Oct 29 22:33:28.363418 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:28.363306 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/989956e8-1893-4bbe-a4bf-6a2081509080-registry-tls\") pod \"image-registry-84866f86bd-2v6d7\" (UID: \"989956e8-1893-4bbe-a4bf-6a2081509080\") " pod="openshift-image-registry/image-registry-84866f86bd-2v6d7" Oct 29 22:33:28.363476 ip-10-0-140-74 kubenswrapper[2577]: E1029 22:33:28.363459 2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 29 22:33:28.363533 ip-10-0-140-74 kubenswrapper[2577]: E1029 22:33:28.363480 2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-84866f86bd-2v6d7: secret "image-registry-tls" not found Oct 29 22:33:28.363533 ip-10-0-140-74 kubenswrapper[2577]: E1029 22:33:28.363528 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/989956e8-1893-4bbe-a4bf-6a2081509080-registry-tls podName:989956e8-1893-4bbe-a4bf-6a2081509080 nodeName:}" failed. No retries permitted until 2026-10-29 22:33:29.363511896 +0000 UTC m=+31.520699580 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/989956e8-1893-4bbe-a4bf-6a2081509080-registry-tls") pod "image-registry-84866f86bd-2v6d7" (UID: "989956e8-1893-4bbe-a4bf-6a2081509080") : secret "image-registry-tls" not found Oct 29 22:33:28.363643 ip-10-0-140-74 kubenswrapper[2577]: E1029 22:33:28.363607 2577 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 29 22:33:28.363691 ip-10-0-140-74 kubenswrapper[2577]: E1029 22:33:28.363643 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a53078b9-a6a8-423c-b852-34eececa7ab1-samples-operator-tls podName:a53078b9-a6a8-423c-b852-34eececa7ab1 nodeName:}" failed. No retries permitted until 2026-10-29 22:33:29.363632664 +0000 UTC m=+31.520820338 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/a53078b9-a6a8-423c-b852-34eececa7ab1-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-p2qv4" (UID: "a53078b9-a6a8-423c-b852-34eececa7ab1") : secret "samples-operator-tls" not found Oct 29 22:33:28.363742 ip-10-0-140-74 kubenswrapper[2577]: E1029 22:33:28.363699 2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 29 22:33:28.363742 ip-10-0-140-74 kubenswrapper[2577]: E1029 22:33:28.363710 2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-78db97464f-6ljzh: secret "image-registry-tls" not found Oct 29 22:33:28.363742 ip-10-0-140-74 kubenswrapper[2577]: E1029 22:33:28.363739 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5296979a-7927-4ee2-84b9-a93780a2ad22-registry-tls podName:5296979a-7927-4ee2-84b9-a93780a2ad22 nodeName:}" failed. No retries permitted until 2026-10-29 22:33:29.363729981 +0000 UTC m=+31.520917663 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5296979a-7927-4ee2-84b9-a93780a2ad22-registry-tls") pod "image-registry-78db97464f-6ljzh" (UID: "5296979a-7927-4ee2-84b9-a93780a2ad22") : secret "image-registry-tls" not found Oct 29 22:33:28.363887 ip-10-0-140-74 kubenswrapper[2577]: E1029 22:33:28.363805 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/71d80eec-b5e7-41d0-b58d-8a8d9a6c769c-service-ca-bundle podName:71d80eec-b5e7-41d0-b58d-8a8d9a6c769c nodeName:}" failed. No retries permitted until 2026-10-29 22:33:29.363795309 +0000 UTC m=+31.520982987 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/71d80eec-b5e7-41d0-b58d-8a8d9a6c769c-service-ca-bundle") pod "router-default-5d8d7cc579-8cwjp" (UID: "71d80eec-b5e7-41d0-b58d-8a8d9a6c769c") : configmap references non-existent config key: service-ca.crt Oct 29 22:33:28.363887 ip-10-0-140-74 kubenswrapper[2577]: E1029 22:33:28.363859 2577 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 29 22:33:28.363980 ip-10-0-140-74 kubenswrapper[2577]: E1029 22:33:28.363891 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2accd775-2173-4bfe-a413-7d243bd0820f-networking-console-plugin-cert podName:2accd775-2173-4bfe-a413-7d243bd0820f nodeName:}" failed. No retries permitted until 2026-10-29 22:33:29.363881254 +0000 UTC m=+31.521068917 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/2accd775-2173-4bfe-a413-7d243bd0820f-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-6rr4f" (UID: "2accd775-2173-4bfe-a413-7d243bd0820f") : secret "networking-console-plugin-cert" not found Oct 29 22:33:28.363980 ip-10-0-140-74 kubenswrapper[2577]: E1029 22:33:28.363940 2577 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 29 22:33:28.363980 ip-10-0-140-74 kubenswrapper[2577]: E1029 22:33:28.363969 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71d80eec-b5e7-41d0-b58d-8a8d9a6c769c-metrics-certs podName:71d80eec-b5e7-41d0-b58d-8a8d9a6c769c nodeName:}" failed. No retries permitted until 2026-10-29 22:33:29.36395956 +0000 UTC m=+31.521147236 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/71d80eec-b5e7-41d0-b58d-8a8d9a6c769c-metrics-certs") pod "router-default-5d8d7cc579-8cwjp" (UID: "71d80eec-b5e7-41d0-b58d-8a8d9a6c769c") : secret "router-metrics-certs-default" not found Oct 29 22:33:28.378987 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:28.378960 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-mrbgh"] Oct 29 22:33:28.382319 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:28.382296 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-vj6md"] Oct 29 22:33:28.426102 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:28.426075 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-b7bwd"] Oct 29 22:33:28.428982 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:28.428959 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-pk97l"] Oct 29 22:33:28.433496 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:33:28.433457 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65c96dfb_06e9_4bea_bb83_f63b4c62d860.slice/crio-d4c08358fd86fbf544eeb14448141bb045e6b6e45eb23b1d549aa8870fe1c4e1 WatchSource:0}: Error finding container d4c08358fd86fbf544eeb14448141bb045e6b6e45eb23b1d549aa8870fe1c4e1: Status 404 returned error can't find the container with id d4c08358fd86fbf544eeb14448141bb045e6b6e45eb23b1d549aa8870fe1c4e1 Oct 29 22:33:28.463803 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:28.463778 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/641f9ec1-c8e2-4774-9d4f-6a7a2da97b6f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-8c5hr\" (UID: \"641f9ec1-c8e2-4774-9d4f-6a7a2da97b6f\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-8c5hr" Oct 29 22:33:28.463905 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:28.463843 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6b34b3b1-84af-45f3-92ee-1ca720e8b263-metrics-tls\") pod \"dns-default-jjz47\" (UID: \"6b34b3b1-84af-45f3-92ee-1ca720e8b263\") " pod="openshift-dns/dns-default-jjz47" Oct 29 22:33:28.463966 ip-10-0-140-74 kubenswrapper[2577]: E1029 22:33:28.463953 2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 29 22:33:28.464011 ip-10-0-140-74 kubenswrapper[2577]: E1029 22:33:28.463993 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b34b3b1-84af-45f3-92ee-1ca720e8b263-metrics-tls podName:6b34b3b1-84af-45f3-92ee-1ca720e8b263 nodeName:}" failed. No retries permitted until 2026-10-29 22:33:29.463981964 +0000 UTC m=+31.621169626 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6b34b3b1-84af-45f3-92ee-1ca720e8b263-metrics-tls") pod "dns-default-jjz47" (UID: "6b34b3b1-84af-45f3-92ee-1ca720e8b263") : secret "dns-default-metrics-tls" not found Oct 29 22:33:28.464060 ip-10-0-140-74 kubenswrapper[2577]: E1029 22:33:28.464046 2577 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 29 22:33:28.464107 ip-10-0-140-74 kubenswrapper[2577]: E1029 22:33:28.464068 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/641f9ec1-c8e2-4774-9d4f-6a7a2da97b6f-cluster-monitoring-operator-tls podName:641f9ec1-c8e2-4774-9d4f-6a7a2da97b6f nodeName:}" failed. No retries permitted until 2026-10-29 22:33:29.464061624 +0000 UTC m=+31.621249286 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/641f9ec1-c8e2-4774-9d4f-6a7a2da97b6f-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-8c5hr" (UID: "641f9ec1-c8e2-4774-9d4f-6a7a2da97b6f") : secret "cluster-monitoring-operator-tls" not found Oct 29 22:33:28.475179 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:28.470790 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-w5d4m"] Oct 29 22:33:28.476478 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:33:28.476448 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc54ef754_c91e_4bc5_8e83_08523aa09d37.slice/crio-7f2c50e656caef7bde619a98fc6022b5a14ed2ab51fb4b904fe9185e4ef04c86 WatchSource:0}: Error finding container 7f2c50e656caef7bde619a98fc6022b5a14ed2ab51fb4b904fe9185e4ef04c86: Status 404 returned error can't find the container with id 7f2c50e656caef7bde619a98fc6022b5a14ed2ab51fb4b904fe9185e4ef04c86 Oct 29 22:33:28.492578 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:28.492553 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-p9ds4"] Oct 29 22:33:28.500847 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:33:28.500187 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd4e19ce_ef37_44e5_8b57_9dafc590685c.slice/crio-6d1c35a93c18cc546622631e919464a48df503c2a5b066a732af5226f05dce9c WatchSource:0}: Error finding container 6d1c35a93c18cc546622631e919464a48df503c2a5b066a732af5226f05dce9c: Status 404 returned error can't find the container with id 6d1c35a93c18cc546622631e919464a48df503c2a5b066a732af5226f05dce9c Oct 29 22:33:28.504467 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:28.504447 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-sk5qq"] Oct 29 22:33:28.507652 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:33:28.507629 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4071b0c7_c4f3_419a_8d6e_6acc144fc9a4.slice/crio-9bb152b2ed8e5b9f7090432bf0a139a9d586707e7d782cba121f6bb7b126aed4 WatchSource:0}: Error finding container 9bb152b2ed8e5b9f7090432bf0a139a9d586707e7d782cba121f6bb7b126aed4: Status 404 returned error can't find the container with id 9bb152b2ed8e5b9f7090432bf0a139a9d586707e7d782cba121f6bb7b126aed4 Oct 29 22:33:28.532110 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:28.532050 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-czfl9"] Oct 29 22:33:28.538973 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:33:28.538952 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8406bb9f_3a4e_42f3_a822_f754d9b8e7a8.slice/crio-2dbb0c4a7640796e280454d201a82932159022b408fb05d309bb0cdc047866c6 WatchSource:0}: Error finding container 2dbb0c4a7640796e280454d201a82932159022b408fb05d309bb0cdc047866c6: Status 404 returned error can't find the container with id 2dbb0c4a7640796e280454d201a82932159022b408fb05d309bb0cdc047866c6 Oct 29 22:33:28.608639 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:28.608601 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-czfl9" event={"ID":"8406bb9f-3a4e-42f3-a822-f754d9b8e7a8","Type":"ContainerStarted","Data":"2dbb0c4a7640796e280454d201a82932159022b408fb05d309bb0cdc047866c6"} Oct 29 22:33:28.609710 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:28.609688 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-w5d4m" event={"ID":"c54ef754-c91e-4bc5-8e83-08523aa09d37","Type":"ContainerStarted","Data":"7f2c50e656caef7bde619a98fc6022b5a14ed2ab51fb4b904fe9185e4ef04c86"} Oct 29 22:33:28.611008 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:28.610978 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-mrbgh" event={"ID":"4ff372bf-82aa-4007-b74c-df0c779e5271","Type":"ContainerStarted","Data":"8cf0cefeda93b02053fc5ef6a3a1708eeed241e057a4da99f7af00d688e1753a"} Oct 29 22:33:28.611083 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:28.611028 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-mrbgh" event={"ID":"4ff372bf-82aa-4007-b74c-df0c779e5271","Type":"ContainerStarted","Data":"d5f1b36c251b39df9f18518a01a694a79b6ab1196949d53c53d8b750c0775edd"} Oct 29 22:33:28.612368 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:28.612347 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-5fjlz" event={"ID":"79ad3230-163e-447d-9f04-a072479ff957","Type":"ContainerStarted","Data":"572d9d4e6aa47e6ef35aff209ea8b089c85dc860c5922382f3feec3a49e06b89"} Oct 29 22:33:28.612433 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:28.612386 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-5fjlz" event={"ID":"79ad3230-163e-447d-9f04-a072479ff957","Type":"ContainerStarted","Data":"8aec9d2b53599ac849ba2c32c12dd340a8651fcd1e0f2ff829e14e25ffd558ee"} Oct 29 22:33:28.613387 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:28.613344 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-sk5qq" event={"ID":"4071b0c7-c4f3-419a-8d6e-6acc144fc9a4","Type":"ContainerStarted","Data":"9bb152b2ed8e5b9f7090432bf0a139a9d586707e7d782cba121f6bb7b126aed4"} Oct 29 22:33:28.614841 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:28.614813 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-jgvr6" event={"ID":"587893f5-e85a-46a9-829f-545a42c10056","Type":"ContainerStarted","Data":"84d636367a8b3b1b47423b3615b1b14b43d6435c8178a9e98eea49970c49cbd1"} Oct 29 22:33:28.614913 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:28.614846 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-jgvr6" event={"ID":"587893f5-e85a-46a9-829f-545a42c10056","Type":"ContainerStarted","Data":"ccdd3c46b100f9e0da773d4ceccf7e6ea046a4f55344cee20eea5b936e882816"} Oct 29 22:33:28.615895 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:28.615877 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-p9ds4" event={"ID":"dd4e19ce-ef37-44e5-8b57-9dafc590685c","Type":"ContainerStarted","Data":"6d1c35a93c18cc546622631e919464a48df503c2a5b066a732af5226f05dce9c"} Oct 29 22:33:28.616993 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:28.616974 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-pk97l" event={"ID":"c635d035-053b-49aa-9559-79c31e8e49ff","Type":"ContainerStarted","Data":"54fa8fc945d5c93ccfdd34d26d813ccb81a4c450cd4a8ed1ced70ba9319e4dad"} Oct 29 22:33:28.618018 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:28.617994 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-b7bwd" event={"ID":"65c96dfb-06e9-4bea-bb83-f63b4c62d860","Type":"ContainerStarted","Data":"d4c08358fd86fbf544eeb14448141bb045e6b6e45eb23b1d549aa8870fe1c4e1"} Oct 29 22:33:28.618911 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:28.618894 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-vj6md" event={"ID":"aa6a190e-3283-458a-9dfd-2de7cb193fc5","Type":"ContainerStarted","Data":"a850aad9d1fa77621f0945112e249020d542379254765dba31625cadfce0b375"} Oct 29 22:33:28.645677 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:28.645640 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-mrbgh" podStartSLOduration=1040.645630411 podStartE2EDuration="17m20.645630411s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:16:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:33:28.644779178 +0000 UTC m=+30.801966880" watchObservedRunningTime="2026-10-29 22:33:28.645630411 +0000 UTC m=+30.802818125" Oct 29 22:33:28.679198 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:28.679161 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-jgvr6" podStartSLOduration=1040.679146302 podStartE2EDuration="17m20.679146302s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:16:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:33:28.678479419 +0000 UTC m=+30.835667096" watchObservedRunningTime="2026-10-29 22:33:28.679146302 +0000 UTC m=+30.836334005" Oct 29 22:33:28.715568 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:28.715534 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-5fjlz" podStartSLOduration=1040.715520857 podStartE2EDuration="17m20.715520857s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:16:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:33:28.715095849 +0000 UTC m=+30.872283547" watchObservedRunningTime="2026-10-29 22:33:28.715520857 +0000 UTC m=+30.872708540" Oct 29 22:33:29.325987 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:29.324917 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bp46g" Oct 29 22:33:29.325987 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:29.325417 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7jv5g" Oct 29 22:33:29.325987 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:29.325663 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hzlmg" Oct 29 22:33:29.339862 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:29.339398 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 29 22:33:29.339862 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:29.339566 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-dvjrv\"" Oct 29 22:33:29.339862 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:29.339695 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-zrmx7\"" Oct 29 22:33:29.339862 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:29.339821 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-rgvrg\"" Oct 29 22:33:29.340463 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:29.339924 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 29 22:33:29.373440 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:29.373420 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5296979a-7927-4ee2-84b9-a93780a2ad22-registry-tls\") pod \"image-registry-78db97464f-6ljzh\" (UID: \"5296979a-7927-4ee2-84b9-a93780a2ad22\") " pod="openshift-image-registry/image-registry-78db97464f-6ljzh" Oct 29 22:33:29.373550 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:29.373499 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/71d80eec-b5e7-41d0-b58d-8a8d9a6c769c-service-ca-bundle\") pod \"router-default-5d8d7cc579-8cwjp\" (UID: \"71d80eec-b5e7-41d0-b58d-8a8d9a6c769c\") " pod="openshift-ingress/router-default-5d8d7cc579-8cwjp" Oct 29 22:33:29.373609 ip-10-0-140-74 kubenswrapper[2577]: E1029 22:33:29.373555 2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 29 22:33:29.373609 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:29.373573 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2accd775-2173-4bfe-a413-7d243bd0820f-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-6rr4f\" (UID: \"2accd775-2173-4bfe-a413-7d243bd0820f\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-6rr4f" Oct 29 22:33:29.373704 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:29.373614 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71d80eec-b5e7-41d0-b58d-8a8d9a6c769c-metrics-certs\") pod \"router-default-5d8d7cc579-8cwjp\" (UID: \"71d80eec-b5e7-41d0-b58d-8a8d9a6c769c\") " pod="openshift-ingress/router-default-5d8d7cc579-8cwjp" Oct 29 22:33:29.373704 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:29.373631 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/989956e8-1893-4bbe-a4bf-6a2081509080-registry-tls\") pod \"image-registry-84866f86bd-2v6d7\" (UID: \"989956e8-1893-4bbe-a4bf-6a2081509080\") " pod="openshift-image-registry/image-registry-84866f86bd-2v6d7" Oct 29 22:33:29.374241 ip-10-0-140-74 kubenswrapper[2577]: E1029 22:33:29.373576 2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-78db97464f-6ljzh: secret "image-registry-tls" not found Oct 29 22:33:29.374241 ip-10-0-140-74 kubenswrapper[2577]: E1029 22:33:29.373836 2577 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 29 22:33:29.374241 ip-10-0-140-74 kubenswrapper[2577]: E1029 22:33:29.373673 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/71d80eec-b5e7-41d0-b58d-8a8d9a6c769c-service-ca-bundle podName:71d80eec-b5e7-41d0-b58d-8a8d9a6c769c nodeName:}" failed. No retries permitted until 2026-10-29 22:33:31.373654678 +0000 UTC m=+33.530842344 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/71d80eec-b5e7-41d0-b58d-8a8d9a6c769c-service-ca-bundle") pod "router-default-5d8d7cc579-8cwjp" (UID: "71d80eec-b5e7-41d0-b58d-8a8d9a6c769c") : configmap references non-existent config key: service-ca.crt Oct 29 22:33:29.374241 ip-10-0-140-74 kubenswrapper[2577]: E1029 22:33:29.373734 2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 29 22:33:29.374241 ip-10-0-140-74 kubenswrapper[2577]: E1029 22:33:29.373898 2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-84866f86bd-2v6d7: secret "image-registry-tls" not found Oct 29 22:33:29.374241 ip-10-0-140-74 kubenswrapper[2577]: E1029 22:33:29.373920 2577 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 29 22:33:29.374241 ip-10-0-140-74 kubenswrapper[2577]: E1029 22:33:29.373943 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2accd775-2173-4bfe-a413-7d243bd0820f-networking-console-plugin-cert podName:2accd775-2173-4bfe-a413-7d243bd0820f nodeName:}" failed. No retries permitted until 2026-10-29 22:33:31.373926915 +0000 UTC m=+33.531114585 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/2accd775-2173-4bfe-a413-7d243bd0820f-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-6rr4f" (UID: "2accd775-2173-4bfe-a413-7d243bd0820f") : secret "networking-console-plugin-cert" not found Oct 29 22:33:29.374241 ip-10-0-140-74 kubenswrapper[2577]: E1029 22:33:29.373965 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71d80eec-b5e7-41d0-b58d-8a8d9a6c769c-metrics-certs podName:71d80eec-b5e7-41d0-b58d-8a8d9a6c769c nodeName:}" failed. No retries permitted until 2026-10-29 22:33:31.373955752 +0000 UTC m=+33.531143419 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/71d80eec-b5e7-41d0-b58d-8a8d9a6c769c-metrics-certs") pod "router-default-5d8d7cc579-8cwjp" (UID: "71d80eec-b5e7-41d0-b58d-8a8d9a6c769c") : secret "router-metrics-certs-default" not found Oct 29 22:33:29.374241 ip-10-0-140-74 kubenswrapper[2577]: E1029 22:33:29.373983 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/989956e8-1893-4bbe-a4bf-6a2081509080-registry-tls podName:989956e8-1893-4bbe-a4bf-6a2081509080 nodeName:}" failed. No retries permitted until 2026-10-29 22:33:31.373973402 +0000 UTC m=+33.531161068 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/989956e8-1893-4bbe-a4bf-6a2081509080-registry-tls") pod "image-registry-84866f86bd-2v6d7" (UID: "989956e8-1893-4bbe-a4bf-6a2081509080") : secret "image-registry-tls" not found Oct 29 22:33:29.374241 ip-10-0-140-74 kubenswrapper[2577]: E1029 22:33:29.374002 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5296979a-7927-4ee2-84b9-a93780a2ad22-registry-tls podName:5296979a-7927-4ee2-84b9-a93780a2ad22 nodeName:}" failed. No retries permitted until 2026-10-29 22:33:31.373993616 +0000 UTC m=+33.531181277 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5296979a-7927-4ee2-84b9-a93780a2ad22-registry-tls") pod "image-registry-78db97464f-6ljzh" (UID: "5296979a-7927-4ee2-84b9-a93780a2ad22") : secret "image-registry-tls" not found Oct 29 22:33:29.374241 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:29.374065 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a53078b9-a6a8-423c-b852-34eececa7ab1-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-p2qv4\" (UID: \"a53078b9-a6a8-423c-b852-34eececa7ab1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-p2qv4" Oct 29 22:33:29.374241 ip-10-0-140-74 kubenswrapper[2577]: E1029 22:33:29.374177 2577 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 29 22:33:29.374241 ip-10-0-140-74 kubenswrapper[2577]: E1029 22:33:29.374217 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a53078b9-a6a8-423c-b852-34eececa7ab1-samples-operator-tls podName:a53078b9-a6a8-423c-b852-34eececa7ab1 nodeName:}" failed. No retries permitted until 2026-10-29 22:33:31.37420612 +0000 UTC m=+33.531393804 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/a53078b9-a6a8-423c-b852-34eececa7ab1-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-p2qv4" (UID: "a53078b9-a6a8-423c-b852-34eececa7ab1") : secret "samples-operator-tls" not found Oct 29 22:33:29.475793 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:29.474991 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/641f9ec1-c8e2-4774-9d4f-6a7a2da97b6f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-8c5hr\" (UID: \"641f9ec1-c8e2-4774-9d4f-6a7a2da97b6f\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-8c5hr" Oct 29 22:33:29.475793 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:29.475083 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6b34b3b1-84af-45f3-92ee-1ca720e8b263-metrics-tls\") pod \"dns-default-jjz47\" (UID: \"6b34b3b1-84af-45f3-92ee-1ca720e8b263\") " pod="openshift-dns/dns-default-jjz47" Oct 29 22:33:29.475793 ip-10-0-140-74 kubenswrapper[2577]: E1029 22:33:29.475352 2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 29 22:33:29.475793 ip-10-0-140-74 kubenswrapper[2577]: E1029 22:33:29.475409 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b34b3b1-84af-45f3-92ee-1ca720e8b263-metrics-tls podName:6b34b3b1-84af-45f3-92ee-1ca720e8b263 nodeName:}" failed. No retries permitted until 2026-10-29 22:33:31.475392431 +0000 UTC m=+33.632580106 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6b34b3b1-84af-45f3-92ee-1ca720e8b263-metrics-tls") pod "dns-default-jjz47" (UID: "6b34b3b1-84af-45f3-92ee-1ca720e8b263") : secret "dns-default-metrics-tls" not found Oct 29 22:33:29.475793 ip-10-0-140-74 kubenswrapper[2577]: E1029 22:33:29.475710 2577 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 29 22:33:29.475793 ip-10-0-140-74 kubenswrapper[2577]: E1029 22:33:29.475760 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/641f9ec1-c8e2-4774-9d4f-6a7a2da97b6f-cluster-monitoring-operator-tls podName:641f9ec1-c8e2-4774-9d4f-6a7a2da97b6f nodeName:}" failed. No retries permitted until 2026-10-29 22:33:31.475745644 +0000 UTC m=+33.632933331 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/641f9ec1-c8e2-4774-9d4f-6a7a2da97b6f-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-8c5hr" (UID: "641f9ec1-c8e2-4774-9d4f-6a7a2da97b6f") : secret "cluster-monitoring-operator-tls" not found Oct 29 22:33:29.515166 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:29.515134 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:33:29.636686 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:29.636612 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:33:31.396694 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:31.396250 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a53078b9-a6a8-423c-b852-34eececa7ab1-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-p2qv4\" (UID: \"a53078b9-a6a8-423c-b852-34eececa7ab1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-p2qv4" Oct 29 22:33:31.396694 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:31.396312 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5296979a-7927-4ee2-84b9-a93780a2ad22-registry-tls\") pod \"image-registry-78db97464f-6ljzh\" (UID: \"5296979a-7927-4ee2-84b9-a93780a2ad22\") " pod="openshift-image-registry/image-registry-78db97464f-6ljzh" Oct 29 22:33:31.396694 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:31.396397 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/71d80eec-b5e7-41d0-b58d-8a8d9a6c769c-service-ca-bundle\") pod \"router-default-5d8d7cc579-8cwjp\" (UID: \"71d80eec-b5e7-41d0-b58d-8a8d9a6c769c\") " pod="openshift-ingress/router-default-5d8d7cc579-8cwjp" Oct 29 22:33:31.396694 ip-10-0-140-74 kubenswrapper[2577]: E1029 22:33:31.396433 2577 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 29 22:33:31.396694 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:31.396457 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2accd775-2173-4bfe-a413-7d243bd0820f-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-6rr4f\" (UID: \"2accd775-2173-4bfe-a413-7d243bd0820f\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-6rr4f" Oct 29 22:33:31.396694 ip-10-0-140-74 kubenswrapper[2577]: E1029 22:33:31.396498 2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 29 22:33:31.396694 ip-10-0-140-74 kubenswrapper[2577]: E1029 22:33:31.396511 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a53078b9-a6a8-423c-b852-34eececa7ab1-samples-operator-tls podName:a53078b9-a6a8-423c-b852-34eececa7ab1 nodeName:}" failed. No retries permitted until 2026-10-29 22:33:35.396490761 +0000 UTC m=+37.553678437 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/a53078b9-a6a8-423c-b852-34eececa7ab1-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-p2qv4" (UID: "a53078b9-a6a8-423c-b852-34eececa7ab1") : secret "samples-operator-tls" not found Oct 29 22:33:31.396694 ip-10-0-140-74 kubenswrapper[2577]: E1029 22:33:31.396515 2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-78db97464f-6ljzh: secret "image-registry-tls" not found Oct 29 22:33:31.396694 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:31.396537 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71d80eec-b5e7-41d0-b58d-8a8d9a6c769c-metrics-certs\") pod \"router-default-5d8d7cc579-8cwjp\" (UID: \"71d80eec-b5e7-41d0-b58d-8a8d9a6c769c\") " pod="openshift-ingress/router-default-5d8d7cc579-8cwjp" Oct 29 22:33:31.396694 ip-10-0-140-74 kubenswrapper[2577]: E1029 22:33:31.396553 2577 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 29 22:33:31.396694 ip-10-0-140-74 kubenswrapper[2577]: E1029 22:33:31.396567 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5296979a-7927-4ee2-84b9-a93780a2ad22-registry-tls podName:5296979a-7927-4ee2-84b9-a93780a2ad22 nodeName:}" failed. No retries permitted until 2026-10-29 22:33:35.396550341 +0000 UTC m=+37.553738024 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5296979a-7927-4ee2-84b9-a93780a2ad22-registry-tls") pod "image-registry-78db97464f-6ljzh" (UID: "5296979a-7927-4ee2-84b9-a93780a2ad22") : secret "image-registry-tls" not found Oct 29 22:33:31.396694 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:31.396589 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/989956e8-1893-4bbe-a4bf-6a2081509080-registry-tls\") pod \"image-registry-84866f86bd-2v6d7\" (UID: \"989956e8-1893-4bbe-a4bf-6a2081509080\") " pod="openshift-image-registry/image-registry-84866f86bd-2v6d7" Oct 29 22:33:31.396694 ip-10-0-140-74 kubenswrapper[2577]: E1029 22:33:31.396611 2577 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 29 22:33:31.396694 ip-10-0-140-74 kubenswrapper[2577]: E1029 22:33:31.396635 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/71d80eec-b5e7-41d0-b58d-8a8d9a6c769c-service-ca-bundle podName:71d80eec-b5e7-41d0-b58d-8a8d9a6c769c nodeName:}" failed. No retries permitted until 2026-10-29 22:33:35.396625155 +0000 UTC m=+37.553812835 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/71d80eec-b5e7-41d0-b58d-8a8d9a6c769c-service-ca-bundle") pod "router-default-5d8d7cc579-8cwjp" (UID: "71d80eec-b5e7-41d0-b58d-8a8d9a6c769c") : configmap references non-existent config key: service-ca.crt Oct 29 22:33:31.396694 ip-10-0-140-74 kubenswrapper[2577]: E1029 22:33:31.396651 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2accd775-2173-4bfe-a413-7d243bd0820f-networking-console-plugin-cert podName:2accd775-2173-4bfe-a413-7d243bd0820f nodeName:}" failed. No retries permitted until 2026-10-29 22:33:35.396643463 +0000 UTC m=+37.553831125 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/2accd775-2173-4bfe-a413-7d243bd0820f-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-6rr4f" (UID: "2accd775-2173-4bfe-a413-7d243bd0820f") : secret "networking-console-plugin-cert" not found Oct 29 22:33:31.397684 ip-10-0-140-74 kubenswrapper[2577]: E1029 22:33:31.396665 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71d80eec-b5e7-41d0-b58d-8a8d9a6c769c-metrics-certs podName:71d80eec-b5e7-41d0-b58d-8a8d9a6c769c nodeName:}" failed. No retries permitted until 2026-10-29 22:33:35.396656851 +0000 UTC m=+37.553844518 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/71d80eec-b5e7-41d0-b58d-8a8d9a6c769c-metrics-certs") pod "router-default-5d8d7cc579-8cwjp" (UID: "71d80eec-b5e7-41d0-b58d-8a8d9a6c769c") : secret "router-metrics-certs-default" not found Oct 29 22:33:31.397684 ip-10-0-140-74 kubenswrapper[2577]: E1029 22:33:31.396730 2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 29 22:33:31.397684 ip-10-0-140-74 kubenswrapper[2577]: E1029 22:33:31.396740 2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-84866f86bd-2v6d7: secret "image-registry-tls" not found Oct 29 22:33:31.397684 ip-10-0-140-74 kubenswrapper[2577]: E1029 22:33:31.396769 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/989956e8-1893-4bbe-a4bf-6a2081509080-registry-tls podName:989956e8-1893-4bbe-a4bf-6a2081509080 nodeName:}" failed. No retries permitted until 2026-10-29 22:33:35.396760245 +0000 UTC m=+37.553947906 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/989956e8-1893-4bbe-a4bf-6a2081509080-registry-tls") pod "image-registry-84866f86bd-2v6d7" (UID: "989956e8-1893-4bbe-a4bf-6a2081509080") : secret "image-registry-tls" not found Oct 29 22:33:31.497088 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:31.497005 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/641f9ec1-c8e2-4774-9d4f-6a7a2da97b6f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-8c5hr\" (UID: \"641f9ec1-c8e2-4774-9d4f-6a7a2da97b6f\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-8c5hr" Oct 29 22:33:31.497250 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:31.497093 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6b34b3b1-84af-45f3-92ee-1ca720e8b263-metrics-tls\") pod \"dns-default-jjz47\" (UID: \"6b34b3b1-84af-45f3-92ee-1ca720e8b263\") " pod="openshift-dns/dns-default-jjz47" Oct 29 22:33:31.497250 ip-10-0-140-74 kubenswrapper[2577]: E1029 22:33:31.497152 2577 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 29 22:33:31.497250 ip-10-0-140-74 kubenswrapper[2577]: E1029 22:33:31.497211 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/641f9ec1-c8e2-4774-9d4f-6a7a2da97b6f-cluster-monitoring-operator-tls podName:641f9ec1-c8e2-4774-9d4f-6a7a2da97b6f nodeName:}" failed. No retries permitted until 2026-10-29 22:33:35.49719422 +0000 UTC m=+37.654381892 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/641f9ec1-c8e2-4774-9d4f-6a7a2da97b6f-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-8c5hr" (UID: "641f9ec1-c8e2-4774-9d4f-6a7a2da97b6f") : secret "cluster-monitoring-operator-tls" not found Oct 29 22:33:31.497250 ip-10-0-140-74 kubenswrapper[2577]: E1029 22:33:31.497246 2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 29 22:33:31.497515 ip-10-0-140-74 kubenswrapper[2577]: E1029 22:33:31.497279 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b34b3b1-84af-45f3-92ee-1ca720e8b263-metrics-tls podName:6b34b3b1-84af-45f3-92ee-1ca720e8b263 nodeName:}" failed. No retries permitted until 2026-10-29 22:33:35.497268953 +0000 UTC m=+37.654456630 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6b34b3b1-84af-45f3-92ee-1ca720e8b263-metrics-tls") pod "dns-default-jjz47" (UID: "6b34b3b1-84af-45f3-92ee-1ca720e8b263") : secret "dns-default-metrics-tls" not found Oct 29 22:33:31.634291 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:31.633983 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-w5d4m" event={"ID":"c54ef754-c91e-4bc5-8e83-08523aa09d37","Type":"ContainerStarted","Data":"55d08e8ce4360732fa1e7c4a2ba268359d62833659b71013a03375ae74958122"} Oct 29 22:33:31.783613 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:31.783554 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:33:31.912280 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:31.912241 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:33:32.106052 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:32.105252 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f8f92bfb-5cef-412e-a6c1-4bd793c59fe7-metrics-certs\") pod \"network-metrics-daemon-hzlmg\" (UID: \"f8f92bfb-5cef-412e-a6c1-4bd793c59fe7\") " pod="openshift-multus/network-metrics-daemon-hzlmg" Oct 29 22:33:32.106052 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:32.105357 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6f50ee57-a710-4325-bd45-4f861452ede4-original-pull-secret\") pod \"global-pull-secret-syncer-bp46g\" (UID: \"6f50ee57-a710-4325-bd45-4f861452ede4\") " pod="kube-system/global-pull-secret-syncer-bp46g" Oct 29 22:33:32.106052 ip-10-0-140-74 kubenswrapper[2577]: E1029 22:33:32.106002 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 29 22:33:32.106434 ip-10-0-140-74 kubenswrapper[2577]: E1029 22:33:32.106071 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f8f92bfb-5cef-412e-a6c1-4bd793c59fe7-metrics-certs podName:f8f92bfb-5cef-412e-a6c1-4bd793c59fe7 nodeName:}" failed. No retries permitted until 2026-10-29 22:34:04.106052207 +0000 UTC m=+66.263239873 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f8f92bfb-5cef-412e-a6c1-4bd793c59fe7-metrics-certs") pod "network-metrics-daemon-hzlmg" (UID: "f8f92bfb-5cef-412e-a6c1-4bd793c59fe7") : secret "metrics-daemon-secret" not found Oct 29 22:33:32.112610 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:32.112555 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6f50ee57-a710-4325-bd45-4f861452ede4-original-pull-secret\") pod \"global-pull-secret-syncer-bp46g\" (UID: \"6f50ee57-a710-4325-bd45-4f861452ede4\") " pod="kube-system/global-pull-secret-syncer-bp46g" Oct 29 22:33:32.206207 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:32.206170 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9srv\" (UniqueName: \"kubernetes.io/projected/a4e1c52c-1eeb-4617-a0c9-d052521360d3-kube-api-access-l9srv\") pod \"network-check-target-7jv5g\" (UID: \"a4e1c52c-1eeb-4617-a0c9-d052521360d3\") " pod="openshift-network-diagnostics/network-check-target-7jv5g" Oct 29 22:33:32.216879 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:32.216854 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9srv\" (UniqueName: \"kubernetes.io/projected/a4e1c52c-1eeb-4617-a0c9-d052521360d3-kube-api-access-l9srv\") pod \"network-check-target-7jv5g\" (UID: \"a4e1c52c-1eeb-4617-a0c9-d052521360d3\") " pod="openshift-network-diagnostics/network-check-target-7jv5g" Oct 29 22:33:32.377623 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:32.377537 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bp46g" Oct 29 22:33:32.394194 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:32.393975 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7jv5g" Oct 29 22:33:32.639019 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:32.638953 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-vj6md_aa6a190e-3283-458a-9dfd-2de7cb193fc5/console-operator/0.log" Oct 29 22:33:32.639019 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:32.638993 2577 generic.go:356] "Generic (PLEG): container finished" podID="aa6a190e-3283-458a-9dfd-2de7cb193fc5" containerID="a1f1f3830c569aebd82cc9d513fe278b3aded44e6a1e067c58131d8879a597d0" exitCode=255 Oct 29 22:33:32.639542 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:32.639023 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-vj6md" event={"ID":"aa6a190e-3283-458a-9dfd-2de7cb193fc5","Type":"ContainerDied","Data":"a1f1f3830c569aebd82cc9d513fe278b3aded44e6a1e067c58131d8879a597d0"} Oct 29 22:33:32.639542 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:32.639299 2577 scope.go:122] "RemoveContainer" containerID="a1f1f3830c569aebd82cc9d513fe278b3aded44e6a1e067c58131d8879a597d0" Oct 29 22:33:32.677092 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:32.677034 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-w5d4m" podStartSLOduration=1007.493334748 podStartE2EDuration="16m49.677016012s" totalImagesPullingTime="2.183681264s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:16:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:33:31.670013713 +0000 UTC m=+33.827201409" watchObservedRunningTime="2026-10-29 22:33:32.677016012 +0000 UTC m=+34.834203696" Oct 29 22:33:32.731167 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:32.730650 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-bp46g"] Oct 29 22:33:32.735431 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:33:32.735326 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f50ee57_a710_4325_bd45_4f861452ede4.slice/crio-d7df54a9e709ebfabcf14a3ad76c25faf0cc254389004e41ba7f1eded00b1a80 WatchSource:0}: Error finding container d7df54a9e709ebfabcf14a3ad76c25faf0cc254389004e41ba7f1eded00b1a80: Status 404 returned error can't find the container with id d7df54a9e709ebfabcf14a3ad76c25faf0cc254389004e41ba7f1eded00b1a80 Oct 29 22:33:33.028984 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:33.028947 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-7jv5g"] Oct 29 22:33:33.037728 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:33:33.037694 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4e1c52c_1eeb_4617_a0c9_d052521360d3.slice/crio-f2028b5b83ac2c14a3e4c4b56a439b3dcf82bf043749a960aa5867a0992c7837 WatchSource:0}: Error finding container f2028b5b83ac2c14a3e4c4b56a439b3dcf82bf043749a960aa5867a0992c7837: Status 404 returned error can't find the container with id f2028b5b83ac2c14a3e4c4b56a439b3dcf82bf043749a960aa5867a0992c7837 Oct 29 22:33:33.561896 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:33.561830 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-d2dnn"] Oct 29 22:33:33.579259 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:33.579229 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-d2dnn" Oct 29 22:33:33.584645 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:33.584619 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-d2dnn"] Oct 29 22:33:33.591542 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:33.591516 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 29 22:33:33.591631 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:33.591524 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 29 22:33:33.591671 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:33.591629 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-c4xfj\"" Oct 29 22:33:33.622257 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:33.622227 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ln6mw\" (UniqueName: \"kubernetes.io/projected/079ed4e4-bdc4-405c-90a1-a22b84fcf948-kube-api-access-ln6mw\") pod \"insights-runtime-extractor-d2dnn\" (UID: \"079ed4e4-bdc4-405c-90a1-a22b84fcf948\") " pod="openshift-insights/insights-runtime-extractor-d2dnn" Oct 29 22:33:33.622397 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:33.622279 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/079ed4e4-bdc4-405c-90a1-a22b84fcf948-crio-socket\") pod \"insights-runtime-extractor-d2dnn\" (UID: \"079ed4e4-bdc4-405c-90a1-a22b84fcf948\") " pod="openshift-insights/insights-runtime-extractor-d2dnn" Oct 29 22:33:33.622472 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:33.622404 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/079ed4e4-bdc4-405c-90a1-a22b84fcf948-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-d2dnn\" (UID: \"079ed4e4-bdc4-405c-90a1-a22b84fcf948\") " pod="openshift-insights/insights-runtime-extractor-d2dnn" Oct 29 22:33:33.622606 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:33.622580 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/079ed4e4-bdc4-405c-90a1-a22b84fcf948-data-volume\") pod \"insights-runtime-extractor-d2dnn\" (UID: \"079ed4e4-bdc4-405c-90a1-a22b84fcf948\") " pod="openshift-insights/insights-runtime-extractor-d2dnn" Oct 29 22:33:33.622707 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:33.622611 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/079ed4e4-bdc4-405c-90a1-a22b84fcf948-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-d2dnn\" (UID: \"079ed4e4-bdc4-405c-90a1-a22b84fcf948\") " pod="openshift-insights/insights-runtime-extractor-d2dnn" Oct 29 22:33:33.630443 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:33.630418 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-lnrkb-btcvq"] Oct 29 22:33:33.640194 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:33.640174 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-lnrkb-btcvq" Oct 29 22:33:33.646889 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:33.646859 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-pk97l" event={"ID":"c635d035-053b-49aa-9559-79c31e8e49ff","Type":"ContainerStarted","Data":"a3fb0a65f00623c0a7fbc591def44660ef1faa56e5f686dc9671c0cebfecf081"} Oct 29 22:33:33.649852 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:33.649809 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-b7bwd" event={"ID":"65c96dfb-06e9-4bea-bb83-f63b4c62d860","Type":"ContainerStarted","Data":"9bd7d26c51b8fd48c4f3e9eb445aebd730ae353e8e68dc8efd4cd906f797088e"} Oct 29 22:33:33.650882 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:33.650856 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-lnrkb-btcvq"] Oct 29 22:33:33.660261 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:33.660197 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:33:33.672817 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:33.672797 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-vj6md_aa6a190e-3283-458a-9dfd-2de7cb193fc5/console-operator/1.log" Oct 29 22:33:33.673298 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:33.673279 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-vj6md_aa6a190e-3283-458a-9dfd-2de7cb193fc5/console-operator/0.log" Oct 29 22:33:33.674206 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:33.673313 2577 generic.go:356] "Generic (PLEG): container finished" podID="aa6a190e-3283-458a-9dfd-2de7cb193fc5" containerID="e5ab727c17d3a45185a300b3b2a3688775b919a83c6e93f3a510f87ee56a0fb7" exitCode=255 Oct 29 22:33:33.674206 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:33.673421 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-vj6md" event={"ID":"aa6a190e-3283-458a-9dfd-2de7cb193fc5","Type":"ContainerDied","Data":"e5ab727c17d3a45185a300b3b2a3688775b919a83c6e93f3a510f87ee56a0fb7"} Oct 29 22:33:33.674206 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:33.673473 2577 scope.go:122] "RemoveContainer" containerID="a1f1f3830c569aebd82cc9d513fe278b3aded44e6a1e067c58131d8879a597d0" Oct 29 22:33:33.674206 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:33.673643 2577 scope.go:122] "RemoveContainer" containerID="e5ab727c17d3a45185a300b3b2a3688775b919a83c6e93f3a510f87ee56a0fb7" Oct 29 22:33:33.674206 ip-10-0-140-74 kubenswrapper[2577]: E1029 22:33:33.673892 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-vj6md_openshift-console-operator(aa6a190e-3283-458a-9dfd-2de7cb193fc5)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-vj6md" podUID="aa6a190e-3283-458a-9dfd-2de7cb193fc5" Oct 29 22:33:33.677130 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:33.677084 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-bp46g" event={"ID":"6f50ee57-a710-4325-bd45-4f861452ede4","Type":"ContainerStarted","Data":"d7df54a9e709ebfabcf14a3ad76c25faf0cc254389004e41ba7f1eded00b1a80"} Oct 29 22:33:33.678704 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:33.678661 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-sk5qq" event={"ID":"4071b0c7-c4f3-419a-8d6e-6acc144fc9a4","Type":"ContainerStarted","Data":"1eb5a229ae3b60cb6a6342d3cfa649361c63bcc57b0708eb0d314b868634e43d"} Oct 29 22:33:33.680200 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:33.680175 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-7jv5g" event={"ID":"a4e1c52c-1eeb-4617-a0c9-d052521360d3","Type":"ContainerStarted","Data":"f2028b5b83ac2c14a3e4c4b56a439b3dcf82bf043749a960aa5867a0992c7837"} Oct 29 22:33:33.723957 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:33.723934 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ln6mw\" (UniqueName: \"kubernetes.io/projected/079ed4e4-bdc4-405c-90a1-a22b84fcf948-kube-api-access-ln6mw\") pod \"insights-runtime-extractor-d2dnn\" (UID: \"079ed4e4-bdc4-405c-90a1-a22b84fcf948\") " pod="openshift-insights/insights-runtime-extractor-d2dnn" Oct 29 22:33:33.724043 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:33.723964 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/f176efe0-febd-4652-94fc-91548ee3ab93-archives-path\") pod \"periodic-gathering-lnrkb-btcvq\" (UID: \"f176efe0-febd-4652-94fc-91548ee3ab93\") " pod="openshift-insights/periodic-gathering-lnrkb-btcvq" Oct 29 22:33:33.724043 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:33.723980 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9pvq\" (UniqueName: \"kubernetes.io/projected/f176efe0-febd-4652-94fc-91548ee3ab93-kube-api-access-p9pvq\") pod \"periodic-gathering-lnrkb-btcvq\" (UID: \"f176efe0-febd-4652-94fc-91548ee3ab93\") " pod="openshift-insights/periodic-gathering-lnrkb-btcvq" Oct 29 22:33:33.724154 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:33.724052 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/079ed4e4-bdc4-405c-90a1-a22b84fcf948-crio-socket\") pod \"insights-runtime-extractor-d2dnn\" (UID: \"079ed4e4-bdc4-405c-90a1-a22b84fcf948\") " pod="openshift-insights/insights-runtime-extractor-d2dnn" Oct 29 22:33:33.724154 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:33.724083 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/079ed4e4-bdc4-405c-90a1-a22b84fcf948-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-d2dnn\" (UID: \"079ed4e4-bdc4-405c-90a1-a22b84fcf948\") " pod="openshift-insights/insights-runtime-extractor-d2dnn" Oct 29 22:33:33.724154 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:33.724140 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f176efe0-febd-4652-94fc-91548ee3ab93-trusted-ca-bundle\") pod \"periodic-gathering-lnrkb-btcvq\" (UID: \"f176efe0-febd-4652-94fc-91548ee3ab93\") " pod="openshift-insights/periodic-gathering-lnrkb-btcvq" Oct 29 22:33:33.724298 ip-10-0-140-74 kubenswrapper[2577]: E1029 22:33:33.724178 2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 29 22:33:33.724298 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:33.724204 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/079ed4e4-bdc4-405c-90a1-a22b84fcf948-crio-socket\") pod \"insights-runtime-extractor-d2dnn\" (UID: \"079ed4e4-bdc4-405c-90a1-a22b84fcf948\") " pod="openshift-insights/insights-runtime-extractor-d2dnn" Oct 29 22:33:33.724298 ip-10-0-140-74 kubenswrapper[2577]: E1029 22:33:33.724261 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/079ed4e4-bdc4-405c-90a1-a22b84fcf948-insights-runtime-extractor-tls podName:079ed4e4-bdc4-405c-90a1-a22b84fcf948 nodeName:}" failed. No retries permitted until 2026-10-29 22:33:34.224242625 +0000 UTC m=+36.381430308 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/079ed4e4-bdc4-405c-90a1-a22b84fcf948-insights-runtime-extractor-tls") pod "insights-runtime-extractor-d2dnn" (UID: "079ed4e4-bdc4-405c-90a1-a22b84fcf948") : secret "insights-runtime-extractor-tls" not found Oct 29 22:33:33.724654 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:33.724631 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/079ed4e4-bdc4-405c-90a1-a22b84fcf948-data-volume\") pod \"insights-runtime-extractor-d2dnn\" (UID: \"079ed4e4-bdc4-405c-90a1-a22b84fcf948\") " pod="openshift-insights/insights-runtime-extractor-d2dnn" Oct 29 22:33:33.724714 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:33.724667 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/079ed4e4-bdc4-405c-90a1-a22b84fcf948-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-d2dnn\" (UID: \"079ed4e4-bdc4-405c-90a1-a22b84fcf948\") " pod="openshift-insights/insights-runtime-extractor-d2dnn" Oct 29 22:33:33.724714 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:33.724698 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f176efe0-febd-4652-94fc-91548ee3ab93-service-ca-bundle\") pod \"periodic-gathering-lnrkb-btcvq\" (UID: \"f176efe0-febd-4652-94fc-91548ee3ab93\") " pod="openshift-insights/periodic-gathering-lnrkb-btcvq" Oct 29 22:33:33.725119 ip-10-0-140-74 kubenswrapper[2577]: E1029 22:33:33.725099 2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 29 22:33:33.725261 ip-10-0-140-74 kubenswrapper[2577]: E1029 22:33:33.725244 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/079ed4e4-bdc4-405c-90a1-a22b84fcf948-kube-rbac-proxy-cm podName:079ed4e4-bdc4-405c-90a1-a22b84fcf948 nodeName:}" failed. No retries permitted until 2026-10-29 22:33:34.225163243 +0000 UTC m=+36.382350923 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/079ed4e4-bdc4-405c-90a1-a22b84fcf948-kube-rbac-proxy-cm") pod "insights-runtime-extractor-d2dnn" (UID: "079ed4e4-bdc4-405c-90a1-a22b84fcf948") : configmap "kube-rbac-proxy" not found Oct 29 22:33:33.725551 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:33.725534 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/079ed4e4-bdc4-405c-90a1-a22b84fcf948-data-volume\") pod \"insights-runtime-extractor-d2dnn\" (UID: \"079ed4e4-bdc4-405c-90a1-a22b84fcf948\") " pod="openshift-insights/insights-runtime-extractor-d2dnn" Oct 29 22:33:33.745221 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:33.745201 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ln6mw\" (UniqueName: \"kubernetes.io/projected/079ed4e4-bdc4-405c-90a1-a22b84fcf948-kube-api-access-ln6mw\") pod \"insights-runtime-extractor-d2dnn\" (UID: \"079ed4e4-bdc4-405c-90a1-a22b84fcf948\") " pod="openshift-insights/insights-runtime-extractor-d2dnn" Oct 29 22:33:33.750428 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:33.750387 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-operator-7c4986d449-b7bwd" podStartSLOduration=1004.445025867 podStartE2EDuration="16m48.750370944s" totalImagesPullingTime="4.305345077s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:16:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:33:33.749704251 +0000 UTC m=+35.906891954" watchObservedRunningTime="2026-10-29 22:33:33.750370944 +0000 UTC m=+35.907558633" Oct 29 22:33:33.770283 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:33.770211 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:33:33.787487 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:33.786027 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-pk97l" podStartSLOduration=1000.910001005 podStartE2EDuration="16m45.786011853s" totalImagesPullingTime="4.876010848s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:16:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:33:33.784226552 +0000 UTC m=+35.941414237" watchObservedRunningTime="2026-10-29 22:33:33.786011853 +0000 UTC m=+35.943199538" Oct 29 22:33:33.823325 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:33.823236 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-sk5qq" podStartSLOduration=1004.027602361 podStartE2EDuration="16m48.823215544s" totalImagesPullingTime="4.795613183s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:16:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:33:33.821695045 +0000 UTC m=+35.978882757" watchObservedRunningTime="2026-10-29 22:33:33.823215544 +0000 UTC m=+35.980403228" Oct 29 22:33:33.825529 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:33.825499 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f176efe0-febd-4652-94fc-91548ee3ab93-service-ca-bundle\") pod \"periodic-gathering-lnrkb-btcvq\" (UID: \"f176efe0-febd-4652-94fc-91548ee3ab93\") " pod="openshift-insights/periodic-gathering-lnrkb-btcvq" Oct 29 22:33:33.825629 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:33.825586 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/f176efe0-febd-4652-94fc-91548ee3ab93-archives-path\") pod \"periodic-gathering-lnrkb-btcvq\" (UID: \"f176efe0-febd-4652-94fc-91548ee3ab93\") " pod="openshift-insights/periodic-gathering-lnrkb-btcvq" Oct 29 22:33:33.825629 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:33.825620 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9pvq\" (UniqueName: \"kubernetes.io/projected/f176efe0-febd-4652-94fc-91548ee3ab93-kube-api-access-p9pvq\") pod \"periodic-gathering-lnrkb-btcvq\" (UID: \"f176efe0-febd-4652-94fc-91548ee3ab93\") " pod="openshift-insights/periodic-gathering-lnrkb-btcvq" Oct 29 22:33:33.825743 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:33.825704 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f176efe0-febd-4652-94fc-91548ee3ab93-trusted-ca-bundle\") pod \"periodic-gathering-lnrkb-btcvq\" (UID: \"f176efe0-febd-4652-94fc-91548ee3ab93\") " pod="openshift-insights/periodic-gathering-lnrkb-btcvq" Oct 29 22:33:33.826472 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:33.826175 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/f176efe0-febd-4652-94fc-91548ee3ab93-archives-path\") pod \"periodic-gathering-lnrkb-btcvq\" (UID: \"f176efe0-febd-4652-94fc-91548ee3ab93\") " pod="openshift-insights/periodic-gathering-lnrkb-btcvq" Oct 29 22:33:33.826588 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:33.826499 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f176efe0-febd-4652-94fc-91548ee3ab93-service-ca-bundle\") pod \"periodic-gathering-lnrkb-btcvq\" (UID: \"f176efe0-febd-4652-94fc-91548ee3ab93\") " pod="openshift-insights/periodic-gathering-lnrkb-btcvq" Oct 29 22:33:33.827624 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:33.827603 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f176efe0-febd-4652-94fc-91548ee3ab93-trusted-ca-bundle\") pod \"periodic-gathering-lnrkb-btcvq\" (UID: \"f176efe0-febd-4652-94fc-91548ee3ab93\") " pod="openshift-insights/periodic-gathering-lnrkb-btcvq" Oct 29 22:33:33.849675 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:33.849632 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9pvq\" (UniqueName: \"kubernetes.io/projected/f176efe0-febd-4652-94fc-91548ee3ab93-kube-api-access-p9pvq\") pod \"periodic-gathering-lnrkb-btcvq\" (UID: \"f176efe0-febd-4652-94fc-91548ee3ab93\") " pod="openshift-insights/periodic-gathering-lnrkb-btcvq" Oct 29 22:33:33.973711 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:33.973506 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-lnrkb-btcvq" Oct 29 22:33:34.093370 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:34.093265 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:33:34.212882 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:34.212846 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:33:34.229169 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:34.229143 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/079ed4e4-bdc4-405c-90a1-a22b84fcf948-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-d2dnn\" (UID: \"079ed4e4-bdc4-405c-90a1-a22b84fcf948\") " pod="openshift-insights/insights-runtime-extractor-d2dnn" Oct 29 22:33:34.229286 ip-10-0-140-74 kubenswrapper[2577]: E1029 22:33:34.229244 2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 29 22:33:34.229355 ip-10-0-140-74 kubenswrapper[2577]: E1029 22:33:34.229306 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/079ed4e4-bdc4-405c-90a1-a22b84fcf948-kube-rbac-proxy-cm podName:079ed4e4-bdc4-405c-90a1-a22b84fcf948 nodeName:}" failed. No retries permitted until 2026-10-29 22:33:35.229289551 +0000 UTC m=+37.386477220 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/079ed4e4-bdc4-405c-90a1-a22b84fcf948-kube-rbac-proxy-cm") pod "insights-runtime-extractor-d2dnn" (UID: "079ed4e4-bdc4-405c-90a1-a22b84fcf948") : configmap "kube-rbac-proxy" not found Oct 29 22:33:34.229355 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:34.229242 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/079ed4e4-bdc4-405c-90a1-a22b84fcf948-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-d2dnn\" (UID: \"079ed4e4-bdc4-405c-90a1-a22b84fcf948\") " pod="openshift-insights/insights-runtime-extractor-d2dnn" Oct 29 22:33:34.229355 ip-10-0-140-74 kubenswrapper[2577]: E1029 22:33:34.229316 2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 29 22:33:34.229498 ip-10-0-140-74 kubenswrapper[2577]: E1029 22:33:34.229394 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/079ed4e4-bdc4-405c-90a1-a22b84fcf948-insights-runtime-extractor-tls podName:079ed4e4-bdc4-405c-90a1-a22b84fcf948 nodeName:}" failed. No retries permitted until 2026-10-29 22:33:35.229382223 +0000 UTC m=+37.386569899 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/079ed4e4-bdc4-405c-90a1-a22b84fcf948-insights-runtime-extractor-tls") pod "insights-runtime-extractor-d2dnn" (UID: "079ed4e4-bdc4-405c-90a1-a22b84fcf948") : secret "insights-runtime-extractor-tls" not found Oct 29 22:33:34.275034 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:34.274543 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-c7kcc"] Oct 29 22:33:34.300516 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:34.300493 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-vjlkr"] Oct 29 22:33:34.301186 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:34.300692 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-c7kcc" Oct 29 22:33:34.333322 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:34.322077 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Oct 29 22:33:34.333322 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:34.322873 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Oct 29 22:33:34.333322 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:34.323815 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-t9ks9\"" Oct 29 22:33:34.333322 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:34.331519 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hfdj9\" (UniqueName: \"kubernetes.io/projected/b5961d4b-f3c0-4f17-b1bc-2781cb3dcd6e-kube-api-access-hfdj9\") pod \"migrator-66df5d496b-c7kcc\" (UID: \"b5961d4b-f3c0-4f17-b1bc-2781cb3dcd6e\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-c7kcc" Oct 29 22:33:34.333322 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:34.333239 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:33:34.334746 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:34.333804 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-c7kcc"] Oct 29 22:33:34.334746 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:34.334457 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-vjlkr" Oct 29 22:33:34.346275 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:34.346221 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-vjlkr"] Oct 29 22:33:34.432641 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:34.432615 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfd8g\" (UniqueName: \"kubernetes.io/projected/652bedff-87f1-4b2f-917f-571a37dd2f3f-kube-api-access-mfd8g\") pod \"migrator-66df5d496b-vjlkr\" (UID: \"652bedff-87f1-4b2f-917f-571a37dd2f3f\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-vjlkr" Oct 29 22:33:34.432865 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:34.432834 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfdj9\" (UniqueName: \"kubernetes.io/projected/b5961d4b-f3c0-4f17-b1bc-2781cb3dcd6e-kube-api-access-hfdj9\") pod \"migrator-66df5d496b-c7kcc\" (UID: \"b5961d4b-f3c0-4f17-b1bc-2781cb3dcd6e\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-c7kcc" Oct 29 22:33:34.442003 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:34.441980 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:33:34.456613 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:34.456584 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hfdj9\" (UniqueName: \"kubernetes.io/projected/b5961d4b-f3c0-4f17-b1bc-2781cb3dcd6e-kube-api-access-hfdj9\") pod \"migrator-66df5d496b-c7kcc\" (UID: \"b5961d4b-f3c0-4f17-b1bc-2781cb3dcd6e\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-c7kcc" Oct 29 22:33:34.507615 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:34.507582 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-9xmdb"] Oct 29 22:33:34.527852 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:34.527821 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-9xmdb" Oct 29 22:33:34.531022 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:34.530998 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-9xmdb"] Oct 29 22:33:34.534493 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:34.534465 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/49c67ce3-90fd-478d-9a15-e4267438ff69-signing-cabundle\") pod \"service-ca-5fffc97bdf-9xmdb\" (UID: \"49c67ce3-90fd-478d-9a15-e4267438ff69\") " pod="openshift-service-ca/service-ca-5fffc97bdf-9xmdb" Oct 29 22:33:34.534597 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:34.534556 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/49c67ce3-90fd-478d-9a15-e4267438ff69-signing-key\") pod \"service-ca-5fffc97bdf-9xmdb\" (UID: \"49c67ce3-90fd-478d-9a15-e4267438ff69\") " pod="openshift-service-ca/service-ca-5fffc97bdf-9xmdb" Oct 29 22:33:34.534709 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:34.534681 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfd8g\" (UniqueName: \"kubernetes.io/projected/652bedff-87f1-4b2f-917f-571a37dd2f3f-kube-api-access-mfd8g\") pod \"migrator-66df5d496b-vjlkr\" (UID: \"652bedff-87f1-4b2f-917f-571a37dd2f3f\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-vjlkr" Oct 29 22:33:34.534763 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:34.534724 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qd7k\" (UniqueName: \"kubernetes.io/projected/49c67ce3-90fd-478d-9a15-e4267438ff69-kube-api-access-9qd7k\") pod \"service-ca-5fffc97bdf-9xmdb\" (UID: \"49c67ce3-90fd-478d-9a15-e4267438ff69\") " pod="openshift-service-ca/service-ca-5fffc97bdf-9xmdb" Oct 29 22:33:34.540718 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:34.540487 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Oct 29 22:33:34.540718 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:34.540601 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-ct6ft\"" Oct 29 22:33:34.540718 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:34.540693 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Oct 29 22:33:34.540928 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:34.540794 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Oct 29 22:33:34.541793 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:34.541758 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Oct 29 22:33:34.555626 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:34.555603 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfd8g\" (UniqueName: \"kubernetes.io/projected/652bedff-87f1-4b2f-917f-571a37dd2f3f-kube-api-access-mfd8g\") pod \"migrator-66df5d496b-vjlkr\" (UID: \"652bedff-87f1-4b2f-917f-571a37dd2f3f\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-vjlkr" Oct 29 22:33:34.557074 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:34.557045 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:33:34.588514 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:34.588495 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-lnrkb-btcvq"] Oct 29 22:33:34.591540 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:33:34.591516 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf176efe0_febd_4652_94fc_91548ee3ab93.slice/crio-137914b9aa694c6fd2799975b8439b50e6c4d98b6b27025bd3874923c51ad0c6 WatchSource:0}: Error finding container 137914b9aa694c6fd2799975b8439b50e6c4d98b6b27025bd3874923c51ad0c6: Status 404 returned error can't find the container with id 137914b9aa694c6fd2799975b8439b50e6c4d98b6b27025bd3874923c51ad0c6 Oct 29 22:33:34.636042 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:34.636011 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/49c67ce3-90fd-478d-9a15-e4267438ff69-signing-cabundle\") pod \"service-ca-5fffc97bdf-9xmdb\" (UID: \"49c67ce3-90fd-478d-9a15-e4267438ff69\") " pod="openshift-service-ca/service-ca-5fffc97bdf-9xmdb" Oct 29 22:33:34.636145 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:34.636070 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/49c67ce3-90fd-478d-9a15-e4267438ff69-signing-key\") pod \"service-ca-5fffc97bdf-9xmdb\" (UID: \"49c67ce3-90fd-478d-9a15-e4267438ff69\") " pod="openshift-service-ca/service-ca-5fffc97bdf-9xmdb" Oct 29 22:33:34.636305 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:34.636283 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qd7k\" (UniqueName: \"kubernetes.io/projected/49c67ce3-90fd-478d-9a15-e4267438ff69-kube-api-access-9qd7k\") pod \"service-ca-5fffc97bdf-9xmdb\" (UID: \"49c67ce3-90fd-478d-9a15-e4267438ff69\") " pod="openshift-service-ca/service-ca-5fffc97bdf-9xmdb" Oct 29 22:33:34.636878 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:34.636855 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/49c67ce3-90fd-478d-9a15-e4267438ff69-signing-cabundle\") pod \"service-ca-5fffc97bdf-9xmdb\" (UID: \"49c67ce3-90fd-478d-9a15-e4267438ff69\") " pod="openshift-service-ca/service-ca-5fffc97bdf-9xmdb" Oct 29 22:33:34.638588 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:34.638570 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/49c67ce3-90fd-478d-9a15-e4267438ff69-signing-key\") pod \"service-ca-5fffc97bdf-9xmdb\" (UID: \"49c67ce3-90fd-478d-9a15-e4267438ff69\") " pod="openshift-service-ca/service-ca-5fffc97bdf-9xmdb" Oct 29 22:33:34.657044 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:34.657022 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qd7k\" (UniqueName: \"kubernetes.io/projected/49c67ce3-90fd-478d-9a15-e4267438ff69-kube-api-access-9qd7k\") pod \"service-ca-5fffc97bdf-9xmdb\" (UID: \"49c67ce3-90fd-478d-9a15-e4267438ff69\") " pod="openshift-service-ca/service-ca-5fffc97bdf-9xmdb" Oct 29 22:33:34.685545 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:34.685516 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-lnrkb-btcvq" event={"ID":"f176efe0-febd-4652-94fc-91548ee3ab93","Type":"ContainerStarted","Data":"b7903026ceba7033f380cb6218731c4d3fc52045ff5b91e796c1a3658dd2219d"} Oct 29 22:33:34.685630 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:34.685572 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-lnrkb-btcvq" event={"ID":"f176efe0-febd-4652-94fc-91548ee3ab93","Type":"ContainerStarted","Data":"137914b9aa694c6fd2799975b8439b50e6c4d98b6b27025bd3874923c51ad0c6"} Oct 29 22:33:34.687294 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:34.687269 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-7jv5g" event={"ID":"a4e1c52c-1eeb-4617-a0c9-d052521360d3","Type":"ContainerStarted","Data":"7437fa4b26b871bf6a6f9dd7a723f5d1f8ad1d10465e72bba8550abfebf778f9"} Oct 29 22:33:34.687413 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:34.687364 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-7jv5g" Oct 29 22:33:34.689142 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:34.689120 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-p9ds4" event={"ID":"dd4e19ce-ef37-44e5-8b57-9dafc590685c","Type":"ContainerStarted","Data":"f28eb8000eab8fdee8a018926ee1431f560dcb2f0b807b4ce0a13d808e780d05"} Oct 29 22:33:34.689355 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:34.689312 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-c7kcc" Oct 29 22:33:34.690838 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:34.690821 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-vj6md_aa6a190e-3283-458a-9dfd-2de7cb193fc5/console-operator/1.log" Oct 29 22:33:34.691221 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:34.691103 2577 scope.go:122] "RemoveContainer" containerID="e5ab727c17d3a45185a300b3b2a3688775b919a83c6e93f3a510f87ee56a0fb7" Oct 29 22:33:34.691304 ip-10-0-140-74 kubenswrapper[2577]: E1029 22:33:34.691273 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-vj6md_openshift-console-operator(aa6a190e-3283-458a-9dfd-2de7cb193fc5)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-vj6md" podUID="aa6a190e-3283-458a-9dfd-2de7cb193fc5" Oct 29 22:33:34.692742 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:34.692690 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-czfl9" event={"ID":"8406bb9f-3a4e-42f3-a822-f754d9b8e7a8","Type":"ContainerStarted","Data":"db6036c89fc169a5310e7fdd8c3968ffabc2f67944bb8cd43dfc36df338a0ba6"} Oct 29 22:33:34.712395 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:34.712374 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-vjlkr" Oct 29 22:33:34.722721 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:34.722681 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-7jv5g" podStartSLOduration=35.676997893 podStartE2EDuration="36.722668856s" totalImagesPullingTime="1.045670963s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:32:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:33:34.721131387 +0000 UTC m=+36.878319071" watchObservedRunningTime="2026-10-29 22:33:34.722668856 +0000 UTC m=+36.879856532" Oct 29 22:33:34.800710 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:34.800664 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-p9ds4" podStartSLOduration=1006.027382356 podStartE2EDuration="16m51.800648034s" totalImagesPullingTime="5.773265678s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:16:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:33:34.800510496 +0000 UTC m=+36.957698179" watchObservedRunningTime="2026-10-29 22:33:34.800648034 +0000 UTC m=+36.957835716" Oct 29 22:33:34.822045 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:34.822020 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:33:34.844104 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:34.844053 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-czfl9" podStartSLOduration=2.611745972 podStartE2EDuration="7.844015185s" totalImagesPullingTime="5.232269213s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:33:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:33:34.841570659 +0000 UTC m=+36.998758369" watchObservedRunningTime="2026-10-29 22:33:34.844015185 +0000 UTC m=+37.001202870" Oct 29 22:33:34.857923 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:34.857858 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-9xmdb" Oct 29 22:33:34.932448 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:34.932416 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:33:35.011083 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:35.011054 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-c7kcc"] Oct 29 22:33:35.012224 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:33:35.012193 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5961d4b_f3c0_4f17_b1bc_2781cb3dcd6e.slice/crio-ffc8236d320661b5d814a75ef36e7f088f46bf70179419b19ecb9da171ab1ab4 WatchSource:0}: Error finding container ffc8236d320661b5d814a75ef36e7f088f46bf70179419b19ecb9da171ab1ab4: Status 404 returned error can't find the container with id ffc8236d320661b5d814a75ef36e7f088f46bf70179419b19ecb9da171ab1ab4 Oct 29 22:33:35.056657 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:35.056632 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-vjlkr"] Oct 29 22:33:35.059189 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:33:35.059161 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod652bedff_87f1_4b2f_917f_571a37dd2f3f.slice/crio-3f2aff1dfc2f332ef98875b32540bc94544ff46bc96652dcfd3100ca3434ad2a WatchSource:0}: Error finding container 3f2aff1dfc2f332ef98875b32540bc94544ff46bc96652dcfd3100ca3434ad2a: Status 404 returned error can't find the container with id 3f2aff1dfc2f332ef98875b32540bc94544ff46bc96652dcfd3100ca3434ad2a Oct 29 22:33:35.193788 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:35.193704 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-9xmdb"] Oct 29 22:33:35.196789 ip-10-0-140-74 kubenswrapper[2577]: W1029 22:33:35.196760 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49c67ce3_90fd_478d_9a15_e4267438ff69.slice/crio-e6876edb30d42c86b24074e617f02daea0bf1d967904b4505b4df4ce737d3508 WatchSource:0}: Error finding container e6876edb30d42c86b24074e617f02daea0bf1d967904b4505b4df4ce737d3508: Status 404 returned error can't find the container with id e6876edb30d42c86b24074e617f02daea0bf1d967904b4505b4df4ce737d3508 Oct 29 22:33:35.245713 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:35.245692 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/079ed4e4-bdc4-405c-90a1-a22b84fcf948-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-d2dnn\" (UID: \"079ed4e4-bdc4-405c-90a1-a22b84fcf948\") " pod="openshift-insights/insights-runtime-extractor-d2dnn" Oct 29 22:33:35.245840 ip-10-0-140-74 kubenswrapper[2577]: E1029 22:33:35.245819 2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 29 22:33:35.245900 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:35.245850 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/079ed4e4-bdc4-405c-90a1-a22b84fcf948-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-d2dnn\" (UID: \"079ed4e4-bdc4-405c-90a1-a22b84fcf948\") " pod="openshift-insights/insights-runtime-extractor-d2dnn" Oct 29 22:33:35.245900 ip-10-0-140-74 kubenswrapper[2577]: E1029 22:33:35.245871 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/079ed4e4-bdc4-405c-90a1-a22b84fcf948-kube-rbac-proxy-cm podName:079ed4e4-bdc4-405c-90a1-a22b84fcf948 nodeName:}" failed. No retries permitted until 2026-10-29 22:33:37.245857915 +0000 UTC m=+39.403045575 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/079ed4e4-bdc4-405c-90a1-a22b84fcf948-kube-rbac-proxy-cm") pod "insights-runtime-extractor-d2dnn" (UID: "079ed4e4-bdc4-405c-90a1-a22b84fcf948") : configmap "kube-rbac-proxy" not found Oct 29 22:33:35.246007 ip-10-0-140-74 kubenswrapper[2577]: E1029 22:33:35.245952 2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 29 22:33:35.246007 ip-10-0-140-74 kubenswrapper[2577]: E1029 22:33:35.246004 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/079ed4e4-bdc4-405c-90a1-a22b84fcf948-insights-runtime-extractor-tls podName:079ed4e4-bdc4-405c-90a1-a22b84fcf948 nodeName:}" failed. No retries permitted until 2026-10-29 22:33:37.245989006 +0000 UTC m=+39.403176684 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/079ed4e4-bdc4-405c-90a1-a22b84fcf948-insights-runtime-extractor-tls") pod "insights-runtime-extractor-d2dnn" (UID: "079ed4e4-bdc4-405c-90a1-a22b84fcf948") : secret "insights-runtime-extractor-tls" not found Oct 29 22:33:35.447683 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:35.447650 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2accd775-2173-4bfe-a413-7d243bd0820f-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-6rr4f\" (UID: \"2accd775-2173-4bfe-a413-7d243bd0820f\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-6rr4f" Oct 29 22:33:35.447865 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:35.447711 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71d80eec-b5e7-41d0-b58d-8a8d9a6c769c-metrics-certs\") pod \"router-default-5d8d7cc579-8cwjp\" (UID: \"71d80eec-b5e7-41d0-b58d-8a8d9a6c769c\") " pod="openshift-ingress/router-default-5d8d7cc579-8cwjp" Oct 29 22:33:35.447865 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:35.447740 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/989956e8-1893-4bbe-a4bf-6a2081509080-registry-tls\") pod \"image-registry-84866f86bd-2v6d7\" (UID: \"989956e8-1893-4bbe-a4bf-6a2081509080\") " pod="openshift-image-registry/image-registry-84866f86bd-2v6d7" Oct 29 22:33:35.447865 ip-10-0-140-74 kubenswrapper[2577]: E1029 22:33:35.447804 2577 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 29 22:33:35.447865 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:35.447812 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a53078b9-a6a8-423c-b852-34eececa7ab1-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-p2qv4\" (UID: \"a53078b9-a6a8-423c-b852-34eececa7ab1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-p2qv4" Oct 29 22:33:35.447865 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:35.447857 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5296979a-7927-4ee2-84b9-a93780a2ad22-registry-tls\") pod \"image-registry-78db97464f-6ljzh\" (UID: \"5296979a-7927-4ee2-84b9-a93780a2ad22\") " pod="openshift-image-registry/image-registry-78db97464f-6ljzh" Oct 29 22:33:35.448075 ip-10-0-140-74 kubenswrapper[2577]: E1029 22:33:35.447872 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2accd775-2173-4bfe-a413-7d243bd0820f-networking-console-plugin-cert podName:2accd775-2173-4bfe-a413-7d243bd0820f nodeName:}" failed. No retries permitted until 2026-10-29 22:33:43.447853668 +0000 UTC m=+45.605041329 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/2accd775-2173-4bfe-a413-7d243bd0820f-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-6rr4f" (UID: "2accd775-2173-4bfe-a413-7d243bd0820f") : secret "networking-console-plugin-cert" not found Oct 29 22:33:35.448075 ip-10-0-140-74 kubenswrapper[2577]: E1029 22:33:35.447883 2577 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 29 22:33:35.448075 ip-10-0-140-74 kubenswrapper[2577]: E1029 22:33:35.447942 2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 29 22:33:35.448075 ip-10-0-140-74 kubenswrapper[2577]: E1029 22:33:35.447960 2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-84866f86bd-2v6d7: secret "image-registry-tls" not found Oct 29 22:33:35.448075 ip-10-0-140-74 kubenswrapper[2577]: E1029 22:33:35.447960 2577 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 29 22:33:35.448075 ip-10-0-140-74 kubenswrapper[2577]: E1029 22:33:35.447942 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71d80eec-b5e7-41d0-b58d-8a8d9a6c769c-metrics-certs podName:71d80eec-b5e7-41d0-b58d-8a8d9a6c769c nodeName:}" failed. No retries permitted until 2026-10-29 22:33:43.447930941 +0000 UTC m=+45.605118609 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/71d80eec-b5e7-41d0-b58d-8a8d9a6c769c-metrics-certs") pod "router-default-5d8d7cc579-8cwjp" (UID: "71d80eec-b5e7-41d0-b58d-8a8d9a6c769c") : secret "router-metrics-certs-default" not found Oct 29 22:33:35.448075 ip-10-0-140-74 kubenswrapper[2577]: E1029 22:33:35.448013 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/989956e8-1893-4bbe-a4bf-6a2081509080-registry-tls podName:989956e8-1893-4bbe-a4bf-6a2081509080 nodeName:}" failed. No retries permitted until 2026-10-29 22:33:43.448000032 +0000 UTC m=+45.605187703 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/989956e8-1893-4bbe-a4bf-6a2081509080-registry-tls") pod "image-registry-84866f86bd-2v6d7" (UID: "989956e8-1893-4bbe-a4bf-6a2081509080") : secret "image-registry-tls" not found Oct 29 22:33:35.448075 ip-10-0-140-74 kubenswrapper[2577]: E1029 22:33:35.448030 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a53078b9-a6a8-423c-b852-34eececa7ab1-samples-operator-tls podName:a53078b9-a6a8-423c-b852-34eececa7ab1 nodeName:}" failed. No retries permitted until 2026-10-29 22:33:43.448022073 +0000 UTC m=+45.605209734 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/a53078b9-a6a8-423c-b852-34eececa7ab1-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-p2qv4" (UID: "a53078b9-a6a8-423c-b852-34eececa7ab1") : secret "samples-operator-tls" not found Oct 29 22:33:35.448075 ip-10-0-140-74 kubenswrapper[2577]: E1029 22:33:35.448045 2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 29 22:33:35.448075 ip-10-0-140-74 kubenswrapper[2577]: E1029 22:33:35.448056 2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-78db97464f-6ljzh: secret "image-registry-tls" not found Oct 29 22:33:35.448075 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:35.448074 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/71d80eec-b5e7-41d0-b58d-8a8d9a6c769c-service-ca-bundle\") pod \"router-default-5d8d7cc579-8cwjp\" (UID: \"71d80eec-b5e7-41d0-b58d-8a8d9a6c769c\") " pod="openshift-ingress/router-default-5d8d7cc579-8cwjp" Oct 29 22:33:35.448583 ip-10-0-140-74 kubenswrapper[2577]: E1029 22:33:35.448087 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5296979a-7927-4ee2-84b9-a93780a2ad22-registry-tls podName:5296979a-7927-4ee2-84b9-a93780a2ad22 nodeName:}" failed. No retries permitted until 2026-10-29 22:33:43.44807715 +0000 UTC m=+45.605264818 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5296979a-7927-4ee2-84b9-a93780a2ad22-registry-tls") pod "image-registry-78db97464f-6ljzh" (UID: "5296979a-7927-4ee2-84b9-a93780a2ad22") : secret "image-registry-tls" not found Oct 29 22:33:35.448583 ip-10-0-140-74 kubenswrapper[2577]: E1029 22:33:35.448138 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/71d80eec-b5e7-41d0-b58d-8a8d9a6c769c-service-ca-bundle podName:71d80eec-b5e7-41d0-b58d-8a8d9a6c769c nodeName:}" failed. No retries permitted until 2026-10-29 22:33:43.448129311 +0000 UTC m=+45.605316989 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/71d80eec-b5e7-41d0-b58d-8a8d9a6c769c-service-ca-bundle") pod "router-default-5d8d7cc579-8cwjp" (UID: "71d80eec-b5e7-41d0-b58d-8a8d9a6c769c") : configmap references non-existent config key: service-ca.crt Oct 29 22:33:35.549502 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:35.549472 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/641f9ec1-c8e2-4774-9d4f-6a7a2da97b6f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-8c5hr\" (UID: \"641f9ec1-c8e2-4774-9d4f-6a7a2da97b6f\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-8c5hr" Oct 29 22:33:35.549666 ip-10-0-140-74 kubenswrapper[2577]: E1029 22:33:35.549629 2577 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 29 22:33:35.549723 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:35.549668 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6b34b3b1-84af-45f3-92ee-1ca720e8b263-metrics-tls\") pod \"dns-default-jjz47\" (UID: \"6b34b3b1-84af-45f3-92ee-1ca720e8b263\") " pod="openshift-dns/dns-default-jjz47" Oct 29 22:33:35.549723 ip-10-0-140-74 kubenswrapper[2577]: E1029 22:33:35.549695 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/641f9ec1-c8e2-4774-9d4f-6a7a2da97b6f-cluster-monitoring-operator-tls podName:641f9ec1-c8e2-4774-9d4f-6a7a2da97b6f nodeName:}" failed. No retries permitted until 2026-10-29 22:33:43.5496762 +0000 UTC m=+45.706863887 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/641f9ec1-c8e2-4774-9d4f-6a7a2da97b6f-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-8c5hr" (UID: "641f9ec1-c8e2-4774-9d4f-6a7a2da97b6f") : secret "cluster-monitoring-operator-tls" not found Oct 29 22:33:35.549800 ip-10-0-140-74 kubenswrapper[2577]: E1029 22:33:35.549737 2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 29 22:33:35.549800 ip-10-0-140-74 kubenswrapper[2577]: E1029 22:33:35.549774 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b34b3b1-84af-45f3-92ee-1ca720e8b263-metrics-tls podName:6b34b3b1-84af-45f3-92ee-1ca720e8b263 nodeName:}" failed. No retries permitted until 2026-10-29 22:33:43.54976423 +0000 UTC m=+45.706951892 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6b34b3b1-84af-45f3-92ee-1ca720e8b263-metrics-tls") pod "dns-default-jjz47" (UID: "6b34b3b1-84af-45f3-92ee-1ca720e8b263") : secret "dns-default-metrics-tls" not found Oct 29 22:33:35.567703 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:35.567623 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:33:35.699526 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:35.699445 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-c7kcc" event={"ID":"b5961d4b-f3c0-4f17-b1bc-2781cb3dcd6e","Type":"ContainerStarted","Data":"ffc8236d320661b5d814a75ef36e7f088f46bf70179419b19ecb9da171ab1ab4"} Oct 29 22:33:35.701981 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:35.701911 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-9xmdb" event={"ID":"49c67ce3-90fd-478d-9a15-e4267438ff69","Type":"ContainerStarted","Data":"91489e5868c153563688325eaf3fbabfecb751f432e17d677fa5dd9127ba5d2c"} Oct 29 22:33:35.701981 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:35.701956 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-9xmdb" event={"ID":"49c67ce3-90fd-478d-9a15-e4267438ff69","Type":"ContainerStarted","Data":"e6876edb30d42c86b24074e617f02daea0bf1d967904b4505b4df4ce737d3508"} Oct 29 22:33:35.703618 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:35.703560 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-vjlkr" event={"ID":"652bedff-87f1-4b2f-917f-571a37dd2f3f","Type":"ContainerStarted","Data":"3f2aff1dfc2f332ef98875b32540bc94544ff46bc96652dcfd3100ca3434ad2a"} Oct 29 22:33:35.775097 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:35.775040 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca/service-ca-5fffc97bdf-9xmdb" podStartSLOduration=1.7750244899999998 podStartE2EDuration="1.77502449s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:33:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:33:35.739200892 +0000 UTC m=+37.896388576" watchObservedRunningTime="2026-10-29 22:33:35.77502449 +0000 UTC m=+37.932212178" Oct 29 22:33:35.775914 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:35.775874 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-lnrkb-btcvq" podStartSLOduration=2.775864921 podStartE2EDuration="2.775864921s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:33:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:33:35.774219184 +0000 UTC m=+37.931406887" watchObservedRunningTime="2026-10-29 22:33:35.775864921 +0000 UTC m=+37.933052606" Oct 29 22:33:36.277866 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:36.277799 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:33:36.413348 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:36.412807 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:33:36.534949 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:36.534871 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:33:36.644948 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:36.644911 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:33:36.758235 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:36.758200 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:33:37.266947 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:37.266910 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/079ed4e4-bdc4-405c-90a1-a22b84fcf948-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-d2dnn\" (UID: \"079ed4e4-bdc4-405c-90a1-a22b84fcf948\") " pod="openshift-insights/insights-runtime-extractor-d2dnn" Oct 29 22:33:37.267131 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:37.267004 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/079ed4e4-bdc4-405c-90a1-a22b84fcf948-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-d2dnn\" (UID: \"079ed4e4-bdc4-405c-90a1-a22b84fcf948\") " pod="openshift-insights/insights-runtime-extractor-d2dnn" Oct 29 22:33:37.267131 ip-10-0-140-74 kubenswrapper[2577]: E1029 22:33:37.267039 2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 29 22:33:37.267131 ip-10-0-140-74 kubenswrapper[2577]: E1029 22:33:37.267108 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/079ed4e4-bdc4-405c-90a1-a22b84fcf948-kube-rbac-proxy-cm podName:079ed4e4-bdc4-405c-90a1-a22b84fcf948 nodeName:}" failed. No retries permitted until 2026-10-29 22:33:41.267089092 +0000 UTC m=+43.424276766 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/079ed4e4-bdc4-405c-90a1-a22b84fcf948-kube-rbac-proxy-cm") pod "insights-runtime-extractor-d2dnn" (UID: "079ed4e4-bdc4-405c-90a1-a22b84fcf948") : configmap "kube-rbac-proxy" not found Oct 29 22:33:37.267298 ip-10-0-140-74 kubenswrapper[2577]: E1029 22:33:37.267168 2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 29 22:33:37.267298 ip-10-0-140-74 kubenswrapper[2577]: E1029 22:33:37.267216 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/079ed4e4-bdc4-405c-90a1-a22b84fcf948-insights-runtime-extractor-tls podName:079ed4e4-bdc4-405c-90a1-a22b84fcf948 nodeName:}" failed. No retries permitted until 2026-10-29 22:33:41.267205011 +0000 UTC m=+43.424392685 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/079ed4e4-bdc4-405c-90a1-a22b84fcf948-insights-runtime-extractor-tls") pod "insights-runtime-extractor-d2dnn" (UID: "079ed4e4-bdc4-405c-90a1-a22b84fcf948") : secret "insights-runtime-extractor-tls" not found Oct 29 22:33:37.712601 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:37.712559 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-vjlkr" event={"ID":"652bedff-87f1-4b2f-917f-571a37dd2f3f","Type":"ContainerStarted","Data":"cec166f7380c6da8eda863709280714a7dba2ccb59549b3bc8b10dfbfde938fe"} Oct 29 22:33:37.714234 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:37.714204 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-c7kcc" event={"ID":"b5961d4b-f3c0-4f17-b1bc-2781cb3dcd6e","Type":"ContainerStarted","Data":"769a6b84404da84ab12de6c4a960aab8eebab34ce10a838091c6ce97e1743d5b"} Oct 29 22:33:37.764880 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:37.764823 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-vjlkr" podStartSLOduration=1.9749077750000001 podStartE2EDuration="3.764806523s" totalImagesPullingTime="1.789898748s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:33:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:33:37.762560251 +0000 UTC m=+39.919747962" watchObservedRunningTime="2026-10-29 22:33:37.764806523 +0000 UTC m=+39.921994207" Oct 29 22:33:37.801303 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:37.801250 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-c7kcc" podStartSLOduration=1.967795132 podStartE2EDuration="3.801234371s" totalImagesPullingTime="1.833439239s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:33:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:33:37.801043181 +0000 UTC m=+39.958230864" watchObservedRunningTime="2026-10-29 22:33:37.801234371 +0000 UTC m=+39.958422054" Oct 29 22:33:38.006927 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:38.006830 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:33:38.029410 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:38.029381 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-vj6md" Oct 29 22:33:38.029552 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:38.029447 2577 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-vj6md" Oct 29 22:33:38.029846 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:38.029828 2577 scope.go:122] "RemoveContainer" containerID="e5ab727c17d3a45185a300b3b2a3688775b919a83c6e93f3a510f87ee56a0fb7" Oct 29 22:33:38.030066 ip-10-0-140-74 kubenswrapper[2577]: E1029 22:33:38.030043 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-vj6md_openshift-console-operator(aa6a190e-3283-458a-9dfd-2de7cb193fc5)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-vj6md" podUID="aa6a190e-3283-458a-9dfd-2de7cb193fc5" Oct 29 22:33:38.132317 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:38.132281 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:33:38.238547 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:38.238455 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:33:38.358718 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:38.358639 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:33:38.559072 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:38.559042 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:33:38.702323 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:38.702295 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-dpsd9_0e1c2279-b6d4-44e0-84e6-94eb1206d2a0/dns-node-resolver/0.log" Oct 29 22:33:38.762740 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:38.762712 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:33:39.149266 ip-10-0-140-74 kubenswrapper[2577]: I1029 22:33:39.149171 2577 dynamic_cafile_content.go:175] "Shutting down controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 29 22:33:39.149367 ip-10-0-140-74 systemd[1]: Stopping Kubernetes Kubelet... Oct 29 22:33:39.163708 ip-10-0-140-74 systemd[1]: kubelet.service: Deactivated successfully. Oct 29 22:33:39.163871 ip-10-0-140-74 systemd[1]: Stopped Kubernetes Kubelet. Oct 29 22:33:39.163921 ip-10-0-140-74 systemd[1]: kubelet.service: Consumed 3.146s CPU time. Oct 29 22:33:39.171902 ip-10-0-140-74 systemd[1]: Starting Kubernetes Kubelet... Oct 29 22:33:39.247021 ip-10-0-140-74 kubenswrapper[5057]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 29 22:33:39.247021 ip-10-0-140-74 kubenswrapper[5057]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 29 22:33:39.247021 ip-10-0-140-74 kubenswrapper[5057]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 29 22:33:39.247520 ip-10-0-140-74 kubenswrapper[5057]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 29 22:33:39.249346 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.249248 5057 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 29 22:33:39.249346 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.249345 5057 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 29 22:33:39.249502 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.249351 5057 feature_gate.go:386] unrecognized feature gate: EVPN Oct 29 22:33:39.249502 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.249356 5057 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 29 22:33:39.249502 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.249361 5057 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 29 22:33:39.249502 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.249365 5057 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 29 22:33:39.249502 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.249369 5057 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 29 22:33:39.249502 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.249379 5057 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 29 22:33:39.249502 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.249384 5057 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 29 22:33:39.249502 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.249388 5057 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 29 22:33:39.249502 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.249392 5057 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 29 22:33:39.249502 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.249396 5057 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 29 22:33:39.249502 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.249400 5057 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 29 22:33:39.249502 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.249404 5057 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 29 22:33:39.249502 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.249408 5057 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 29 22:33:39.249502 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.249412 5057 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 29 22:33:39.249502 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.249416 5057 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 29 22:33:39.249502 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.249420 5057 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 29 22:33:39.249502 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.249424 5057 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 29 22:33:39.249502 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.249430 5057 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 29 22:33:39.249502 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.249436 5057 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 29 22:33:39.250408 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.249440 5057 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 29 22:33:39.250408 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.249444 5057 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 29 22:33:39.250408 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.249448 5057 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 29 22:33:39.250408 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.249452 5057 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 29 22:33:39.250408 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.249456 5057 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 29 22:33:39.250408 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.249460 5057 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 29 22:33:39.250408 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.249465 5057 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 29 22:33:39.250408 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.249469 5057 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 29 22:33:39.250408 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.249476 5057 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 29 22:33:39.250408 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.249480 5057 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 29 22:33:39.250408 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.249485 5057 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 29 22:33:39.250408 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.249489 5057 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 29 22:33:39.250408 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.249493 5057 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 29 22:33:39.250408 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.249498 5057 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 29 22:33:39.250408 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.249503 5057 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 29 22:33:39.250408 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.249507 5057 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 29 22:33:39.250408 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.249511 5057 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 29 22:33:39.250408 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.249516 5057 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 29 22:33:39.250408 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.249521 5057 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 29 22:33:39.251291 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.249525 5057 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 29 22:33:39.251291 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.249530 5057 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 29 22:33:39.251291 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.249534 5057 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 29 22:33:39.251291 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.249539 5057 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 29 22:33:39.251291 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.249543 5057 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 29 22:33:39.251291 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.249547 5057 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 29 22:33:39.251291 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.249551 5057 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 29 22:33:39.251291 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.249556 5057 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 29 22:33:39.251291 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.249560 5057 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 29 22:33:39.251291 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.249564 5057 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 29 22:33:39.251291 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.249568 5057 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 29 22:33:39.251291 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.249572 5057 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 29 22:33:39.251291 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.249576 5057 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 29 22:33:39.251291 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.249581 5057 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 29 22:33:39.251291 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.249585 5057 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 29 22:33:39.251291 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.249590 5057 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 29 22:33:39.251291 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.249594 5057 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 29 22:33:39.251291 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.249598 5057 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 29 22:33:39.251291 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.249603 5057 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 29 22:33:39.251291 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.249607 5057 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 29 22:33:39.252193 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.249612 5057 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 29 22:33:39.252193 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.249616 5057 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 29 22:33:39.252193 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.249621 5057 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 29 22:33:39.252193 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.249627 5057 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 29 22:33:39.252193 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.249632 5057 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 29 22:33:39.252193 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.249636 5057 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 29 22:33:39.252193 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.249640 5057 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 29 22:33:39.252193 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.249645 5057 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 29 22:33:39.252193 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.249649 5057 feature_gate.go:386] unrecognized feature gate: Example Oct 29 22:33:39.252193 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.249653 5057 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 29 22:33:39.252193 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.249657 5057 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 29 22:33:39.252193 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.249662 5057 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 29 22:33:39.252193 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.249668 5057 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 29 22:33:39.252193 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.249673 5057 feature_gate.go:386] unrecognized feature gate: Example2 Oct 29 22:33:39.252193 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.249678 5057 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 29 22:33:39.252193 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.249682 5057 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 29 22:33:39.252193 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.249686 5057 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 29 22:33:39.252193 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.249690 5057 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 29 22:33:39.252193 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.249694 5057 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 29 22:33:39.252193 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.249698 5057 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 29 22:33:39.253121 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.249703 5057 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 29 22:33:39.253121 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.249708 5057 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 29 22:33:39.253121 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.249712 5057 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 29 22:33:39.253121 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.249716 5057 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 29 22:33:39.253121 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.249721 5057 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 29 22:33:39.253121 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.249725 5057 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 29 22:33:39.253121 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.249729 5057 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 29 22:33:39.253121 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.249734 5057 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 29 22:33:39.253121 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.249738 5057 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 29 22:33:39.253121 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.249745 5057 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 29 22:33:39.253121 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.249752 5057 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 29 22:33:39.253121 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.249756 5057 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 29 22:33:39.253121 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.249761 5057 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 29 22:33:39.253121 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.249765 5057 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 29 22:33:39.253121 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.249769 5057 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 29 22:33:39.253121 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.249774 5057 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 29 22:33:39.253121 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.249778 5057 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 29 22:33:39.253121 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.249782 5057 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 29 22:33:39.253121 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.249787 5057 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 29 22:33:39.254036 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.249791 5057 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 29 22:33:39.254036 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.249796 5057 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 29 22:33:39.254036 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.249800 5057 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 29 22:33:39.254036 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.249804 5057 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 29 22:33:39.254036 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.249809 5057 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 29 22:33:39.254036 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.249813 5057 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 29 22:33:39.254036 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.249818 5057 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 29 22:33:39.254036 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.251101 5057 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 29 22:33:39.254036 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.251114 5057 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 29 22:33:39.254036 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.251119 5057 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 29 22:33:39.254036 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.251123 5057 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 29 22:33:39.254036 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.251128 5057 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 29 22:33:39.254036 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.251133 5057 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 29 22:33:39.254036 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.251138 5057 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 29 22:33:39.254036 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.251142 5057 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 29 22:33:39.254036 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.251147 5057 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 29 22:33:39.254036 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.251152 5057 feature_gate.go:386] unrecognized feature gate: EVPN Oct 29 22:33:39.254036 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.251156 5057 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 29 22:33:39.254036 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.251160 5057 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 29 22:33:39.254036 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.251164 5057 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 29 22:33:39.254964 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.251169 5057 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 29 22:33:39.254964 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.251173 5057 feature_gate.go:386] unrecognized feature gate: Example2 Oct 29 22:33:39.254964 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.251178 5057 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 29 22:33:39.254964 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.251182 5057 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 29 22:33:39.254964 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.251186 5057 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 29 22:33:39.254964 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.251193 5057 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 29 22:33:39.254964 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.251199 5057 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 29 22:33:39.254964 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.251203 5057 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 29 22:33:39.254964 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.251208 5057 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 29 22:33:39.254964 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.251212 5057 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 29 22:33:39.254964 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.251216 5057 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 29 22:33:39.254964 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.251221 5057 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 29 22:33:39.254964 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.251226 5057 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 29 22:33:39.254964 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.251231 5057 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 29 22:33:39.254964 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.251235 5057 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 29 22:33:39.254964 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.251240 5057 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 29 22:33:39.254964 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.251243 5057 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 29 22:33:39.254964 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.251247 5057 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 29 22:33:39.254964 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.251252 5057 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 29 22:33:39.254964 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.251257 5057 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 29 22:33:39.255845 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.251261 5057 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 29 22:33:39.255845 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.251265 5057 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 29 22:33:39.255845 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.251270 5057 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 29 22:33:39.255845 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.251274 5057 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 29 22:33:39.255845 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.251279 5057 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 29 22:33:39.255845 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.251283 5057 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 29 22:33:39.255845 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.251287 5057 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 29 22:33:39.255845 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.251292 5057 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 29 22:33:39.255845 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.251296 5057 feature_gate.go:386] unrecognized feature gate: Example Oct 29 22:33:39.255845 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.251300 5057 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 29 22:33:39.255845 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.251304 5057 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 29 22:33:39.255845 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.251313 5057 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 29 22:33:39.255845 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.251318 5057 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 29 22:33:39.255845 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.251322 5057 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 29 22:33:39.255845 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.251340 5057 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 29 22:33:39.255845 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.251345 5057 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 29 22:33:39.255845 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.251349 5057 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 29 22:33:39.255845 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.251354 5057 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 29 22:33:39.255845 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.251358 5057 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 29 22:33:39.255845 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.251363 5057 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 29 22:33:39.256370 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.251367 5057 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 29 22:33:39.256370 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.251371 5057 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 29 22:33:39.256370 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.251376 5057 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 29 22:33:39.256370 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.251380 5057 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 29 22:33:39.256370 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.251385 5057 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 29 22:33:39.256370 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.251391 5057 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 29 22:33:39.256370 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.251395 5057 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 29 22:33:39.256370 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.251400 5057 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 29 22:33:39.256370 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.251404 5057 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 29 22:33:39.256370 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.251409 5057 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 29 22:33:39.256370 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.251413 5057 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 29 22:33:39.256370 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.251417 5057 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 29 22:33:39.256370 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.251427 5057 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 29 22:33:39.256370 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.251432 5057 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 29 22:33:39.256370 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.251436 5057 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 29 22:33:39.256370 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.251441 5057 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 29 22:33:39.256370 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.251445 5057 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 29 22:33:39.256370 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.251449 5057 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 29 22:33:39.256370 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.251454 5057 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 29 22:33:39.256861 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.251458 5057 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 29 22:33:39.256861 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.251462 5057 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 29 22:33:39.256861 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.251466 5057 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 29 22:33:39.256861 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.251471 5057 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 29 22:33:39.256861 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.251477 5057 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 29 22:33:39.256861 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.251481 5057 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 29 22:33:39.256861 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.251486 5057 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 29 22:33:39.256861 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.251490 5057 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 29 22:33:39.256861 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.251494 5057 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 29 22:33:39.256861 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.251499 5057 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 29 22:33:39.256861 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.251504 5057 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 29 22:33:39.256861 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.251508 5057 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 29 22:33:39.256861 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.251512 5057 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 29 22:33:39.256861 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.251516 5057 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 29 22:33:39.256861 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.251522 5057 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 29 22:33:39.256861 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.251528 5057 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 29 22:33:39.256861 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.251534 5057 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 29 22:33:39.256861 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.251539 5057 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 29 22:33:39.256861 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.251544 5057 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 29 22:33:39.257473 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.251548 5057 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 29 22:33:39.257473 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.251552 5057 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 29 22:33:39.257473 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.251558 5057 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 29 22:33:39.257473 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.251563 5057 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 29 22:33:39.257473 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.251567 5057 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 29 22:33:39.257473 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.251571 5057 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 29 22:33:39.257473 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.251576 5057 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 29 22:33:39.257473 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.251594 5057 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 29 22:33:39.257473 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.251599 5057 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 29 22:33:39.257473 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.251604 5057 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 29 22:33:39.257473 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.251608 5057 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 29 22:33:39.257473 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.251612 5057 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 29 22:33:39.257473 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.251617 5057 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 29 22:33:39.257473 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.251621 5057 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 29 22:33:39.257473 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.251626 5057 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 29 22:33:39.257473 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.252508 5057 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 29 22:33:39.257473 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.252520 5057 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 29 22:33:39.257473 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.252528 5057 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 29 22:33:39.257473 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.252533 5057 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 29 22:33:39.257943 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.252537 5057 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 29 22:33:39.257943 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.252543 5057 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 29 22:33:39.257943 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.252548 5057 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 29 22:33:39.257943 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.252552 5057 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 29 22:33:39.257943 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.252557 5057 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 29 22:33:39.257943 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.252561 5057 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 29 22:33:39.257943 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.252566 5057 feature_gate.go:386] unrecognized feature gate: Example2 Oct 29 22:33:39.257943 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.252570 5057 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 29 22:33:39.257943 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.252574 5057 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 29 22:33:39.257943 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.252579 5057 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 29 22:33:39.257943 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.252584 5057 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 29 22:33:39.257943 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.252588 5057 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 29 22:33:39.257943 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.252593 5057 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 29 22:33:39.257943 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.252597 5057 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 29 22:33:39.257943 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.252602 5057 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 29 22:33:39.257943 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.252606 5057 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 29 22:33:39.257943 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.252610 5057 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 29 22:33:39.257943 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.252614 5057 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 29 22:33:39.257943 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.252618 5057 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 29 22:33:39.257943 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.252623 5057 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 29 22:33:39.258447 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.252633 5057 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 29 22:33:39.258447 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.252638 5057 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 29 22:33:39.258447 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.252642 5057 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 29 22:33:39.258447 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.252647 5057 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 29 22:33:39.258447 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.252654 5057 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 29 22:33:39.258447 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.252660 5057 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 29 22:33:39.258447 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.252665 5057 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 29 22:33:39.258447 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.252670 5057 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 29 22:33:39.258447 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.252675 5057 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 29 22:33:39.258447 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.252679 5057 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 29 22:33:39.258447 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.252684 5057 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 29 22:33:39.258447 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.252689 5057 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 29 22:33:39.258447 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.252693 5057 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 29 22:33:39.258447 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.252698 5057 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 29 22:33:39.258447 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.252702 5057 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 29 22:33:39.258447 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.252706 5057 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 29 22:33:39.258447 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.252711 5057 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 29 22:33:39.258447 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.252715 5057 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 29 22:33:39.258447 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.252719 5057 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 29 22:33:39.259070 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.252724 5057 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 29 22:33:39.259070 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.252728 5057 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 29 22:33:39.259070 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.252732 5057 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 29 22:33:39.259070 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.252739 5057 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 29 22:33:39.259070 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.252743 5057 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 29 22:33:39.259070 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.252748 5057 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 29 22:33:39.259070 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.252752 5057 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 29 22:33:39.259070 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.252756 5057 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 29 22:33:39.259070 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.252761 5057 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 29 22:33:39.259070 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.252765 5057 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 29 22:33:39.259070 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.252769 5057 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 29 22:33:39.259070 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.252773 5057 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 29 22:33:39.259070 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.252780 5057 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 29 22:33:39.259070 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.252784 5057 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 29 22:33:39.259070 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.252789 5057 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 29 22:33:39.259070 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.252794 5057 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 29 22:33:39.259070 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.252798 5057 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 29 22:33:39.259070 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.252802 5057 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 29 22:33:39.259070 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.252807 5057 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 29 22:33:39.259865 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.252811 5057 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 29 22:33:39.259865 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.252815 5057 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 29 22:33:39.259865 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.252819 5057 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 29 22:33:39.259865 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.252824 5057 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 29 22:33:39.259865 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.252829 5057 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 29 22:33:39.259865 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.252833 5057 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 29 22:33:39.259865 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.252838 5057 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 29 22:33:39.259865 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.252842 5057 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 29 22:33:39.259865 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.252846 5057 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 29 22:33:39.259865 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.252851 5057 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 29 22:33:39.259865 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.252855 5057 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 29 22:33:39.259865 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.252859 5057 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 29 22:33:39.259865 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.252863 5057 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 29 22:33:39.259865 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.252868 5057 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 29 22:33:39.259865 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.252872 5057 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 29 22:33:39.259865 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.252876 5057 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 29 22:33:39.259865 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.252882 5057 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 29 22:33:39.259865 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.252886 5057 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 29 22:33:39.259865 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.252890 5057 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 29 22:33:39.259865 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.252894 5057 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 29 22:33:39.260394 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.252898 5057 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 29 22:33:39.260394 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.252903 5057 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 29 22:33:39.260394 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.252908 5057 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 29 22:33:39.260394 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.252912 5057 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 29 22:33:39.260394 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.252916 5057 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 29 22:33:39.260394 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.252920 5057 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 29 22:33:39.260394 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.252925 5057 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 29 22:33:39.260394 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.252929 5057 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 29 22:33:39.260394 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.252934 5057 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 29 22:33:39.260394 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.252939 5057 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 29 22:33:39.260394 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.252943 5057 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 29 22:33:39.260394 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.252947 5057 feature_gate.go:386] unrecognized feature gate: Example Oct 29 22:33:39.260394 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.252952 5057 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 29 22:33:39.260394 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.252956 5057 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 29 22:33:39.260394 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.252961 5057 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 29 22:33:39.260394 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.252965 5057 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 29 22:33:39.260394 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.252971 5057 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 29 22:33:39.260394 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.252978 5057 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 29 22:33:39.260394 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.252983 5057 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 29 22:33:39.260881 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.252988 5057 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 29 22:33:39.260881 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.252994 5057 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 29 22:33:39.260881 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.253000 5057 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 29 22:33:39.260881 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.253004 5057 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 29 22:33:39.260881 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.253008 5057 feature_gate.go:386] unrecognized feature gate: EVPN Oct 29 22:33:39.260881 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.253144 5057 flags.go:64] FLAG: --address="0.0.0.0" Oct 29 22:33:39.260881 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.253156 5057 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 29 22:33:39.260881 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.253166 5057 flags.go:64] FLAG: --anonymous-auth="true" Oct 29 22:33:39.260881 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.253174 5057 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 29 22:33:39.260881 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.253183 5057 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 29 22:33:39.260881 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.253188 5057 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 29 22:33:39.260881 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.253194 5057 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 29 22:33:39.260881 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.253201 5057 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 29 22:33:39.260881 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.253206 5057 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 29 22:33:39.260881 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.253212 5057 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 29 22:33:39.260881 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.253217 5057 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 29 22:33:39.260881 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.253223 5057 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 29 22:33:39.260881 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.253228 5057 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 29 22:33:39.260881 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.253233 5057 flags.go:64] FLAG: --cgroup-root="" Oct 29 22:33:39.260881 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.253237 5057 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 29 22:33:39.260881 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.253242 5057 flags.go:64] FLAG: --client-ca-file="" Oct 29 22:33:39.260881 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.253247 5057 flags.go:64] FLAG: --cloud-provider="external" Oct 29 22:33:39.260881 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.253253 5057 flags.go:64] FLAG: --cluster-dns="[]" Oct 29 22:33:39.260881 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.253259 5057 flags.go:64] FLAG: --cluster-domain="" Oct 29 22:33:39.261471 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.253264 5057 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 29 22:33:39.261471 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.253269 5057 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 29 22:33:39.261471 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.253274 5057 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 29 22:33:39.261471 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.253279 5057 flags.go:64] FLAG: --container-log-max-files="5" Oct 29 22:33:39.261471 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.253285 5057 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 29 22:33:39.261471 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.253292 5057 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 29 22:33:39.261471 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.253297 5057 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 29 22:33:39.261471 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.253302 5057 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 29 22:33:39.261471 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.253307 5057 flags.go:64] FLAG: --contention-profiling="false" Oct 29 22:33:39.261471 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.253312 5057 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 29 22:33:39.261471 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.253317 5057 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 29 22:33:39.261471 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.253322 5057 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 29 22:33:39.261471 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.253342 5057 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 29 22:33:39.261471 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.253350 5057 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 29 22:33:39.261471 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.253355 5057 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 29 22:33:39.261471 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.253360 5057 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 29 22:33:39.261471 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.253364 5057 flags.go:64] FLAG: --enable-load-reader="false" Oct 29 22:33:39.261471 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.253372 5057 flags.go:64] FLAG: --enable-server="true" Oct 29 22:33:39.261471 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.253377 5057 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 29 22:33:39.261471 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.253384 5057 flags.go:64] FLAG: --event-burst="100" Oct 29 22:33:39.261471 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.253389 5057 flags.go:64] FLAG: --event-qps="50" Oct 29 22:33:39.261471 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.253394 5057 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 29 22:33:39.261471 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.253399 5057 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 29 22:33:39.261471 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.253404 5057 flags.go:64] FLAG: --eviction-hard="" Oct 29 22:33:39.262041 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.253410 5057 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 29 22:33:39.262041 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.253415 5057 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 29 22:33:39.262041 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.253420 5057 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 29 22:33:39.262041 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.253425 5057 flags.go:64] FLAG: --eviction-soft="" Oct 29 22:33:39.262041 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.253431 5057 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 29 22:33:39.262041 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.253436 5057 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 29 22:33:39.262041 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.253441 5057 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 29 22:33:39.262041 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.253446 5057 flags.go:64] FLAG: --experimental-mounter-path="" Oct 29 22:33:39.262041 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.253451 5057 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 29 22:33:39.262041 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.253455 5057 flags.go:64] FLAG: --fail-swap-on="true" Oct 29 22:33:39.262041 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.253461 5057 flags.go:64] FLAG: --feature-gates="" Oct 29 22:33:39.262041 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.253467 5057 flags.go:64] FLAG: --file-check-frequency="20s" Oct 29 22:33:39.262041 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.253472 5057 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 29 22:33:39.262041 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.253479 5057 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 29 22:33:39.262041 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.253484 5057 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 29 22:33:39.262041 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.253489 5057 flags.go:64] FLAG: --healthz-port="10248" Oct 29 22:33:39.262041 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.253494 5057 flags.go:64] FLAG: --help="false" Oct 29 22:33:39.262041 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.253499 5057 flags.go:64] FLAG: --hostname-override="ip-10-0-140-74.ec2.internal" Oct 29 22:33:39.262041 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.253504 5057 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 29 22:33:39.262041 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.253509 5057 flags.go:64] FLAG: --http-check-frequency="20s" Oct 29 22:33:39.262041 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.253514 5057 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 29 22:33:39.262041 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.253519 5057 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 29 22:33:39.262041 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.253525 5057 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 29 22:33:39.262709 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.253529 5057 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 29 22:33:39.262709 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.253534 5057 flags.go:64] FLAG: --image-service-endpoint="" Oct 29 22:33:39.262709 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.253543 5057 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 29 22:33:39.262709 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.253548 5057 flags.go:64] FLAG: --kube-api-burst="100" Oct 29 22:33:39.262709 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.253553 5057 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 29 22:33:39.262709 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.253558 5057 flags.go:64] FLAG: --kube-api-qps="50" Oct 29 22:33:39.262709 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.253563 5057 flags.go:64] FLAG: --kube-reserved="" Oct 29 22:33:39.262709 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.253568 5057 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 29 22:33:39.262709 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.253572 5057 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 29 22:33:39.262709 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.253577 5057 flags.go:64] FLAG: --kubelet-cgroups="" Oct 29 22:33:39.262709 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.253582 5057 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 29 22:33:39.262709 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.253587 5057 flags.go:64] FLAG: --lock-file="" Oct 29 22:33:39.262709 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.253592 5057 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 29 22:33:39.262709 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.253596 5057 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 29 22:33:39.262709 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.253601 5057 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 29 22:33:39.262709 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.253610 5057 flags.go:64] FLAG: --log-json-split-stream="false" Oct 29 22:33:39.262709 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.253615 5057 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 29 22:33:39.262709 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.253625 5057 flags.go:64] FLAG: --log-text-split-stream="false" Oct 29 22:33:39.262709 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.253630 5057 flags.go:64] FLAG: --logging-format="text" Oct 29 22:33:39.262709 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.253635 5057 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 29 22:33:39.262709 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.253640 5057 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 29 22:33:39.262709 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.253645 5057 flags.go:64] FLAG: --manifest-url="" Oct 29 22:33:39.262709 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.253651 5057 flags.go:64] FLAG: --manifest-url-header="" Oct 29 22:33:39.262709 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.253658 5057 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 29 22:33:39.262709 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.253662 5057 flags.go:64] FLAG: --max-open-files="1000000" Oct 29 22:33:39.263370 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.253669 5057 flags.go:64] FLAG: --max-pods="110" Oct 29 22:33:39.263370 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.253674 5057 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 29 22:33:39.263370 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.253679 5057 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 29 22:33:39.263370 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.253683 5057 flags.go:64] FLAG: --memory-manager-policy="None" Oct 29 22:33:39.263370 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.253688 5057 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 29 22:33:39.263370 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.253693 5057 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 29 22:33:39.263370 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.253698 5057 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 29 22:33:39.263370 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.253703 5057 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 29 22:33:39.263370 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.253714 5057 flags.go:64] FLAG: --node-status-max-images="50" Oct 29 22:33:39.263370 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.253721 5057 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 29 22:33:39.263370 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.253726 5057 flags.go:64] FLAG: --oom-score-adj="-999" Oct 29 22:33:39.263370 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.253731 5057 flags.go:64] FLAG: --pod-cidr="" Oct 29 22:33:39.263370 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.253736 5057 flags.go:64] FLAG: --pod-manifest-path="" Oct 29 22:33:39.263370 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.253740 5057 flags.go:64] FLAG: --pod-max-pids="-1" Oct 29 22:33:39.263370 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.253745 5057 flags.go:64] FLAG: --pods-per-core="0" Oct 29 22:33:39.263370 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.253751 5057 flags.go:64] FLAG: --port="10250" Oct 29 22:33:39.263370 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.253755 5057 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 29 22:33:39.263370 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.253760 5057 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0beae7170e413ea33" Oct 29 22:33:39.263370 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.253765 5057 flags.go:64] FLAG: --qos-reserved="" Oct 29 22:33:39.263370 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.253770 5057 flags.go:64] FLAG: --read-only-port="10255" Oct 29 22:33:39.263370 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.253775 5057 flags.go:64] FLAG: --register-node="true" Oct 29 22:33:39.263370 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.253780 5057 flags.go:64] FLAG: --register-with-taints="" Oct 29 22:33:39.263370 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.253787 5057 flags.go:64] FLAG: --registry-burst="10" Oct 29 22:33:39.263370 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.253792 5057 flags.go:64] FLAG: --registry-qps="5" Oct 29 22:33:39.263370 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.253797 5057 flags.go:64] FLAG: --reserved-cpus="" Oct 29 22:33:39.263370 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.253802 5057 flags.go:64] FLAG: --reserved-memory="" Oct 29 22:33:39.264160 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.253809 5057 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 29 22:33:39.264160 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.253814 5057 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 29 22:33:39.264160 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.253818 5057 flags.go:64] FLAG: --rotate-certificates="false" Oct 29 22:33:39.264160 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.253825 5057 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 29 22:33:39.264160 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.253830 5057 flags.go:64] FLAG: --runonce="false" Oct 29 22:33:39.264160 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.253835 5057 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 29 22:33:39.264160 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.253840 5057 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 29 22:33:39.264160 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.253845 5057 flags.go:64] FLAG: --seccomp-default="false" Oct 29 22:33:39.264160 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.253850 5057 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 29 22:33:39.264160 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.253855 5057 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 29 22:33:39.264160 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.253860 5057 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 29 22:33:39.264160 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.253865 5057 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 29 22:33:39.264160 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.253869 5057 flags.go:64] FLAG: --storage-driver-password="root" Oct 29 22:33:39.264160 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.253874 5057 flags.go:64] FLAG: --storage-driver-secure="false" Oct 29 22:33:39.264160 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.253879 5057 flags.go:64] FLAG: --storage-driver-table="stats" Oct 29 22:33:39.264160 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.253885 5057 flags.go:64] FLAG: --storage-driver-user="root" Oct 29 22:33:39.264160 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.253890 5057 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 29 22:33:39.264160 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.253895 5057 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 29 22:33:39.264160 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.253900 5057 flags.go:64] FLAG: --system-cgroups="" Oct 29 22:33:39.264160 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.253905 5057 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 29 22:33:39.264160 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.253914 5057 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 29 22:33:39.264160 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.253918 5057 flags.go:64] FLAG: --tls-cert-file="" Oct 29 22:33:39.264160 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.253923 5057 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 29 22:33:39.264160 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.253929 5057 flags.go:64] FLAG: --tls-min-version="" Oct 29 22:33:39.264160 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.253934 5057 flags.go:64] FLAG: --tls-private-key-file="" Oct 29 22:33:39.264893 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.253938 5057 flags.go:64] FLAG: --topology-manager-policy="none" Oct 29 22:33:39.264893 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.253943 5057 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 29 22:33:39.264893 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.253948 5057 flags.go:64] FLAG: --topology-manager-scope="container" Oct 29 22:33:39.264893 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.253953 5057 flags.go:64] FLAG: --v="2" Oct 29 22:33:39.264893 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.253959 5057 flags.go:64] FLAG: --version="false" Oct 29 22:33:39.264893 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.253966 5057 flags.go:64] FLAG: --vmodule="" Oct 29 22:33:39.264893 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.253972 5057 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 29 22:33:39.264893 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.253979 5057 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 29 22:33:39.264893 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.254209 5057 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 29 22:33:39.264893 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.254216 5057 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 29 22:33:39.264893 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.254222 5057 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 29 22:33:39.264893 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.254226 5057 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 29 22:33:39.264893 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.254231 5057 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 29 22:33:39.264893 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.254236 5057 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 29 22:33:39.264893 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.254240 5057 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 29 22:33:39.264893 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.254245 5057 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 29 22:33:39.264893 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.254250 5057 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 29 22:33:39.264893 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.254254 5057 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 29 22:33:39.264893 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.254259 5057 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 29 22:33:39.264893 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.254263 5057 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 29 22:33:39.264893 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.254267 5057 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 29 22:33:39.264893 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.254272 5057 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 29 22:33:39.265640 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.254279 5057 feature_gate.go:386] unrecognized feature gate: EVPN Oct 29 22:33:39.265640 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.254283 5057 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 29 22:33:39.265640 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.254288 5057 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 29 22:33:39.265640 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.254292 5057 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 29 22:33:39.265640 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.254296 5057 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 29 22:33:39.265640 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.254301 5057 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 29 22:33:39.265640 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.254305 5057 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 29 22:33:39.265640 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.254309 5057 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 29 22:33:39.265640 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.254313 5057 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 29 22:33:39.265640 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.254317 5057 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 29 22:33:39.265640 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.254322 5057 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 29 22:33:39.265640 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.254343 5057 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 29 22:33:39.265640 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.254349 5057 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 29 22:33:39.265640 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.254354 5057 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 29 22:33:39.265640 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.254358 5057 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 29 22:33:39.265640 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.254362 5057 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 29 22:33:39.265640 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.254366 5057 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 29 22:33:39.265640 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.254370 5057 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 29 22:33:39.265640 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.254375 5057 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 29 22:33:39.265640 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.254380 5057 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 29 22:33:39.266180 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.254384 5057 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 29 22:33:39.266180 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.254388 5057 feature_gate.go:386] unrecognized feature gate: Example Oct 29 22:33:39.266180 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.254392 5057 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 29 22:33:39.266180 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.254397 5057 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 29 22:33:39.266180 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.254401 5057 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 29 22:33:39.266180 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.254406 5057 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 29 22:33:39.266180 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.254410 5057 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 29 22:33:39.266180 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.254414 5057 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 29 22:33:39.266180 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.254418 5057 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 29 22:33:39.266180 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.254422 5057 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 29 22:33:39.266180 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.254427 5057 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 29 22:33:39.266180 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.254431 5057 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 29 22:33:39.266180 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.254437 5057 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 29 22:33:39.266180 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.254442 5057 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 29 22:33:39.266180 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.254446 5057 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 29 22:33:39.266180 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.254450 5057 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 29 22:33:39.266180 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.254457 5057 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 29 22:33:39.266180 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.254463 5057 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 29 22:33:39.266180 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.254468 5057 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 29 22:33:39.266180 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.254473 5057 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 29 22:33:39.266690 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.254478 5057 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 29 22:33:39.266690 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.254483 5057 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 29 22:33:39.266690 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.254490 5057 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 29 22:33:39.266690 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.254495 5057 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 29 22:33:39.266690 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.254500 5057 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 29 22:33:39.266690 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.254504 5057 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 29 22:33:39.266690 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.254509 5057 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 29 22:33:39.266690 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.254513 5057 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 29 22:33:39.266690 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.254517 5057 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 29 22:33:39.266690 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.254521 5057 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 29 22:33:39.266690 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.254525 5057 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 29 22:33:39.266690 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.254530 5057 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 29 22:33:39.266690 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.254535 5057 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 29 22:33:39.266690 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.254540 5057 feature_gate.go:386] unrecognized feature gate: Example2 Oct 29 22:33:39.266690 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.254544 5057 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 29 22:33:39.266690 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.254548 5057 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 29 22:33:39.266690 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.254552 5057 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 29 22:33:39.266690 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.254557 5057 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 29 22:33:39.266690 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.254561 5057 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 29 22:33:39.267289 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.254565 5057 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 29 22:33:39.267289 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.254569 5057 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 29 22:33:39.267289 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.254573 5057 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 29 22:33:39.267289 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.254578 5057 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 29 22:33:39.267289 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.254582 5057 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 29 22:33:39.267289 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.254589 5057 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 29 22:33:39.267289 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.254593 5057 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 29 22:33:39.267289 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.254597 5057 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 29 22:33:39.267289 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.254602 5057 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 29 22:33:39.267289 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.254606 5057 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 29 22:33:39.267289 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.254610 5057 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 29 22:33:39.267289 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.254614 5057 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 29 22:33:39.267289 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.254618 5057 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 29 22:33:39.267289 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.254623 5057 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 29 22:33:39.267289 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.254627 5057 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 29 22:33:39.267289 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.254632 5057 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 29 22:33:39.267289 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.254637 5057 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 29 22:33:39.267289 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.254641 5057 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 29 22:33:39.267289 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.254646 5057 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 29 22:33:39.268039 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.254650 5057 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 29 22:33:39.268039 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.254654 5057 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 29 22:33:39.268039 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.254658 5057 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 29 22:33:39.268039 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.254662 5057 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 29 22:33:39.268039 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.254667 5057 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 29 22:33:39.268039 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.254672 5057 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 29 22:33:39.268039 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.254677 5057 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 29 22:33:39.268039 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.254681 5057 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 29 22:33:39.268039 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.254685 5057 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 29 22:33:39.268039 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.254689 5057 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 29 22:33:39.268039 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.254694 5057 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 29 22:33:39.268039 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.254698 5057 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 29 22:33:39.268039 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.254702 5057 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 29 22:33:39.268039 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.254706 5057 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 29 22:33:39.268039 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.254747 5057 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 29 22:33:39.268550 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.254803 5057 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 29 22:33:39.268550 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.254810 5057 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 29 22:33:39.268550 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.254816 5057 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 29 22:33:39.268550 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.254823 5057 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 29 22:33:39.268550 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.254828 5057 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 29 22:33:39.268550 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.254834 5057 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 29 22:33:39.268550 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.254838 5057 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 29 22:33:39.268550 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.254843 5057 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 29 22:33:39.268550 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.254849 5057 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 29 22:33:39.268550 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.254853 5057 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 29 22:33:39.268550 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.254857 5057 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 29 22:33:39.268550 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.254862 5057 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 29 22:33:39.268550 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.254866 5057 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 29 22:33:39.268550 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.254870 5057 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 29 22:33:39.268550 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.254875 5057 feature_gate.go:386] unrecognized feature gate: EVPN Oct 29 22:33:39.268550 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.254879 5057 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 29 22:33:39.268550 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.254884 5057 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 29 22:33:39.268550 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.254888 5057 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 29 22:33:39.268550 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.254892 5057 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 29 22:33:39.268550 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.254897 5057 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 29 22:33:39.269081 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.254901 5057 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 29 22:33:39.269081 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.254906 5057 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 29 22:33:39.269081 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.254910 5057 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 29 22:33:39.269081 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.254914 5057 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 29 22:33:39.269081 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.254919 5057 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 29 22:33:39.269081 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.254924 5057 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 29 22:33:39.269081 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.254928 5057 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 29 22:33:39.269081 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.254933 5057 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 29 22:33:39.269081 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.254937 5057 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 29 22:33:39.269081 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.254941 5057 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 29 22:33:39.269081 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.254945 5057 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 29 22:33:39.269081 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.254950 5057 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 29 22:33:39.269081 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.254954 5057 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 29 22:33:39.269081 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.254958 5057 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 29 22:33:39.269081 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.254962 5057 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 29 22:33:39.269081 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.254968 5057 feature_gate.go:386] unrecognized feature gate: Example Oct 29 22:33:39.269081 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.254972 5057 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 29 22:33:39.269081 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.254979 5057 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 29 22:33:39.269081 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.254983 5057 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 29 22:33:39.269081 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.254989 5057 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 29 22:33:39.269619 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.254994 5057 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 29 22:33:39.269619 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.254999 5057 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 29 22:33:39.269619 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.255003 5057 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 29 22:33:39.269619 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.255008 5057 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 29 22:33:39.269619 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.255012 5057 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 29 22:33:39.269619 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.255017 5057 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 29 22:33:39.269619 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.255021 5057 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 29 22:33:39.269619 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.255026 5057 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 29 22:33:39.269619 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.255030 5057 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 29 22:33:39.269619 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.255034 5057 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 29 22:33:39.269619 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.255041 5057 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 29 22:33:39.269619 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.255047 5057 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 29 22:33:39.269619 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.255052 5057 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 29 22:33:39.269619 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.255057 5057 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 29 22:33:39.269619 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.255061 5057 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 29 22:33:39.269619 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.255065 5057 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 29 22:33:39.269619 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.255071 5057 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 29 22:33:39.269619 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.255079 5057 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 29 22:33:39.269619 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.255084 5057 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 29 22:33:39.270074 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.255088 5057 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 29 22:33:39.270074 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.255093 5057 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 29 22:33:39.270074 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.255097 5057 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 29 22:33:39.270074 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.255101 5057 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 29 22:33:39.270074 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.255106 5057 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 29 22:33:39.270074 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.255110 5057 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 29 22:33:39.270074 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.255115 5057 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 29 22:33:39.270074 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.255119 5057 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 29 22:33:39.270074 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.255124 5057 feature_gate.go:386] unrecognized feature gate: Example2 Oct 29 22:33:39.270074 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.255128 5057 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 29 22:33:39.270074 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.255134 5057 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 29 22:33:39.270074 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.255139 5057 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 29 22:33:39.270074 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.255143 5057 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 29 22:33:39.270074 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.255147 5057 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 29 22:33:39.270074 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.255152 5057 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 29 22:33:39.270074 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.255156 5057 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 29 22:33:39.270074 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.255160 5057 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 29 22:33:39.270074 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.255165 5057 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 29 22:33:39.270074 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.255169 5057 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 29 22:33:39.270074 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.255173 5057 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 29 22:33:39.270624 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.255178 5057 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 29 22:33:39.270624 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.255182 5057 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 29 22:33:39.270624 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.255186 5057 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 29 22:33:39.270624 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.255191 5057 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 29 22:33:39.270624 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.255195 5057 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 29 22:33:39.270624 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.255199 5057 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 29 22:33:39.270624 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.255204 5057 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 29 22:33:39.270624 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.255208 5057 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 29 22:33:39.270624 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.255212 5057 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 29 22:33:39.270624 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.255217 5057 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 29 22:33:39.270624 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.255221 5057 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 29 22:33:39.270624 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.255226 5057 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 29 22:33:39.270624 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.255231 5057 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 29 22:33:39.270624 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.255236 5057 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 29 22:33:39.270624 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.255240 5057 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 29 22:33:39.270624 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.255244 5057 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 29 22:33:39.270624 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.255248 5057 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 29 22:33:39.270624 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.255253 5057 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 29 22:33:39.270624 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.255258 5057 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 29 22:33:39.271220 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.255262 5057 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 29 22:33:39.271220 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.255267 5057 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 29 22:33:39.271220 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.255271 5057 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 29 22:33:39.271220 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.255277 5057 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 29 22:33:39.271220 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.255282 5057 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 29 22:33:39.271220 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.255286 5057 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 29 22:33:39.271220 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.255290 5057 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 29 22:33:39.271220 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.255295 5057 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 29 22:33:39.271220 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.259838 5057 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 29 22:33:39.271220 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.259856 5057 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 29 22:33:39.271220 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.259874 5057 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 29 22:33:39.271220 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.259880 5057 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 29 22:33:39.271220 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.259938 5057 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 29 22:33:39.271220 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.259944 5057 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 29 22:33:39.271220 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.259947 5057 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 29 22:33:39.271220 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.259951 5057 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 29 22:33:39.271220 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.259954 5057 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 29 22:33:39.271220 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.259957 5057 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 29 22:33:39.271220 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.259960 5057 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 29 22:33:39.271947 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.259963 5057 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 29 22:33:39.271947 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.259965 5057 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 29 22:33:39.271947 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.259968 5057 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 29 22:33:39.271947 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.259971 5057 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 29 22:33:39.271947 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.259974 5057 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 29 22:33:39.271947 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.259976 5057 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 29 22:33:39.271947 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.259979 5057 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 29 22:33:39.271947 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.259982 5057 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 29 22:33:39.271947 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.259985 5057 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 29 22:33:39.271947 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.259988 5057 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 29 22:33:39.271947 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.259991 5057 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 29 22:33:39.271947 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.259993 5057 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 29 22:33:39.271947 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.259996 5057 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 29 22:33:39.271947 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.259999 5057 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 29 22:33:39.271947 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260001 5057 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 29 22:33:39.271947 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260004 5057 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 29 22:33:39.271947 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260007 5057 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 29 22:33:39.271947 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260010 5057 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 29 22:33:39.271947 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260012 5057 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 29 22:33:39.271947 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260015 5057 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 29 22:33:39.272594 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260018 5057 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 29 22:33:39.272594 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260021 5057 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 29 22:33:39.272594 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260024 5057 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 29 22:33:39.272594 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260026 5057 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 29 22:33:39.272594 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260029 5057 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 29 22:33:39.272594 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260032 5057 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 29 22:33:39.272594 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260035 5057 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 29 22:33:39.272594 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260038 5057 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 29 22:33:39.272594 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260041 5057 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 29 22:33:39.272594 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260043 5057 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 29 22:33:39.272594 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260046 5057 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 29 22:33:39.272594 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260048 5057 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 29 22:33:39.272594 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260051 5057 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 29 22:33:39.272594 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260054 5057 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 29 22:33:39.272594 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260056 5057 feature_gate.go:386] unrecognized feature gate: EVPN Oct 29 22:33:39.272594 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260059 5057 feature_gate.go:386] unrecognized feature gate: Example2 Oct 29 22:33:39.272594 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260061 5057 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 29 22:33:39.272594 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260064 5057 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 29 22:33:39.272594 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260067 5057 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 29 22:33:39.272594 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260070 5057 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 29 22:33:39.273146 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260073 5057 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 29 22:33:39.273146 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260076 5057 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 29 22:33:39.273146 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260079 5057 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 29 22:33:39.273146 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260082 5057 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 29 22:33:39.273146 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260085 5057 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 29 22:33:39.273146 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260087 5057 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 29 22:33:39.273146 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260090 5057 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 29 22:33:39.273146 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260093 5057 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 29 22:33:39.273146 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260096 5057 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 29 22:33:39.273146 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260099 5057 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 29 22:33:39.273146 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260101 5057 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 29 22:33:39.273146 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260104 5057 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 29 22:33:39.273146 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260107 5057 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 29 22:33:39.273146 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260111 5057 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 29 22:33:39.273146 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260114 5057 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 29 22:33:39.273146 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260117 5057 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 29 22:33:39.273146 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260120 5057 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 29 22:33:39.273146 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260122 5057 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 29 22:33:39.273146 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260125 5057 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 29 22:33:39.273617 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260128 5057 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 29 22:33:39.273617 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260130 5057 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 29 22:33:39.273617 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260133 5057 feature_gate.go:386] unrecognized feature gate: Example Oct 29 22:33:39.273617 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260136 5057 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 29 22:33:39.273617 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260138 5057 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 29 22:33:39.273617 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260141 5057 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 29 22:33:39.273617 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260143 5057 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 29 22:33:39.273617 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260146 5057 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 29 22:33:39.273617 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260149 5057 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 29 22:33:39.273617 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260151 5057 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 29 22:33:39.273617 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260154 5057 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 29 22:33:39.273617 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260157 5057 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 29 22:33:39.273617 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260161 5057 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 29 22:33:39.273617 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260166 5057 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 29 22:33:39.273617 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260170 5057 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 29 22:33:39.273617 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260173 5057 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 29 22:33:39.273617 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260176 5057 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 29 22:33:39.273617 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260178 5057 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 29 22:33:39.273617 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260181 5057 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 29 22:33:39.274081 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260183 5057 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 29 22:33:39.274081 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260186 5057 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 29 22:33:39.274081 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260189 5057 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 29 22:33:39.274081 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260192 5057 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 29 22:33:39.274081 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260195 5057 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 29 22:33:39.274081 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260197 5057 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 29 22:33:39.274081 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260200 5057 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 29 22:33:39.274081 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260203 5057 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 29 22:33:39.274081 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260205 5057 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 29 22:33:39.274081 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260208 5057 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 29 22:33:39.274081 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260210 5057 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 29 22:33:39.274081 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260213 5057 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 29 22:33:39.274081 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260216 5057 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 29 22:33:39.274081 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260219 5057 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 29 22:33:39.274081 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260221 5057 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 29 22:33:39.274081 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260224 5057 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 29 22:33:39.274081 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260227 5057 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 29 22:33:39.274081 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260230 5057 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 29 22:33:39.274081 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260232 5057 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 29 22:33:39.274081 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260235 5057 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 29 22:33:39.274583 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260237 5057 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 29 22:33:39.274583 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.260266 5057 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 29 22:33:39.274583 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260419 5057 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 29 22:33:39.274583 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260424 5057 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 29 22:33:39.274583 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260427 5057 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 29 22:33:39.274583 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260430 5057 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 29 22:33:39.274583 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260433 5057 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 29 22:33:39.274583 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260436 5057 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 29 22:33:39.274583 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260439 5057 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 29 22:33:39.274583 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260441 5057 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 29 22:33:39.274583 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260444 5057 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 29 22:33:39.274583 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260447 5057 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 29 22:33:39.274583 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260449 5057 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 29 22:33:39.274583 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260452 5057 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 29 22:33:39.274583 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260455 5057 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 29 22:33:39.274583 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260457 5057 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 29 22:33:39.275283 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260460 5057 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 29 22:33:39.275283 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260462 5057 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 29 22:33:39.275283 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260465 5057 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 29 22:33:39.275283 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260468 5057 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 29 22:33:39.275283 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260470 5057 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 29 22:33:39.275283 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260473 5057 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 29 22:33:39.275283 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260475 5057 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 29 22:33:39.275283 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260478 5057 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 29 22:33:39.275283 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260480 5057 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 29 22:33:39.275283 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260483 5057 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 29 22:33:39.275283 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260485 5057 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 29 22:33:39.275283 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260488 5057 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 29 22:33:39.275283 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260490 5057 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 29 22:33:39.275283 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260493 5057 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 29 22:33:39.275283 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260495 5057 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 29 22:33:39.275283 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260498 5057 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 29 22:33:39.275283 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260501 5057 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 29 22:33:39.275283 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260505 5057 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 29 22:33:39.275283 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260509 5057 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 29 22:33:39.276086 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260513 5057 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 29 22:33:39.276086 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260516 5057 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 29 22:33:39.276086 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260518 5057 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 29 22:33:39.276086 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260521 5057 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 29 22:33:39.276086 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260524 5057 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 29 22:33:39.276086 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260527 5057 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 29 22:33:39.276086 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260529 5057 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 29 22:33:39.276086 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260532 5057 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 29 22:33:39.276086 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260534 5057 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 29 22:33:39.276086 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260537 5057 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 29 22:33:39.276086 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260540 5057 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 29 22:33:39.276086 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260543 5057 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 29 22:33:39.276086 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260545 5057 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 29 22:33:39.276086 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260553 5057 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 29 22:33:39.276086 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260556 5057 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 29 22:33:39.276086 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260559 5057 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 29 22:33:39.276086 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260561 5057 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 29 22:33:39.276086 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260564 5057 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 29 22:33:39.276086 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260566 5057 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 29 22:33:39.276595 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260569 5057 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 29 22:33:39.276595 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260572 5057 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 29 22:33:39.276595 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260574 5057 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 29 22:33:39.276595 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260577 5057 feature_gate.go:386] unrecognized feature gate: Example2 Oct 29 22:33:39.276595 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260579 5057 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 29 22:33:39.276595 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260582 5057 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 29 22:33:39.276595 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260584 5057 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 29 22:33:39.276595 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260587 5057 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 29 22:33:39.276595 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260590 5057 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 29 22:33:39.276595 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260592 5057 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 29 22:33:39.276595 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260595 5057 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 29 22:33:39.276595 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260597 5057 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 29 22:33:39.276595 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260600 5057 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 29 22:33:39.276595 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260603 5057 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 29 22:33:39.276595 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260606 5057 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 29 22:33:39.276595 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260608 5057 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 29 22:33:39.276595 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260611 5057 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 29 22:33:39.276595 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260613 5057 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 29 22:33:39.276595 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260616 5057 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 29 22:33:39.276595 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260619 5057 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 29 22:33:39.277119 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260622 5057 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 29 22:33:39.277119 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260625 5057 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 29 22:33:39.277119 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260627 5057 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 29 22:33:39.277119 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260633 5057 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 29 22:33:39.277119 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260636 5057 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 29 22:33:39.277119 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260638 5057 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 29 22:33:39.277119 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260641 5057 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 29 22:33:39.277119 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260643 5057 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 29 22:33:39.277119 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260646 5057 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 29 22:33:39.277119 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260649 5057 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 29 22:33:39.277119 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260651 5057 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 29 22:33:39.277119 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260654 5057 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 29 22:33:39.277119 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260656 5057 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 29 22:33:39.277119 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260659 5057 feature_gate.go:386] unrecognized feature gate: Example Oct 29 22:33:39.277119 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260661 5057 feature_gate.go:386] unrecognized feature gate: EVPN Oct 29 22:33:39.277119 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260664 5057 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 29 22:33:39.277119 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260666 5057 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 29 22:33:39.277119 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260669 5057 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 29 22:33:39.277119 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260671 5057 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 29 22:33:39.277119 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260674 5057 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 29 22:33:39.277821 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260677 5057 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 29 22:33:39.277821 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260679 5057 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 29 22:33:39.277821 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260682 5057 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 29 22:33:39.277821 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260684 5057 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 29 22:33:39.277821 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260687 5057 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 29 22:33:39.277821 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260690 5057 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 29 22:33:39.277821 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260694 5057 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 29 22:33:39.277821 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260697 5057 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 29 22:33:39.277821 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260701 5057 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 29 22:33:39.277821 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260703 5057 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 29 22:33:39.277821 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260709 5057 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 29 22:33:39.277821 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260712 5057 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 29 22:33:39.277821 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260715 5057 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 29 22:33:39.277821 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260718 5057 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 29 22:33:39.277821 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.260742 5057 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 29 22:33:39.278194 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260774 5057 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 29 22:33:39.278194 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260779 5057 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 29 22:33:39.278194 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260783 5057 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 29 22:33:39.278194 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260786 5057 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 29 22:33:39.278194 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260788 5057 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 29 22:33:39.278194 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260791 5057 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 29 22:33:39.278194 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260794 5057 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 29 22:33:39.278194 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260797 5057 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 29 22:33:39.278194 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260799 5057 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 29 22:33:39.278194 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260802 5057 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 29 22:33:39.278194 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260804 5057 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 29 22:33:39.278194 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260807 5057 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 29 22:33:39.278194 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260810 5057 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 29 22:33:39.278194 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260813 5057 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 29 22:33:39.278194 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260816 5057 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 29 22:33:39.278194 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260818 5057 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 29 22:33:39.278194 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260821 5057 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 29 22:33:39.278194 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260824 5057 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 29 22:33:39.278194 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260826 5057 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 29 22:33:39.278677 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260829 5057 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 29 22:33:39.278677 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260831 5057 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 29 22:33:39.278677 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260834 5057 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 29 22:33:39.278677 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260836 5057 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 29 22:33:39.278677 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260839 5057 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 29 22:33:39.278677 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260842 5057 feature_gate.go:386] unrecognized feature gate: Example2 Oct 29 22:33:39.278677 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260844 5057 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 29 22:33:39.278677 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260847 5057 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 29 22:33:39.278677 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260849 5057 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 29 22:33:39.278677 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260853 5057 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 29 22:33:39.278677 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260856 5057 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 29 22:33:39.278677 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260859 5057 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 29 22:33:39.278677 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260862 5057 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 29 22:33:39.278677 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260866 5057 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 29 22:33:39.278677 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260869 5057 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 29 22:33:39.278677 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260871 5057 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 29 22:33:39.278677 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260874 5057 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 29 22:33:39.278677 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260876 5057 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 29 22:33:39.278677 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260879 5057 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 29 22:33:39.278677 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260882 5057 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 29 22:33:39.279198 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260885 5057 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 29 22:33:39.279198 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260887 5057 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 29 22:33:39.279198 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260890 5057 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 29 22:33:39.279198 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260893 5057 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 29 22:33:39.279198 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260896 5057 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 29 22:33:39.279198 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260899 5057 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 29 22:33:39.279198 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260902 5057 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 29 22:33:39.279198 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260904 5057 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 29 22:33:39.279198 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260907 5057 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 29 22:33:39.279198 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260909 5057 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 29 22:33:39.279198 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260912 5057 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 29 22:33:39.279198 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260914 5057 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 29 22:33:39.279198 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260917 5057 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 29 22:33:39.279198 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260920 5057 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 29 22:33:39.279198 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260922 5057 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 29 22:33:39.279198 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260925 5057 feature_gate.go:386] unrecognized feature gate: Example Oct 29 22:33:39.279198 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260928 5057 feature_gate.go:386] unrecognized feature gate: EVPN Oct 29 22:33:39.279198 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260930 5057 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 29 22:33:39.279198 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260933 5057 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 29 22:33:39.279198 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260936 5057 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 29 22:33:39.280070 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260938 5057 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 29 22:33:39.280070 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260943 5057 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 29 22:33:39.280070 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260946 5057 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 29 22:33:39.280070 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260948 5057 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 29 22:33:39.280070 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260951 5057 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 29 22:33:39.280070 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260955 5057 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 29 22:33:39.280070 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260958 5057 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 29 22:33:39.280070 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260962 5057 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 29 22:33:39.280070 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260965 5057 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 29 22:33:39.280070 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260968 5057 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 29 22:33:39.280070 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260971 5057 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 29 22:33:39.280070 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260974 5057 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 29 22:33:39.280070 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260976 5057 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 29 22:33:39.280070 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260979 5057 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 29 22:33:39.280070 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260982 5057 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 29 22:33:39.280070 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260984 5057 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 29 22:33:39.280070 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260987 5057 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 29 22:33:39.280070 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260990 5057 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 29 22:33:39.280070 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260992 5057 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 29 22:33:39.280904 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260995 5057 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 29 22:33:39.280904 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.260998 5057 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 29 22:33:39.280904 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.261000 5057 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 29 22:33:39.280904 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.261003 5057 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 29 22:33:39.280904 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.261006 5057 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 29 22:33:39.280904 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.261008 5057 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 29 22:33:39.280904 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.261011 5057 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 29 22:33:39.280904 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.261013 5057 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 29 22:33:39.280904 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.261016 5057 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 29 22:33:39.280904 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.261018 5057 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 29 22:33:39.280904 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.261021 5057 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 29 22:33:39.280904 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.261023 5057 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 29 22:33:39.280904 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.261026 5057 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 29 22:33:39.280904 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.261028 5057 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 29 22:33:39.280904 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.261032 5057 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 29 22:33:39.280904 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.261035 5057 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 29 22:33:39.280904 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.261038 5057 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 29 22:33:39.280904 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.261040 5057 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 29 22:33:39.280904 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.261045 5057 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 29 22:33:39.280904 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.261047 5057 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 29 22:33:39.281727 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.261050 5057 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 29 22:33:39.281727 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.261052 5057 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 29 22:33:39.281727 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.261055 5057 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 29 22:33:39.281727 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.261057 5057 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 29 22:33:39.281727 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.261060 5057 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 29 22:33:39.281727 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.261063 5057 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 29 22:33:39.281727 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.261065 5057 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 29 22:33:39.281727 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:39.261069 5057 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 29 22:33:39.281727 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.261127 5057 server.go:957] "Client rotation is on, will bootstrap in background" Oct 29 22:33:39.281727 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.262067 5057 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Oct 29 22:33:39.281727 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.262125 5057 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Oct 29 22:33:39.281727 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.262296 5057 server.go:1014] "Starting client certificate rotation" Oct 29 22:33:39.281727 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.262429 5057 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 29 22:33:39.281727 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.262485 5057 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-28 22:27:58 +0000 UTC" deadline="2028-07-13 07:26:15.226109718 +0000 UTC" Oct 29 22:33:39.281727 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.262512 5057 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14936h52m35.96359998s" Oct 29 22:33:39.281727 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.262754 5057 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 29 22:33:39.282139 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.263828 5057 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 29 22:33:39.282139 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.265217 5057 log.go:25] "Validated CRI v1 runtime API" Oct 29 22:33:39.282139 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.266916 5057 log.go:25] "Validated CRI v1 image API" Oct 29 22:33:39.282139 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.267672 5057 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 29 22:33:39.282139 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.269629 5057 fs.go:135] Filesystem UUIDs: map[033799f6-a5e8-4944-83ed-8553f8110b52:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 dead43f6-6552-4c20-83b8-c085fcfbbaa6:/dev/nvme0n1p3] Oct 29 22:33:39.282292 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.269650 5057 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/137914b9aa694c6fd2799975b8439b50e6c4d98b6b27025bd3874923c51ad0c6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/137914b9aa694c6fd2799975b8439b50e6c4d98b6b27025bd3874923c51ad0c6/userdata/shm major:0 minor:323 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2c31fee02392017ba8b41b846564c28cb85ccd960603212936086d59f108fd8a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2c31fee02392017ba8b41b846564c28cb85ccd960603212936086d59f108fd8a/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2dbb0c4a7640796e280454d201a82932159022b408fb05d309bb0cdc047866c6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2dbb0c4a7640796e280454d201a82932159022b408fb05d309bb0cdc047866c6/userdata/shm major:0 minor:245 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/39746dc1a72c981c5e9698f1f0c72d7c28b892b2793847e6fcf516fb3f1c727f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/39746dc1a72c981c5e9698f1f0c72d7c28b892b2793847e6fcf516fb3f1c727f/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3f2aff1dfc2f332ef98875b32540bc94544ff46bc96652dcfd3100ca3434ad2a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3f2aff1dfc2f332ef98875b32540bc94544ff46bc96652dcfd3100ca3434ad2a/userdata/shm major:0 minor:356 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/532a2188ba13d70f4724205518db4794b66779cf1496ec8df5fcd266cd31d6db/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/532a2188ba13d70f4724205518db4794b66779cf1496ec8df5fcd266cd31d6db/userdata/shm major:0 minor:74 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/54fa8fc945d5c93ccfdd34d26d813ccb81a4c450cd4a8ed1ced70ba9319e4dad/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/54fa8fc945d5c93ccfdd34d26d813ccb81a4c450cd4a8ed1ced70ba9319e4dad/userdata/shm major:0 minor:238 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/556b333946dd2fef6c2935b79d83a3334ac5de4f63fb2b425f3b36f57a4abca1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/556b333946dd2fef6c2935b79d83a3334ac5de4f63fb2b425f3b36f57a4abca1/userdata/shm major:0 minor:83 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6d1c35a93c18cc546622631e919464a48df503c2a5b066a732af5226f05dce9c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6d1c35a93c18cc546622631e919464a48df503c2a5b066a732af5226f05dce9c/userdata/shm major:0 minor:242 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6f79e832218106c064e6731373a1925dd3862c0b2ad6865dfe89b297a3ff15c2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6f79e832218106c064e6731373a1925dd3862c0b2ad6865dfe89b297a3ff15c2/userdata/shm major:0 minor:82 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/74ff81a79115f8387b397a72e407635fe9068a1bdfda4852af15c18dcc2b2181/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/74ff81a79115f8387b397a72e407635fe9068a1bdfda4852af15c18dcc2b2181/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7f2c50e656caef7bde619a98fc6022b5a14ed2ab51fb4b904fe9185e4ef04c86/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7f2c50e656caef7bde619a98fc6022b5a14ed2ab51fb4b904fe9185e4ef04c86/userdata/shm major:0 minor:240 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8aec9d2b53599ac849ba2c32c12dd340a8651fcd1e0f2ff829e14e25ffd558ee/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8aec9d2b53599ac849ba2c32c12dd340a8651fcd1e0f2ff829e14e25ffd558ee/userdata/shm major:0 minor:228 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/906939625d9827aaf66a0763c523390560b22eb094cee8e38b2583671e2bffa2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/906939625d9827aaf66a0763c523390560b22eb094cee8e38b2583671e2bffa2/userdata/shm major:0 minor:70 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9bb152b2ed8e5b9f7090432bf0a139a9d586707e7d782cba121f6bb7b126aed4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9bb152b2ed8e5b9f7090432bf0a139a9d586707e7d782cba121f6bb7b126aed4/userdata/shm major:0 minor:244 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a19be32c77f0d44cd8abc3ba5e326985413ea3d9dcd29dada2ca1c237c96f2c2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a19be32c77f0d44cd8abc3ba5e326985413ea3d9dcd29dada2ca1c237c96f2c2/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a850aad9d1fa77621f0945112e249020d542379254765dba31625cadfce0b375/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a850aad9d1fa77621f0945112e249020d542379254765dba31625cadfce0b375/userdata/shm major:0 minor:232 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b86a3f547def9a615d8d32a9c5db979d9fc77c413b9595f6b4fca5be95b62f4d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b86a3f547def9a615d8d32a9c5db979d9fc77c413b9595f6b4fca5be95b62f4d/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ccdd3c46b100f9e0da773d4ceccf7e6ea046a4f55344cee20eea5b936e882816/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ccdd3c46b100f9e0da773d4ceccf7e6ea046a4f55344cee20eea5b936e882816/userdata/shm major:0 minor:230 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d4c08358fd86fbf544eeb14448141bb045e6b6e45eb23b1d549aa8870fe1c4e1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d4c08358fd86fbf544eeb14448141bb045e6b6e45eb23b1d549aa8870fe1c4e1/userdata/shm major:0 minor:236 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d5f1b36c251b39df9f18518a01a694a79b6ab1196949d53c53d8b750c0775edd/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d5f1b36c251b39df9f18518a01a694a79b6ab1196949d53c53d8b750c0775edd/userdata/shm major:0 minor:234 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d7df54a9e709ebfabcf14a3ad76c25faf0cc254389004e41ba7f1eded00b1a80/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d7df54a9e709ebfabcf14a3ad76c25faf0cc254389004e41ba7f1eded00b1a80/userdata/shm major:0 minor:294 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e6876edb30d42c86b24074e617f02daea0bf1d967904b4505b4df4ce737d3508/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e6876edb30d42c86b24074e617f02daea0bf1d967904b4505b4df4ce737d3508/userdata/shm major:0 minor:358 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ef6db9261bebd09853a31da06a9228ae19172ae39bb69d4c5d723a543410dd82/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ef6db9261bebd09853a31da06a9228ae19172ae39bb69d4c5d723a543410dd82/userdata/shm major:0 minor:91 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f2028b5b83ac2c14a3e4c4b56a439b3dcf82bf043749a960aa5867a0992c7837/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f2028b5b83ac2c14a3e4c4b56a439b3dcf82bf043749a960aa5867a0992c7837/userdata/shm major:0 minor:296 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ffc8236d320661b5d814a75ef36e7f088f46bf70179419b19ecb9da171ab1ab4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ffc8236d320661b5d814a75ef36e7f088f46bf70179419b19ecb9da171ab1ab4/userdata/shm major:0 minor:348 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/04195c1d-9fb7-479b-8061-f92e3c88aed2/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/04195c1d-9fb7-479b-8061-f92e3c88aed2/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/04195c1d-9fb7-479b-8061-f92e3c88aed2/volumes/kubernetes.io~projected/kube-api-access-tlxfr:{mountpoint:/var/lib/kubelet/pods/04195c1d-9fb7-479b-8061-f92e3c88aed2/volumes/kubernetes.io~projected/kube-api-access-tlxfr major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/04195c1d-9fb7-479b-8061-f92e3c88aed2/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/04195c1d-9fb7-479b-8061-f92e3c88aed2/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/04ff31d0-a870-4527-a0c3-285737359f95/volumes/kubernetes.io~projected/kube-api-access-8jxgn:{mountpoint:/var/lib/kubelet/pods/04ff31d0-a870-4527-a0c3-285737359f95/volumes/kubernetes.io~projected/kube-api-access-8jxgn major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/079ed4e4-bdc4-405c-90a1-a22b84fcf948/volumes/kubernetes.io~projected/kube-api-access-ln6mw:{mountpoint:/var/lib/kubelet/pods/079ed4e4-bdc4-405c-90a1-a22b84fcf948/volumes/kubernetes.io~projected/kube-api-access-ln6mw major:0 minor:312 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0e1c2279-b6d4-44e0-84e6-94eb1206d2a0/volumes/kubernetes.io~projected/kube-api-access-7fmsg:{mountpoint:/var/lib/kubelet/pods/0e1c2279-b6d4-44e0-84e6-94eb1206d2a0/volumes/kubernetes.io~projected/kube-api-access-7fmsg major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2ab2aa22-99af-4451-8d33-9e3543c8d9cd/volumes/kubernetes.io~projected/kube-api-access-9v2r9:{mountpoint:/var/lib/kubelet/pods/2ab2aa22-99af-4451-8d33-9e3543c8d9cd/volumes/kubernetes.io~projected/kube-api-access-9v2r9 major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2d2d7446-0741-4144-9b19-608854f8a04e/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/2d2d7446-0741-4144-9b19-608854f8a04e/volumes/kubernetes.io~secret/agent-certs major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4071b0c7-c4f3-419a-8d6e-6acc144fc9a4/volumes/kubernetes.io~projected/kube-api-access-n8hb6:{mountpoint:/var/lib/kubelet/pods/4071b0c7-c4f3-419a-8d6e-6acc144fc9a4/volumes/kubernetes.io~projected/kube-api-access-n8hb6 major:0 minor:224 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4071b0c7-c4f3-419a-8d6e-6acc144fc9a4/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/4071b0c7-c4f3-419a-8d6e-6acc144fc9a4/volumes/kubernetes.io~secret/serving-cert major:0 minor:222 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/49c67ce3-90fd-478d-9a15-e4267438ff69/volumes/kubernetes.io~projected/kube-api-access-9qd7k:{mountpoint:/var/lib/kubelet/pods/49c67ce3-90fd-478d-9a15-e4267438ff69/volumes/kubernetes.io~projected/kube-api-access-9qd7k major:0 minor:354 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/49c67ce3-90fd-478d-9a15-e4267438ff69/volumes/kubernetes.io~secret/signing-key:{mountpoint:/var/lib/kubelet/pods/49c67ce3-90fd-478d-9a15-e4267438ff69/volumes/kubernetes.io~secret/signing-key major:0 minor:353 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4ff372bf-82aa-4007-b74c-df0c779e5271/volumes/kubernetes.io~projected/kube-api-access-whjbn:{mountpoint:/var/lib/kubelet/pods/4ff372bf-82aa-4007-b74c-df0c779e5271/volumes/kubernetes.io~projected/kube-api-access-whjbn major:0 minor:209 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5296979a-7927-4ee2-84b9-a93780a2ad22/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/5296979a-7927-4ee2-84b9-a93780a2ad22/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:210 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5296979a-7927-4ee2-84b9-a93780a2ad22/volumes/kubernetes.io~projected/kube-api-access-fp2lh:{mountpoint:/var/lib/kubelet/pods/5296979a-7927-4ee2-84b9-a93780a2ad22/volumes/kubernetes.io~projected/kube-api-access-fp2lh major:0 minor:217 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5296979a-7927-4ee2-84b9-a93780a2ad22/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/5296979a-7927-4ee2-84b9-a93780a2ad22/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5296979a-7927-4ee2-84b9-a93780a2ad22/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/5296979a-7927-4ee2-84b9-a93780a2ad22/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/587893f5-e85a-46a9-829f-545a42c10056/volumes/kubernetes.io~projected/kube-api-access-bjwwn:{mountpoint:/var/lib/kubelet/pods/587893f5-e85a-46a9-829f-545a42c10056/volumes/kubernetes.io~projected/kube-api-access-bjwwn major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/641f9ec1-c8e2-4774-9d4f-6a7a2da97b6f/volumes/kubernetes.io~projected/kube-api-access-6sv4r:{mountpoint:/var/lib/kubelet/pods/641f9ec1-c8e2-4774-9d4f-6a7a2da97b6f/volumes/kubernetes.io~projected/kube-api-access-6sv4r major:0 minor:227 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/652bedff-87f1-4b2f-917f-571a37dd2f3f/volumes/kubernetes.io~projected/kube-api-access-mfd8g:{mountpoint:/var/lib/kubelet/pods/652bedff-87f1-4b2f-917f-571a37dd2f3f/volumes/kubernetes.io~projected/kube-api-access-mfd8g major:0 minor:335 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/65c96dfb-06e9-4bea-bb83-f63b4c62d860/volumes/kubernetes.io~projected/kube-api-access-z2tw5:{mountpoint:/var/lib/kubelet/pods/65c96dfb-06e9-4bea-bb83-f63b4c62d860/volumes/kubernetes.io~projected/kube-api-access-z2tw5 major:0 minor:215 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/65c96dfb-06e9-4bea-bb83-f63b4c62d860/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/65c96dfb-06e9-4bea-bb83-f63b4c62d860/volumes/kubernetes.io~secret/serving-cert major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6b34b3b1-84af-45f3-92ee-1ca720e8b263/volumes/kubernetes.io~projected/kube-api-access-8gw9q:{mountpoint:/var/lib/kubelet/pods/6b34b3b1-84af-45f3-92ee-1ca720e8b263/volumes/kubernetes.io~projected/kube-api-access-8gw9q major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6f50ee57-a710-4325-bd45-4f861452ede4/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/6f50ee57-a710-4325-bd45-4f861452ede4/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:288 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/71d80eec-b5e7-41d0-b58d-8a8d9a6c769c/volumes/kubernetes.io~projected/kube-api-access-2tdnl:{mountpoint:/var/lib/kubelet/pods/71d80eec-b5e7-41d0-b58d-8a8d9a6c769c/volumes/kubernetes.io~projected/kube-api-access-2tdnl major:0 minor:219 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/71d80eec-b5e7-41d0-b58d-8a8d9a6c769c/volumes/kubernetes.io~secret/default-certificate:{mountpoint:/var/lib/kubelet/pods/71d80eec-b5e7-41d0-b58d-8a8d9a6c769c/volumes/kubernetes.io~secret/default-certificate major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/71d80eec-b5e7-41d0-b58d-8a8d9a6c769c/volumes/kubernetes.io~secret/stats-auth:{mountpoint:/var/lib/kubelet/pods/71d80eec-b5e7-41d0-b58d-8a8d9a6c769c/volumes/kubernetes.io~secret/stats-auth major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/79ad3230-163e-447d-9f04-a072479ff957/volumes/kubernetes.io~projected/kube-api-access-vv2vp:{mountpoint:/var/lib/kubelet/pods/79ad3230-163e-447d-9f04-a072479ff957/volumes/kubernetes.io~projected/kube-api-access-vv2vp major:0 minor:220 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8406bb9f-3a4e-42f3-a822-f754d9b8e7a8/volumes/kubernetes.io~projected/kube-api-access-xvmbg:{mountpoint:/var/lib/kubelet/pods/8406bb9f-3a4e-42f3-a822-f754d9b8e7a8/volumes/kubernetes.io~projected/kube-api-access-xvmbg major:0 minor:226 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8406bb9f-3a4e-42f3-a822-f754d9b8e7a8/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/8406bb9f-3a4e-42f3-a822-f754d9b8e7a8/volumes/kubernetes.io~secret/cert major:0 minor:221 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8a972cc8-dce5-4f7c-b5d5-cb24ccf73bad/volumes/kubernetes.io~projected/kube-api-access-jvgj5:{mountpoint:/var/lib/kubelet/pods/8a972cc8-dce5-4f7c-b5d5-cb24ccf73bad/volumes/kubernetes.io~projected/kube-api-access-jvgj5 major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/93dbae94-f92d-4157-a89b-d3c984a56013/volumes/kubernetes.io~projected/kube-api-access-wf78m:{mountpoint:/var/lib/kubelet/pods/93dbae94-f92d-4157-a89b-d3c984a56013/volumes/kubernetes.io~projected/kube-api-access-wf78m major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/989956e8-1893-4bbe-a4bf-6a2081509080/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/989956e8-1893-4bbe-a4bf-6a2081509080/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:216 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/989956e8-1893-4bbe-a4bf-6a2081509080/volumes/kubernetes.io~projected/kube-api-access-gfk2t:{mountpoint:/var/lib/kubelet/pods/989956e8-1893-4bbe-a4bf-6a2081509080/volumes/kubernetes.io~projected/kube-api-access-gfk2t major:0 minor:214 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/989956e8-1893-4bbe-a4bf-6a2081509080/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/989956e8-1893-4bbe-a4bf-6a2081509080/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/989956e8-1893-4bbe-a4bf-6a2081509080/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/989956e8-1893-4bbe-a4bf-6a2081509080/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:205 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a4e1c52c-1eeb-4617-a0c9-d052521360d3/volumes/kubernetes.io~projected/kube-api-access-l9srv:{mountpoint:/var/lib/kubelet/pods/a4e1c52c-1eeb-4617-a0c9-d052521360d3/volumes/kubernetes.io~projected/kube-api-access-l9srv major:0 minor:293 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a53078b9-a6a8-423c-b852-34eececa7ab1/volumes/kubernetes.io~projected/kube-api-access-698ds:{mountpoint:/var/lib/kubelet/pods/a53078b9-a6a8-423c-b852-34eececa7ab1/volumes/kubernetes.io~projected/kube-api-access-698ds major:0 minor:213 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/aa6a190e-3283-458a-9dfd-2de7cb193fc5/volumes/kubernetes.io~projected/kube-api-access-c4xwz:{mountpoint:/var/lib/kubelet/pods/aa6a190e-3283-458a-9dfd-2de7cb193fc5/volumes/kubernetes.io~projected/kube-api-access-c4xwz major:0 minor:211 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/aa6a190e-3283-458a-9dfd-2de7cb193fc5/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/aa6a190e-3283-458a-9dfd-2de7cb193fc5/volumes/kubernetes.io~secret/serving-cert major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b23027ca-0748-4125-9bc8-9d149a3418e0/volumes/kubernetes.io~projected/kube-api-access-jqxx5:{mountpoint:/var/lib/kubelet/pods/b23027ca-0748-4125-9bc8-9d149a3418e0/volumes/kubernetes.io~projected/kube-api-access-jqxx5 major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b5961d4b-f3c0-4f17-b1bc-2781cb3dcd6e/volumes/kubernetes.io~projected/kube-api-access-hfdj9:{mountpoint:/var/lib/kubelet/pods/b5961d4b-f3c0-4f17-b1bc-2781cb3dcd6e/volumes/kubernetes.io~projected/kube-api-access-hfdj9 major:0 minor:334 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c54ef754-c91e-4bc5-8e83-08523aa09d37/volumes/kubernetes.io~projected/kube-api-access-7h8fj:{mountpoint:/var/lib/kubelet/pods/c54ef754-c91e-4bc5-8e83-08523aa09d37/volumes/kubernetes.io~projected/kube-api-access-7h8fj major:0 minor:212 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c635d035-053b-49aa-9559-79c31e8e49ff/volumes/kubernetes.io~projected/kube-api-access-mkxzs:{mountpoint:/var/lib/kubelet/pods/c635d035-053b-49aa-9559-79c31e8e49ff/volumes/kubernetes.io~projected/kube-api-access-mkxzs major:0 minor:218 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c635d035-053b-49aa-9559-79c31e8e49ff/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/c635d035-053b-49aa-9559-79c31e8e49ff/volumes/kubernetes.io~secret/serving-cert major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dd4e19ce-ef37-44e5-8b57-9dafc590685c/volumes/kubernetes.io~projected/kube-api-access-l8d5d:{mountpoint:/var/lib/kubelet/pods/dd4e19ce-ef37-44e5-8b57-9dafc590685c/volumes/kubernetes.io~projected/kube-api-access-l8d5d major:0 minor:223 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f176efe0-febd-4652-94fc-91548ee3ab93/volumes/kubernetes.io~projected/kube-api-access-p9pvq:{mountpoint:/var/lib/kubelet/pods/f176efe0-febd-4652-94fc-91548ee3ab93/volumes/kubernetes.io~projected/kube-api-access-p9pvq major:0 minor:328 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f190a1a0-6f6a-44d0-aeaa-b053cff4bd9c/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/f190a1a0-6f6a-44d0-aeaa-b053cff4bd9c/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f190a1a0-6f6a-44d0-aeaa-b053cff4bd9c/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/f190a1a0-6f6a-44d0-aeaa-b053cff4bd9c/volumes/kubernetes.io~empty-dir/tmp major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f190a1a0-6f6a-44d0-aeaa-b053cff4bd9c/volumes/kubernetes.io~projected/kube-api-access-4mxpd:{mountpoint:/var/lib/kubelet/pods/f190a1a0-6f6a-44d0-aeaa-b053cff4bd9c/volumes/kubernetes.io~projected/kube-api-access-4mxpd major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f8f92bfb-5cef-412e-a6c1-4bd793c59fe7/volumes/kubernetes.io~projected/kube-api-access-4nczb:{mountpoint:/var/lib/kubelet/pods/f8f92bfb-5cef-412e-a6c1-4bd793c59fe7/volumes/kubernetes.io~projected/kube-api-access-4nczb major:0 minor:66 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/5184dea1ea3dabfe05ba909eb989e4efc093ba31248b54f80938546c04b3434a/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/cf9e6955c8ae6dd792967984875c2a7cb423ad4b222dec87f85dff239a7898a6/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/c50224a4674fc9b1788383409c9aa17880d79d388e5fbc064c4b0b9687fb6a5b/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/0c141c273585de0036c0e743c572895ec57e25b13616baf3c901d150fb0e7ad4/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/cfe6c5050f27a4d12965e78a1c73d78888586cce3cc3ee879fd6eda780de08a1/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/ed5676187a32d098cb9fce993fc0a797d585b8b82d1b1070e5a90f1fac2add48/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/4cf9d405d9b71e9463c9894325523596f6932e304edee3916e12fa758c53af69/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/cc9e2a27da61e95de1049d8e36c38121b1700ded9140509936eacff2c738de0f/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/c981c5a10d6632d43020870c3a4f0ecdfbbc5c409b5ad0685594a750e76eaab1/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-136:{mountpoint:/var/lib/containers/storage/overlay/87529f6a5bdf6ea527103cedcff88440e6f8e3d2b1ba07a7b4f635a311d52528/merged major:0 minor:136 fsType:overlay blockSize:0} overlay_0-138:{mountpoint:/var/lib/containers/storage/overlay/69e066a826c70716f5d5d33fccae7e6745865a70198837cef4fee97af7f5d22f/merged major:0 minor:138 fsType:overlay blockSize:0} overlay_0-143:{mountpoint:/var/lib/containers/storage/overlay/108771d873aaccad718ef0a302216690891fdccd6f0ce91a7f65e772656152b1/merged major:0 minor:143 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/c3a5b00929d3e392b4b4fc1f71e922437cccace28afd8a127ce50f9e04b14f2b/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-154:{mountpoint:/var/lib/containers/storage/overlay/04bd52c31292d069056c85476e20fb71d7a84b904076f5daa8aa36c7ae170710/merged major:0 minor:154 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/d25fc493ac58a6ab21951397338298cfcd11875265eda18472ae0b440013824c/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/3f815af5f95861e06dbb15f38854c89b301051c8ea529a51f87ad5f8b3a160f3/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/4b2a1ee300e140709d052e8b6cf0e8fcc713d0315698f954e57bb3a531c04830/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/cc9a21e417cb449e4c52169971adf4627ee892a3bc3035ed5f70a6cdd1391fa3/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/ce06e1832defe4c5100fc16386e25ef25b585e3bb662449b06de22c4267c6731/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/c21d348c1ff594ffe1183d14b900aa5d302ec81175e2ef4acac0778e7f721b84/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/189ec609eefea12a618260ca4e27f7fd8eb134ad370f0cd4a4e3563891b05f04/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/b423c116d6bf5a6d54e18a4bde61918dbb08e254653bf16539fda6bf4149fd8c/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-250:{mountpoint:/var/lib/containers/storage/overlay/5ba5c6e908d74195001e1a2a3b921be09e0e495b601bd149afceceeb85469921/merged major:0 minor:250 fsType:overlay blockSize:0} overlay_0-252:{mountpoint:/var/lib/containers/storage/overlay/6d81116e840ac663244e63d2806e85f373f03f990f47cde43db0865ff80a2778/merged major:0 minor:252 fsType:overlay blockSize:0} overlay_0-254:{mountpoint:/var/lib/containers/storage/overlay/b2be3cb014f002c4c0da0d4a09943c49d177321ea8f15b85c3f579933ffd9b4c/merged major:0 minor:254 fsType:overlay blockSize:0} overlay_0-256:{mountpoint:/var/lib/containers/storage/overlay/1e20b22d61b459ddee47fa2626446d6020a12b4df09bab06aca3234c6b070bd6/merged major:0 minor:256 fsType:overlay blockSize:0} overlay_0-258:{mountpoint:/var/lib/containers/storage/overlay/1d835e04fb2bc37cb436f144f366a4cc17ff5ae06336532f0930a4e45f3eb130/merged major:0 minor:258 fsType:overlay blockSize:0} overlay_0-260:{mountpoint:/var/lib/containers/storage/overlay/6a0879ea4b62e8ba1d3e2905763a0f52e34be270893eaa8bd8063f7db04bca31/merged major:0 minor:260 fsType:overlay blockSize:0} overlay_0-262:{mountpoint:/var/lib/containers/storage/overlay/156f4aadaab5f9b0480e3cd4af2a4f0af247ac5c500ce4739b5c43276bf52c82/merged major:0 minor:262 fsType:overlay blockSize:0} overlay_0-264:{mountpoint:/var/lib/containers/storage/overlay/1bac1e3fa47817d1ccd2f00b69796a6cdadec070c3eed6917547baed6f78169c/merged major:0 minor:264 fsType:overlay blockSize:0} overlay_0-266:{mountpoint:/var/lib/containers/storage/overlay/4794d7de6f951b2c6420b79dfa835a063652acdeaaea7bdf4b8bc469d5d17e31/merged major:0 minor:266 fsType:overlay blockSize:0} overlay_0-276:{mountpoint:/var/lib/containers/storage/overlay/715b91641ea5b1671b8796a645c72e34fe2727f604c00e8462e73f8a0b5b5cd0/merged major:0 minor:276 fsType:overlay blockSize:0} overlay_0-278:{mountpoint:/var/lib/containers/storage/overlay/4c607032b0eb46143f5c0c97c93b68e0cd4a15537a5c23d04009afb690bbe569/merged major:0 minor:278 fsType:overlay blockSize:0} overlay_0-283:{mountpoint:/var/lib/containers/storage/overlay/774efb60ef6026b088064f4544e454210dc648499736d8efaef00b6b5e51ef7c/merged major:0 minor:283 fsType:overlay blockSize:0} overlay_0-285:{mountpoint:/var/lib/containers/storage/overlay/663881d1c759592ad178c028ffa9268655d7bf07d4322ed05524140c565ee26c/merged major:0 minor:285 fsType:overlay blockSize:0} overlay_0-298:{mountpoint:/var/lib/containers/storage/overlay/920c4ca8c1db115c86e29122b1567db400ab08a447b587389e430bfec26b484f/merged major:0 minor:298 fsType:overlay blockSize:0} overlay_0-300:{mountpoint:/var/lib/containers/storage/overlay/756d2e120afae2c759dc55f5c0c1fa2d05f5869686dde7d9276ba6e875f26801/merged major:0 minor:300 fsType:overlay blockSize:0} overlay_0-302:{mountpoint:/var/lib/containers/storage/overlay/8adc904d6db8926c9824262549acf6e5f382355c93b155daf287abae7521b872/merged major:0 minor:302 fsType:overlay blockSize:0} overlay_0-304:{mountpoint:/var/lib/containers/storage/overlay/7179e065aaa4987ed3d6f25b9155474c8dd18775ec1438e46c052062a024d727/merged major:0 minor:304 fsType:overlay blockSize:0} overlay_0-310:{mountpoint:/var/lib/containers/storage/overlay/3daf2f78659b557421d5ac67184b953e3f03d4475f7f1a0ab02ad40a4ddc27fb/merged major:0 minor:310 fsType:overlay blockSize:0} overlay_0-313:{mountpoint:/var/lib/containers/storage/overlay/bde464cea9a00a92b2dc78f25e664cc964345ce14f6aafd118be7bc4cd623dc3/merged major:0 minor:313 fsType:overlay blockSize:0} overlay_0-330:{mountpoint:/var/lib/containers/storage/overlay/4179484f43bf3632fcaa1c5dcb321af440bef604265418d41f6d00aeed84ea3c/merged major:0 minor:330 fsType:overlay blockSize:0} overlay_0-332:{mountpoint:/var/lib/containers/storage/overlay/94712d838801baf3eb9d8cebf79473b5865da92cfd641d5e8c7c1e4499b5ba69/merged major:0 minor:332 fsType:overlay blockSize:0} overlay_0-344:{mountpoint:/var/lib/containers/storage/overlay/78b9b116661bec264dd56cf561782c03be9a5638d13c52b5852a7c05ae98bcd5/merged major:0 minor:344 fsType:overlay blockSize:0} overlay_0-346:{mountpoint:/var/lib/containers/storage/overlay/a49975229e960d6245ab39f0373e2a9b0b25d3069da8259cbe4d6921f9ad6d5a/merged major:0 minor:346 fsType:overlay blockSize:0} overlay_0-360:{mountpoint:/var/lib/containers/storage/overlay/3c335c4aba4ef7e4088dd0092c143248804af5f75bd23dd824a512dd43798261/merged major:0 minor:360 fsType:overlay blockSize:0} overlay_0-362:{mountpoint:/var/lib/containers/storage/overlay/69dae6b0dda86b06342e2a35d73cd94b8dad174f7a0c14325597fb7a48edace9/merged major:0 minor:362 fsType:overlay blockSize:0} overlay_0-364:{mountpoint:/var/lib/containers/storage/overlay/000bc272c2fdb8509eb845fd7c3f78b4371202e3f2efaadf61f6776cf3948e83/merged major:0 minor:364 fsType:overlay blockSize:0} overlay_0-366:{mountpoint:/var/lib/containers/storage/overlay/4b0fabe2683cb41e018ffb5ce676664f46f350448e5eefef636871ced09c3d6a/merged major:0 minor:366 fsType:overlay blockSize:0} overlay_0-368:{mountpoint:/var/lib/containers/storage/overlay/7aec614e44ea75cf81d8634575cb296237a33c80ece9b7b1e953741e42da741c/merged major:0 minor:368 fsType:overlay blockSize:0} overlay_0-374:{mountpoint:/var/lib/containers/storage/overlay/9f35e4251603d34d09abea04724a5637f2b932c6fb4dce9707b4c937ba6f881b/merged major:0 minor:374 fsType:overlay blockSize:0} overlay_0-376:{mountpoint:/var/lib/containers/storage/overlay/b5fa25f3ab9070b8020018aa711b910f9c2a196891be9cd988fbbe85d79d2698/merged major:0 minor:376 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/4b7e0f950bb048c828a64fb59f53a604b2841c2be169f5be3df4c4177be2c77b/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/bb6d9b69b2ba643854a874fad3bc90a777f7c2d232cb4a05d89482021981b86f/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-68:{mountpoint:/var/lib/containers/storage/overlay/180a8e19cdd8344a836b91edefeb9e8595f0ebbe6d244e23b36e061fe672bedc/merged major:0 minor:68 fsType:overlay blockSize:0} overlay_0-72:{mountpoint:/var/lib/containers/storage/overlay/077315b6bbf533fcb26597aad692f8df45aaeb1f4134c0223fa6e743a8e63aa0/merged major:0 minor:72 fsType:overlay blockSize:0} overlay_0-77:{mountpoint:/var/lib/containers/storage/overlay/7d3546eccd95a375d35e6eaeaeb2ff07b5a7a883a0d7021b44ea85f89078aca5/merged major:0 minor:77 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/a99595c5bb2af0639620c539f487b4fba5ccee32522699b6fc2c7c2a9f74c111/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/342ede6e3aa25a3f4eb368ad8800095d8d711a8d931868d1b8f2a0867877eff3/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/96307f4a79f0b64090872f76667bc16fd0dd00c8ae6b93f004ba8b099f8ce5ba/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-93:{mountpoint:/var/lib/containers/storage/overlay/e995931e981423a2620a706b1db518426a05388eff9f7391b55e3be67ae99786/merged major:0 minor:93 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/79b5944db1c9487ebf44e0bc9d8dff4d64f6cabe813de61e1cb00b5796b90aa6/merged major:0 minor:96 fsType:overlay blockSize:0}] Oct 29 22:33:39.282572 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.278745 5057 manager.go:217] Machine: {Timestamp:2026-10-29 22:33:39.278272585 +0000 UTC m=+0.061486005 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099692 MemoryCapacity:33164337152 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2f41544626a51e83d19832b219b9de SystemUUID:ec2f4154-4626-a51e-83d1-9832b219b9de BootID:0970734c-b3cc-455e-b107-15f90ef1f7ba Filesystems:[{Device:/var/lib/kubelet/pods/04195c1d-9fb7-479b-8061-f92e3c88aed2/volumes/kubernetes.io~projected/kube-api-access-tlxfr DeviceMajor:0 DeviceMinor:56 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/04195c1d-9fb7-479b-8061-f92e3c88aed2/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/4071b0c7-c4f3-419a-8d6e-6acc144fc9a4/volumes/kubernetes.io~projected/kube-api-access-n8hb6 DeviceMajor:0 DeviceMinor:224 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/8406bb9f-3a4e-42f3-a822-f754d9b8e7a8/volumes/kubernetes.io~projected/kube-api-access-xvmbg DeviceMajor:0 DeviceMinor:226 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-254 DeviceMajor:0 DeviceMinor:254 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6f50ee57-a710-4325-bd45-4f861452ede4/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:288 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-346 DeviceMajor:0 DeviceMinor:346 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/556b333946dd2fef6c2935b79d83a3334ac5de4f63fb2b425f3b36f57a4abca1/userdata/shm DeviceMajor:0 DeviceMinor:83 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ccdd3c46b100f9e0da773d4ceccf7e6ea046a4f55344cee20eea5b936e882816/userdata/shm DeviceMajor:0 DeviceMinor:230 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-276 DeviceMajor:0 DeviceMinor:276 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-138 DeviceMajor:0 DeviceMinor:138 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/989956e8-1893-4bbe-a4bf-6a2081509080/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:205 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c635d035-053b-49aa-9559-79c31e8e49ff/volumes/kubernetes.io~projected/kube-api-access-mkxzs DeviceMajor:0 DeviceMinor:218 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d5f1b36c251b39df9f18518a01a694a79b6ab1196949d53c53d8b750c0775edd/userdata/shm DeviceMajor:0 DeviceMinor:234 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b86a3f547def9a615d8d32a9c5db979d9fc77c413b9595f6b4fca5be95b62f4d/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/a53078b9-a6a8-423c-b852-34eececa7ab1/volumes/kubernetes.io~projected/kube-api-access-698ds DeviceMajor:0 DeviceMinor:213 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-260 DeviceMajor:0 DeviceMinor:260 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-368 DeviceMajor:0 DeviceMinor:368 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8a972cc8-dce5-4f7c-b5d5-cb24ccf73bad/volumes/kubernetes.io~projected/kube-api-access-jvgj5 DeviceMajor:0 DeviceMinor:64 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/71d80eec-b5e7-41d0-b58d-8a8d9a6c769c/volumes/kubernetes.io~projected/kube-api-access-2tdnl DeviceMajor:0 DeviceMinor:219 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-304 DeviceMajor:0 DeviceMinor:304 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b5961d4b-f3c0-4f17-b1bc-2781cb3dcd6e/volumes/kubernetes.io~projected/kube-api-access-hfdj9 DeviceMajor:0 DeviceMinor:334 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/4071b0c7-c4f3-419a-8d6e-6acc144fc9a4/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:222 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-278 DeviceMajor:0 DeviceMinor:278 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-283 DeviceMajor:0 DeviceMinor:283 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-300 DeviceMajor:0 DeviceMinor:300 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/137914b9aa694c6fd2799975b8439b50e6c4d98b6b27025bd3874923c51ad0c6/userdata/shm DeviceMajor:0 DeviceMinor:323 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/49c67ce3-90fd-478d-9a15-e4267438ff69/volumes/kubernetes.io~projected/kube-api-access-9qd7k DeviceMajor:0 DeviceMinor:354 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5296979a-7927-4ee2-84b9-a93780a2ad22/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:210 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c54ef754-c91e-4bc5-8e83-08523aa09d37/volumes/kubernetes.io~projected/kube-api-access-7h8fj DeviceMajor:0 DeviceMinor:212 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-264 DeviceMajor:0 DeviceMinor:264 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/641f9ec1-c8e2-4774-9d4f-6a7a2da97b6f/volumes/kubernetes.io~projected/kube-api-access-6sv4r DeviceMajor:0 DeviceMinor:227 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/652bedff-87f1-4b2f-917f-571a37dd2f3f/volumes/kubernetes.io~projected/kube-api-access-mfd8g DeviceMajor:0 DeviceMinor:335 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ef6db9261bebd09853a31da06a9228ae19172ae39bb69d4c5d723a543410dd82/userdata/shm DeviceMajor:0 DeviceMinor:91 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-266 DeviceMajor:0 DeviceMinor:266 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d7df54a9e709ebfabcf14a3ad76c25faf0cc254389004e41ba7f1eded00b1a80/userdata/shm DeviceMajor:0 DeviceMinor:294 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-330 DeviceMajor:0 DeviceMinor:330 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/49c67ce3-90fd-478d-9a15-e4267438ff69/volumes/kubernetes.io~secret/signing-key DeviceMajor:0 DeviceMinor:353 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/2ab2aa22-99af-4451-8d33-9e3543c8d9cd/volumes/kubernetes.io~projected/kube-api-access-9v2r9 DeviceMajor:0 DeviceMinor:67 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/532a2188ba13d70f4724205518db4794b66779cf1496ec8df5fcd266cd31d6db/userdata/shm DeviceMajor:0 DeviceMinor:74 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-285 DeviceMajor:0 DeviceMinor:285 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-332 DeviceMajor:0 DeviceMinor:332 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2c31fee02392017ba8b41b846564c28cb85ccd960603212936086d59f108fd8a/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-72 DeviceMajor:0 DeviceMinor:72 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-77 DeviceMajor:0 DeviceMinor:77 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-258 DeviceMajor:0 DeviceMinor:258 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-374 DeviceMajor:0 DeviceMinor:374 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f190a1a0-6f6a-44d0-aeaa-b053cff4bd9c/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:57 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/2d2d7446-0741-4144-9b19-608854f8a04e/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:58 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f190a1a0-6f6a-44d0-aeaa-b053cff4bd9c/volumes/kubernetes.io~projected/kube-api-access-4mxpd DeviceMajor:0 DeviceMinor:60 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/b23027ca-0748-4125-9bc8-9d149a3418e0/volumes/kubernetes.io~projected/kube-api-access-jqxx5 DeviceMajor:0 DeviceMinor:65 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/54fa8fc945d5c93ccfdd34d26d813ccb81a4c450cd4a8ed1ced70ba9319e4dad/userdata/shm DeviceMajor:0 DeviceMinor:238 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/079ed4e4-bdc4-405c-90a1-a22b84fcf948/volumes/kubernetes.io~projected/kube-api-access-ln6mw DeviceMajor:0 DeviceMinor:312 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-313 DeviceMajor:0 DeviceMinor:313 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-252 DeviceMajor:0 DeviceMinor:252 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-362 DeviceMajor:0 DeviceMinor:362 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/run/containers/storage/overlay-containers/906939625d9827aaf66a0763c523390560b22eb094cee8e38b2583671e2bffa2/userdata/shm DeviceMajor:0 DeviceMinor:70 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/989956e8-1893-4bbe-a4bf-6a2081509080/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:204 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-366 DeviceMajor:0 DeviceMinor:366 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6f79e832218106c064e6731373a1925dd3862c0b2ad6865dfe89b297a3ff15c2/userdata/shm DeviceMajor:0 DeviceMinor:82 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d4c08358fd86fbf544eeb14448141bb045e6b6e45eb23b1d549aa8870fe1c4e1/userdata/shm DeviceMajor:0 DeviceMinor:236 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-344 DeviceMajor:0 DeviceMinor:344 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-68 DeviceMajor:0 DeviceMinor:68 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-302 DeviceMajor:0 DeviceMinor:302 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f8f92bfb-5cef-412e-a6c1-4bd793c59fe7/volumes/kubernetes.io~projected/kube-api-access-4nczb DeviceMajor:0 DeviceMinor:66 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-143 DeviceMajor:0 DeviceMinor:143 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5296979a-7927-4ee2-84b9-a93780a2ad22/volumes/kubernetes.io~projected/kube-api-access-fp2lh DeviceMajor:0 DeviceMinor:217 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/79ad3230-163e-447d-9f04-a072479ff957/volumes/kubernetes.io~projected/kube-api-access-vv2vp DeviceMajor:0 DeviceMinor:220 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/a4e1c52c-1eeb-4617-a0c9-d052521360d3/volumes/kubernetes.io~projected/kube-api-access-l9srv DeviceMajor:0 DeviceMinor:293 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e6876edb30d42c86b24074e617f02daea0bf1d967904b4505b4df4ce737d3508/userdata/shm DeviceMajor:0 DeviceMinor:358 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-154 DeviceMajor:0 DeviceMinor:154 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/65c96dfb-06e9-4bea-bb83-f63b4c62d860/volumes/kubernetes.io~projected/kube-api-access-z2tw5 DeviceMajor:0 DeviceMinor:215 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3f2aff1dfc2f332ef98875b32540bc94544ff46bc96652dcfd3100ca3434ad2a/userdata/shm DeviceMajor:0 DeviceMinor:356 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4ff372bf-82aa-4007-b74c-df0c779e5271/volumes/kubernetes.io~projected/kube-api-access-whjbn DeviceMajor:0 DeviceMinor:209 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/aa6a190e-3283-458a-9dfd-2de7cb193fc5/volumes/kubernetes.io~projected/kube-api-access-c4xwz DeviceMajor:0 DeviceMinor:211 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-364 DeviceMajor:0 DeviceMinor:364 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f190a1a0-6f6a-44d0-aeaa-b053cff4bd9c/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:59 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-93 DeviceMajor:0 DeviceMinor:93 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a19be32c77f0d44cd8abc3ba5e326985413ea3d9dcd29dada2ca1c237c96f2c2/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8aec9d2b53599ac849ba2c32c12dd340a8651fcd1e0f2ff829e14e25ffd558ee/userdata/shm DeviceMajor:0 DeviceMinor:228 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/04195c1d-9fb7-479b-8061-f92e3c88aed2/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:55 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/71d80eec-b5e7-41d0-b58d-8a8d9a6c769c/volumes/kubernetes.io~secret/default-certificate DeviceMajor:0 DeviceMinor:206 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/8406bb9f-3a4e-42f3-a822-f754d9b8e7a8/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:221 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/aa6a190e-3283-458a-9dfd-2de7cb193fc5/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:201 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/989956e8-1893-4bbe-a4bf-6a2081509080/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:216 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/dd4e19ce-ef37-44e5-8b57-9dafc590685c/volumes/kubernetes.io~projected/kube-api-access-l8d5d DeviceMajor:0 DeviceMinor:223 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-256 DeviceMajor:0 DeviceMinor:256 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/39746dc1a72c981c5e9698f1f0c72d7c28b892b2793847e6fcf516fb3f1c727f/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5296979a-7927-4ee2-84b9-a93780a2ad22/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:207 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/587893f5-e85a-46a9-829f-545a42c10056/volumes/kubernetes.io~projected/kube-api-access-bjwwn DeviceMajor:0 DeviceMinor:208 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/6b34b3b1-84af-45f3-92ee-1ca720e8b263/volumes/kubernetes.io~projected/kube-api-access-8gw9q DeviceMajor:0 DeviceMinor:225 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-262 DeviceMajor:0 DeviceMinor:262 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/04ff31d0-a870-4527-a0c3-285737359f95/volumes/kubernetes.io~projected/kube-api-access-8jxgn DeviceMajor:0 DeviceMinor:63 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/74ff81a79115f8387b397a72e407635fe9068a1bdfda4852af15c18dcc2b2181/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/989956e8-1893-4bbe-a4bf-6a2081509080/volumes/kubernetes.io~projected/kube-api-access-gfk2t DeviceMajor:0 DeviceMinor:214 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2dbb0c4a7640796e280454d201a82932159022b408fb05d309bb0cdc047866c6/userdata/shm DeviceMajor:0 DeviceMinor:245 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-376 DeviceMajor:0 DeviceMinor:376 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-136 DeviceMajor:0 DeviceMinor:136 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c635d035-053b-49aa-9559-79c31e8e49ff/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:202 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7f2c50e656caef7bde619a98fc6022b5a14ed2ab51fb4b904fe9185e4ef04c86/userdata/shm DeviceMajor:0 DeviceMinor:240 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9bb152b2ed8e5b9f7090432bf0a139a9d586707e7d782cba121f6bb7b126aed4/userdata/shm DeviceMajor:0 DeviceMinor:244 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f176efe0-febd-4652-94fc-91548ee3ab93/volumes/kubernetes.io~projected/kube-api-access-p9pvq DeviceMajor:0 DeviceMinor:328 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-360 DeviceMajor:0 DeviceMinor:360 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/71d80eec-b5e7-41d0-b58d-8a8d9a6c769c/volumes/kubernetes.io~secret/stats-auth DeviceMajor:0 DeviceMinor:197 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a850aad9d1fa77621f0945112e249020d542379254765dba31625cadfce0b375/userdata/shm DeviceMajor:0 DeviceMinor:232 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-250 DeviceMajor:0 DeviceMinor:250 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-298 DeviceMajor:0 DeviceMinor:298 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ffc8236d320661b5d814a75ef36e7f088f46bf70179419b19ecb9da171ab1ab4/userdata/shm DeviceMajor:0 DeviceMinor:348 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/65c96dfb-06e9-4bea-bb83-f63b4c62d860/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:203 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f2028b5b83ac2c14a3e4c4b56a439b3dcf82bf043749a960aa5867a0992c7837/userdata/shm DeviceMajor:0 DeviceMinor:296 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/93dbae94-f92d-4157-a89b-d3c984a56013/volumes/kubernetes.io~projected/kube-api-access-wf78m DeviceMajor:0 DeviceMinor:61 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/0e1c2279-b6d4-44e0-84e6-94eb1206d2a0/volumes/kubernetes.io~projected/kube-api-access-7fmsg DeviceMajor:0 DeviceMinor:62 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5296979a-7927-4ee2-84b9-a93780a2ad22/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:45 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6d1c35a93c18cc546622631e919464a48df503c2a5b066a732af5226f05dce9c/userdata/shm DeviceMajor:0 DeviceMinor:242 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-310 DeviceMajor:0 DeviceMinor:310 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:137914b9aa694c6 MacAddress:4a:ba:79:04:56:7f Speed:10000 Mtu:8901} {Name:2dbb0c4a7640796 MacAddress:be:d7:03:6c:2a:00 Speed:10000 Mtu:8901} {Name:3f2aff1dfc2f332 MacAddress:9a:ee:4c:dc:2a:ab Speed:10000 Mtu:8901} {Name:54fa8fc945d5c93 MacAddress:5a:41:c6:ec:b1:6a Speed:10000 Mtu:8901} {Name:6d1c35a93c18cc5 MacAddress:1a:1f:11:8a:e5:77 Speed:10000 Mtu:8901} {Name:7f2c50e656caef7 MacAddress:2e:39:db:0f:12:be Speed:10000 Mtu:8901} {Name:8aec9d2b53599ac MacAddress:8e:a8:d2:73:e8:2d Speed:10000 Mtu:8901} {Name:9bb152b2ed8e5b9 MacAddress:96:d2:e3:62:36:fe Speed:10000 Mtu:8901} {Name:a850aad9d1fa776 MacAddress:fa:cb:cf:7d:0c:e7 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:c5:a5:39:73 Speed:0 Mtu:9001} {Name:br-int MacAddress:7a:c9:63:0b:9e:fb Speed:0 Mtu:8901} {Name:ccdd3c46b100f9e MacAddress:92:91:d5:8b:ab:6e Speed:10000 Mtu:8901} {Name:d4c08358fd86fbf MacAddress:5e:67:b6:e9:c3:d4 Speed:10000 Mtu:8901} {Name:d5f1b36c251b39d MacAddress:86:93:7f:62:55:ae Speed:10000 Mtu:8901} {Name:d7df54a9e709ebf MacAddress:52:e3:6f:aa:34:f0 Speed:10000 Mtu:8901} {Name:e6876edb30d42c8 MacAddress:76:0a:ea:21:51:f2 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:c5:a5:39:73 Speed:0 Mtu:9001} {Name:f2028b5b83ac2c1 MacAddress:4e:6f:13:61:8c:df Speed:10000 Mtu:8901} {Name:ffc8236d320661b MacAddress:52:b2:88:3b:7c:2a Speed:10000 Mtu:8901} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:84:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:c6:bb:2c:00:80:2b Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164337152 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 29 22:33:39.282572 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.279096 5057 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 29 22:33:39.282572 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.279139 5057 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 29 22:33:39.282572 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.279502 5057 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 29 22:33:39.282572 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.279530 5057 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-140-74.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 29 22:33:39.282572 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.279735 5057 topology_manager.go:143] "Creating topology manager with none policy" Oct 29 22:33:39.282572 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.279748 5057 container_manager_linux.go:311] "Creating device plugin manager" Oct 29 22:33:39.282572 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.279761 5057 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 29 22:33:39.282572 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.279788 5057 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 29 22:33:39.282572 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.279815 5057 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 29 22:33:39.282572 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.280029 5057 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 29 22:33:39.282572 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.280167 5057 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 29 22:33:39.282572 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.280212 5057 kubelet.go:493] "Attempting to sync node with API server" Oct 29 22:33:39.282572 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.280227 5057 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 29 22:33:39.282572 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.280240 5057 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 29 22:33:39.282572 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.280253 5057 kubelet.go:405] "Adding apiserver pod source" Oct 29 22:33:39.282572 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.280264 5057 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 29 22:33:39.282572 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.281161 5057 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 29 22:33:39.282572 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.281176 5057 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 29 22:33:39.282572 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.281693 5057 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 29 22:33:39.284011 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.283997 5057 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 29 22:33:39.284011 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.284010 5057 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 29 22:33:39.284130 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.284120 5057 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Oct 29 22:33:39.284303 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.284292 5057 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 29 22:33:39.284357 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.284321 5057 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 29 22:33:39.284430 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.284421 5057 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 29 22:33:39.284470 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.284435 5057 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 29 22:33:39.284470 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.284441 5057 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 29 22:33:39.284470 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.284446 5057 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 29 22:33:39.284470 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.284451 5057 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 29 22:33:39.284470 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.284467 5057 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 29 22:33:39.284470 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.284473 5057 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 29 22:33:39.284659 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.284478 5057 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 29 22:33:39.284659 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.284485 5057 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 29 22:33:39.284659 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.284491 5057 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 29 22:33:39.284659 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.284502 5057 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 29 22:33:39.284659 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.284512 5057 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 29 22:33:39.284659 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.284537 5057 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 29 22:33:39.284659 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.284543 5057 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 29 22:33:39.287736 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.287721 5057 server.go:1263] "Started kubelet" Oct 29 22:33:39.288199 ip-10-0-140-74 systemd[1]: Started Kubernetes Kubelet. Oct 29 22:33:39.288997 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.288604 5057 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 29 22:33:39.288997 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.288670 5057 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 29 22:33:39.288997 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.288945 5057 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 29 22:33:39.289197 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.287818 5057 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 29 22:33:39.291946 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.290633 5057 server.go:317] "Adding debug handlers to kubelet server" Oct 29 22:33:39.292102 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.292082 5057 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 29 22:33:39.292445 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.292191 5057 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 29 22:33:39.292445 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.292204 5057 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 29 22:33:39.292445 ip-10-0-140-74 kubenswrapper[5057]: E1029 22:33:39.292252 5057 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-74.ec2.internal\" not found" Oct 29 22:33:39.292445 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.292353 5057 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 29 22:33:39.297581 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.293654 5057 factory.go:55] Registering systemd factory Oct 29 22:33:39.297581 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.293673 5057 factory.go:223] Registration of the systemd container factory successfully Oct 29 22:33:39.297581 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.295738 5057 factory.go:153] Registering CRI-O factory Oct 29 22:33:39.297581 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.295754 5057 factory.go:223] Registration of the crio container factory successfully Oct 29 22:33:39.297581 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.295806 5057 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 29 22:33:39.297581 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.296058 5057 factory.go:103] Registering Raw factory Oct 29 22:33:39.297581 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.296109 5057 manager.go:1199] Started watching for new ooms in manager Oct 29 22:33:39.297581 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.296564 5057 manager.go:319] Starting recovery of all containers Oct 29 22:33:39.297581 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.297421 5057 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 29 22:33:39.301382 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.299946 5057 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f190a1a0-6f6a-44d0-aeaa-b053cff4bd9c" volumeName="kubernetes.io/projected/f190a1a0-6f6a-44d0-aeaa-b053cff4bd9c-kube-api-access-4mxpd" seLinuxMountContext="" Oct 29 22:33:39.301382 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.300001 5057 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="04195c1d-9fb7-479b-8061-f92e3c88aed2" volumeName="kubernetes.io/projected/04195c1d-9fb7-479b-8061-f92e3c88aed2-kube-api-access-tlxfr" seLinuxMountContext="" Oct 29 22:33:39.301382 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.300016 5057 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="079ed4e4-bdc4-405c-90a1-a22b84fcf948" volumeName="kubernetes.io/empty-dir/079ed4e4-bdc4-405c-90a1-a22b84fcf948-data-volume" seLinuxMountContext="" Oct 29 22:33:39.301382 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.300047 5057 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="079ed4e4-bdc4-405c-90a1-a22b84fcf948" volumeName="kubernetes.io/projected/079ed4e4-bdc4-405c-90a1-a22b84fcf948-kube-api-access-ln6mw" seLinuxMountContext="" Oct 29 22:33:39.301382 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.300061 5057 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6b34b3b1-84af-45f3-92ee-1ca720e8b263" volumeName="kubernetes.io/empty-dir/6b34b3b1-84af-45f3-92ee-1ca720e8b263-tmp-dir" seLinuxMountContext="" Oct 29 22:33:39.301382 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.300072 5057 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="989956e8-1893-4bbe-a4bf-6a2081509080" volumeName="kubernetes.io/empty-dir/989956e8-1893-4bbe-a4bf-6a2081509080-ca-trust-extracted" seLinuxMountContext="" Oct 29 22:33:39.301382 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.300084 5057 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="989956e8-1893-4bbe-a4bf-6a2081509080" volumeName="kubernetes.io/projected/989956e8-1893-4bbe-a4bf-6a2081509080-kube-api-access-gfk2t" seLinuxMountContext="" Oct 29 22:33:39.301382 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.300063 5057 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-28 22:28:00 +0000 UTC" deadline="2028-04-28 11:58:55.343581425 +0000 UTC" Oct 29 22:33:39.301382 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.300096 5057 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f190a1a0-6f6a-44d0-aeaa-b053cff4bd9c" volumeName="kubernetes.io/empty-dir/f190a1a0-6f6a-44d0-aeaa-b053cff4bd9c-etc-tuned" seLinuxMountContext="" Oct 29 22:33:39.301382 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.300102 5057 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13117h25m16.0434827s" Oct 29 22:33:39.301382 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.300113 5057 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2ab2aa22-99af-4451-8d33-9e3543c8d9cd" volumeName="kubernetes.io/configmap/2ab2aa22-99af-4451-8d33-9e3543c8d9cd-cni-binary-copy" seLinuxMountContext="" Oct 29 22:33:39.301382 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.300126 5057 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2d2d7446-0741-4144-9b19-608854f8a04e" volumeName="kubernetes.io/secret/2d2d7446-0741-4144-9b19-608854f8a04e-agent-certs" seLinuxMountContext="" Oct 29 22:33:39.301382 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.300141 5057 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="65c96dfb-06e9-4bea-bb83-f63b4c62d860" volumeName="kubernetes.io/empty-dir/65c96dfb-06e9-4bea-bb83-f63b4c62d860-snapshots" seLinuxMountContext="" Oct 29 22:33:39.301382 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.300154 5057 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6f50ee57-a710-4325-bd45-4f861452ede4" volumeName="kubernetes.io/secret/6f50ee57-a710-4325-bd45-4f861452ede4-original-pull-secret" seLinuxMountContext="" Oct 29 22:33:39.301382 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.300165 5057 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="989956e8-1893-4bbe-a4bf-6a2081509080" volumeName="kubernetes.io/configmap/989956e8-1893-4bbe-a4bf-6a2081509080-trusted-ca" seLinuxMountContext="" Oct 29 22:33:39.301382 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.300177 5057 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aa6a190e-3283-458a-9dfd-2de7cb193fc5" volumeName="kubernetes.io/secret/aa6a190e-3283-458a-9dfd-2de7cb193fc5-serving-cert" seLinuxMountContext="" Oct 29 22:33:39.301382 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.300193 5057 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c54ef754-c91e-4bc5-8e83-08523aa09d37" volumeName="kubernetes.io/projected/c54ef754-c91e-4bc5-8e83-08523aa09d37-kube-api-access-7h8fj" seLinuxMountContext="" Oct 29 22:33:39.301382 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.300221 5057 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="641f9ec1-c8e2-4774-9d4f-6a7a2da97b6f" volumeName="kubernetes.io/configmap/641f9ec1-c8e2-4774-9d4f-6a7a2da97b6f-telemetry-config" seLinuxMountContext="" Oct 29 22:33:39.301382 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.300235 5057 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="989956e8-1893-4bbe-a4bf-6a2081509080" volumeName="kubernetes.io/configmap/989956e8-1893-4bbe-a4bf-6a2081509080-registry-certificates" seLinuxMountContext="" Oct 29 22:33:39.301382 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.300248 5057 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f176efe0-febd-4652-94fc-91548ee3ab93" volumeName="kubernetes.io/empty-dir/f176efe0-febd-4652-94fc-91548ee3ab93-archives-path" seLinuxMountContext="" Oct 29 22:33:39.301382 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.300260 5057 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f8f92bfb-5cef-412e-a6c1-4bd793c59fe7" volumeName="kubernetes.io/projected/f8f92bfb-5cef-412e-a6c1-4bd793c59fe7-kube-api-access-4nczb" seLinuxMountContext="" Oct 29 22:33:39.301382 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.300272 5057 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="04195c1d-9fb7-479b-8061-f92e3c88aed2" volumeName="kubernetes.io/configmap/04195c1d-9fb7-479b-8061-f92e3c88aed2-env-overrides" seLinuxMountContext="" Oct 29 22:33:39.301382 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.300283 5057 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="04ff31d0-a870-4527-a0c3-285737359f95" volumeName="kubernetes.io/projected/04ff31d0-a870-4527-a0c3-285737359f95-kube-api-access-8jxgn" seLinuxMountContext="" Oct 29 22:33:39.301382 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.300295 5057 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5296979a-7927-4ee2-84b9-a93780a2ad22" volumeName="kubernetes.io/empty-dir/5296979a-7927-4ee2-84b9-a93780a2ad22-ca-trust-extracted" seLinuxMountContext="" Oct 29 22:33:39.301382 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.300309 5057 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="71d80eec-b5e7-41d0-b58d-8a8d9a6c769c" volumeName="kubernetes.io/projected/71d80eec-b5e7-41d0-b58d-8a8d9a6c769c-kube-api-access-2tdnl" seLinuxMountContext="" Oct 29 22:33:39.301382 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.300322 5057 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="71d80eec-b5e7-41d0-b58d-8a8d9a6c769c" volumeName="kubernetes.io/secret/71d80eec-b5e7-41d0-b58d-8a8d9a6c769c-default-certificate" seLinuxMountContext="" Oct 29 22:33:39.301382 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.300353 5057 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8406bb9f-3a4e-42f3-a822-f754d9b8e7a8" volumeName="kubernetes.io/projected/8406bb9f-3a4e-42f3-a822-f754d9b8e7a8-kube-api-access-xvmbg" seLinuxMountContext="" Oct 29 22:33:39.301382 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.300367 5057 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8a972cc8-dce5-4f7c-b5d5-cb24ccf73bad" volumeName="kubernetes.io/configmap/8a972cc8-dce5-4f7c-b5d5-cb24ccf73bad-cni-binary-copy" seLinuxMountContext="" Oct 29 22:33:39.301382 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.300380 5057 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="989956e8-1893-4bbe-a4bf-6a2081509080" volumeName="kubernetes.io/secret/989956e8-1893-4bbe-a4bf-6a2081509080-installation-pull-secrets" seLinuxMountContext="" Oct 29 22:33:39.301382 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.300392 5057 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="04195c1d-9fb7-479b-8061-f92e3c88aed2" volumeName="kubernetes.io/configmap/04195c1d-9fb7-479b-8061-f92e3c88aed2-ovnkube-script-lib" seLinuxMountContext="" Oct 29 22:33:39.301382 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.300410 5057 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0e1c2279-b6d4-44e0-84e6-94eb1206d2a0" volumeName="kubernetes.io/projected/0e1c2279-b6d4-44e0-84e6-94eb1206d2a0-kube-api-access-7fmsg" seLinuxMountContext="" Oct 29 22:33:39.301382 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.300422 5057 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4071b0c7-c4f3-419a-8d6e-6acc144fc9a4" volumeName="kubernetes.io/secret/4071b0c7-c4f3-419a-8d6e-6acc144fc9a4-serving-cert" seLinuxMountContext="" Oct 29 22:33:39.301382 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.300434 5057 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="65c96dfb-06e9-4bea-bb83-f63b4c62d860" volumeName="kubernetes.io/empty-dir/65c96dfb-06e9-4bea-bb83-f63b4c62d860-tmp" seLinuxMountContext="" Oct 29 22:33:39.301382 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.300453 5057 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="93dbae94-f92d-4157-a89b-d3c984a56013" volumeName="kubernetes.io/projected/93dbae94-f92d-4157-a89b-d3c984a56013-kube-api-access-wf78m" seLinuxMountContext="" Oct 29 22:33:39.301382 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.300471 5057 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a4e1c52c-1eeb-4617-a0c9-d052521360d3" volumeName="kubernetes.io/projected/a4e1c52c-1eeb-4617-a0c9-d052521360d3-kube-api-access-l9srv" seLinuxMountContext="" Oct 29 22:33:39.301382 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.300484 5057 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aa6a190e-3283-458a-9dfd-2de7cb193fc5" volumeName="kubernetes.io/configmap/aa6a190e-3283-458a-9dfd-2de7cb193fc5-trusted-ca" seLinuxMountContext="" Oct 29 22:33:39.301382 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.300497 5057 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c635d035-053b-49aa-9559-79c31e8e49ff" volumeName="kubernetes.io/configmap/c635d035-053b-49aa-9559-79c31e8e49ff-config" seLinuxMountContext="" Oct 29 22:33:39.301382 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.300509 5057 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4071b0c7-c4f3-419a-8d6e-6acc144fc9a4" volumeName="kubernetes.io/configmap/4071b0c7-c4f3-419a-8d6e-6acc144fc9a4-config" seLinuxMountContext="" Oct 29 22:33:39.301382 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.300521 5057 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5296979a-7927-4ee2-84b9-a93780a2ad22" volumeName="kubernetes.io/projected/5296979a-7927-4ee2-84b9-a93780a2ad22-bound-sa-token" seLinuxMountContext="" Oct 29 22:33:39.301382 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.300535 5057 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8a972cc8-dce5-4f7c-b5d5-cb24ccf73bad" volumeName="kubernetes.io/configmap/8a972cc8-dce5-4f7c-b5d5-cb24ccf73bad-whereabouts-flatfile-configmap" seLinuxMountContext="" Oct 29 22:33:39.301382 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.300565 5057 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="04ff31d0-a870-4527-a0c3-285737359f95" volumeName="kubernetes.io/configmap/04ff31d0-a870-4527-a0c3-285737359f95-iptables-alerter-script" seLinuxMountContext="" Oct 29 22:33:39.301382 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.300578 5057 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5296979a-7927-4ee2-84b9-a93780a2ad22" volumeName="kubernetes.io/configmap/5296979a-7927-4ee2-84b9-a93780a2ad22-registry-certificates" seLinuxMountContext="" Oct 29 22:33:39.301382 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.300592 5057 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5296979a-7927-4ee2-84b9-a93780a2ad22" volumeName="kubernetes.io/projected/5296979a-7927-4ee2-84b9-a93780a2ad22-kube-api-access-fp2lh" seLinuxMountContext="" Oct 29 22:33:39.301382 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.300608 5057 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5296979a-7927-4ee2-84b9-a93780a2ad22" volumeName="kubernetes.io/secret/5296979a-7927-4ee2-84b9-a93780a2ad22-image-registry-private-configuration" seLinuxMountContext="" Oct 29 22:33:39.301382 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.300622 5057 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="65c96dfb-06e9-4bea-bb83-f63b4c62d860" volumeName="kubernetes.io/configmap/65c96dfb-06e9-4bea-bb83-f63b4c62d860-service-ca-bundle" seLinuxMountContext="" Oct 29 22:33:39.301382 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.300635 5057 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6b34b3b1-84af-45f3-92ee-1ca720e8b263" volumeName="kubernetes.io/configmap/6b34b3b1-84af-45f3-92ee-1ca720e8b263-config-volume" seLinuxMountContext="" Oct 29 22:33:39.301382 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.300647 5057 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="79ad3230-163e-447d-9f04-a072479ff957" volumeName="kubernetes.io/projected/79ad3230-163e-447d-9f04-a072479ff957-kube-api-access-vv2vp" seLinuxMountContext="" Oct 29 22:33:39.301382 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.300661 5057 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="989956e8-1893-4bbe-a4bf-6a2081509080" volumeName="kubernetes.io/projected/989956e8-1893-4bbe-a4bf-6a2081509080-bound-sa-token" seLinuxMountContext="" Oct 29 22:33:39.301382 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.300674 5057 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="49c67ce3-90fd-478d-9a15-e4267438ff69" volumeName="kubernetes.io/secret/49c67ce3-90fd-478d-9a15-e4267438ff69-signing-key" seLinuxMountContext="" Oct 29 22:33:39.301382 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.300688 5057 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5296979a-7927-4ee2-84b9-a93780a2ad22" volumeName="kubernetes.io/secret/5296979a-7927-4ee2-84b9-a93780a2ad22-installation-pull-secrets" seLinuxMountContext="" Oct 29 22:33:39.301382 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.300701 5057 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8a972cc8-dce5-4f7c-b5d5-cb24ccf73bad" volumeName="kubernetes.io/projected/8a972cc8-dce5-4f7c-b5d5-cb24ccf73bad-kube-api-access-jvgj5" seLinuxMountContext="" Oct 29 22:33:39.301382 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.300714 5057 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c635d035-053b-49aa-9559-79c31e8e49ff" volumeName="kubernetes.io/projected/c635d035-053b-49aa-9559-79c31e8e49ff-kube-api-access-mkxzs" seLinuxMountContext="" Oct 29 22:33:39.301382 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.300728 5057 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f190a1a0-6f6a-44d0-aeaa-b053cff4bd9c" volumeName="kubernetes.io/empty-dir/f190a1a0-6f6a-44d0-aeaa-b053cff4bd9c-tmp" seLinuxMountContext="" Oct 29 22:33:39.301382 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.300740 5057 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="04195c1d-9fb7-479b-8061-f92e3c88aed2" volumeName="kubernetes.io/configmap/04195c1d-9fb7-479b-8061-f92e3c88aed2-ovnkube-config" seLinuxMountContext="" Oct 29 22:33:39.301382 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.300756 5057 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5296979a-7927-4ee2-84b9-a93780a2ad22" volumeName="kubernetes.io/configmap/5296979a-7927-4ee2-84b9-a93780a2ad22-trusted-ca" seLinuxMountContext="" Oct 29 22:33:39.301382 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.300772 5057 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2ab2aa22-99af-4451-8d33-9e3543c8d9cd" volumeName="kubernetes.io/projected/2ab2aa22-99af-4451-8d33-9e3543c8d9cd-kube-api-access-9v2r9" seLinuxMountContext="" Oct 29 22:33:39.301382 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.300786 5057 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2accd775-2173-4bfe-a413-7d243bd0820f" volumeName="kubernetes.io/configmap/2accd775-2173-4bfe-a413-7d243bd0820f-nginx-conf" seLinuxMountContext="" Oct 29 22:33:39.301382 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.300799 5057 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4071b0c7-c4f3-419a-8d6e-6acc144fc9a4" volumeName="kubernetes.io/projected/4071b0c7-c4f3-419a-8d6e-6acc144fc9a4-kube-api-access-n8hb6" seLinuxMountContext="" Oct 29 22:33:39.301382 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.300820 5057 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="652bedff-87f1-4b2f-917f-571a37dd2f3f" volumeName="kubernetes.io/projected/652bedff-87f1-4b2f-917f-571a37dd2f3f-kube-api-access-mfd8g" seLinuxMountContext="" Oct 29 22:33:39.301382 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.300831 5057 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="65c96dfb-06e9-4bea-bb83-f63b4c62d860" volumeName="kubernetes.io/projected/65c96dfb-06e9-4bea-bb83-f63b4c62d860-kube-api-access-z2tw5" seLinuxMountContext="" Oct 29 22:33:39.301382 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.300844 5057 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8a972cc8-dce5-4f7c-b5d5-cb24ccf73bad" volumeName="kubernetes.io/configmap/8a972cc8-dce5-4f7c-b5d5-cb24ccf73bad-cni-sysctl-allowlist" seLinuxMountContext="" Oct 29 22:33:39.301382 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.300856 5057 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="989956e8-1893-4bbe-a4bf-6a2081509080" volumeName="kubernetes.io/secret/989956e8-1893-4bbe-a4bf-6a2081509080-image-registry-private-configuration" seLinuxMountContext="" Oct 29 22:33:39.301382 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.300867 5057 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2d2d7446-0741-4144-9b19-608854f8a04e" volumeName="kubernetes.io/configmap/2d2d7446-0741-4144-9b19-608854f8a04e-konnectivity-ca" seLinuxMountContext="" Oct 29 22:33:39.301382 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.300878 5057 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="587893f5-e85a-46a9-829f-545a42c10056" volumeName="kubernetes.io/projected/587893f5-e85a-46a9-829f-545a42c10056-kube-api-access-bjwwn" seLinuxMountContext="" Oct 29 22:33:39.301382 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.300917 5057 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dd4e19ce-ef37-44e5-8b57-9dafc590685c" volumeName="kubernetes.io/projected/dd4e19ce-ef37-44e5-8b57-9dafc590685c-kube-api-access-l8d5d" seLinuxMountContext="" Oct 29 22:33:39.301382 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.300934 5057 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4ff372bf-82aa-4007-b74c-df0c779e5271" volumeName="kubernetes.io/projected/4ff372bf-82aa-4007-b74c-df0c779e5271-kube-api-access-whjbn" seLinuxMountContext="" Oct 29 22:33:39.301382 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.300946 5057 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8406bb9f-3a4e-42f3-a822-f754d9b8e7a8" volumeName="kubernetes.io/secret/8406bb9f-3a4e-42f3-a822-f754d9b8e7a8-cert" seLinuxMountContext="" Oct 29 22:33:39.301382 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.300958 5057 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a53078b9-a6a8-423c-b852-34eececa7ab1" volumeName="kubernetes.io/configmap/a53078b9-a6a8-423c-b852-34eececa7ab1-samples-operator-config" seLinuxMountContext="" Oct 29 22:33:39.301382 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.300970 5057 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a53078b9-a6a8-423c-b852-34eececa7ab1" volumeName="kubernetes.io/projected/a53078b9-a6a8-423c-b852-34eececa7ab1-kube-api-access-698ds" seLinuxMountContext="" Oct 29 22:33:39.301382 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.300982 5057 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f176efe0-febd-4652-94fc-91548ee3ab93" volumeName="kubernetes.io/projected/f176efe0-febd-4652-94fc-91548ee3ab93-kube-api-access-p9pvq" seLinuxMountContext="" Oct 29 22:33:39.301382 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.300993 5057 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="65c96dfb-06e9-4bea-bb83-f63b4c62d860" volumeName="kubernetes.io/configmap/65c96dfb-06e9-4bea-bb83-f63b4c62d860-trusted-ca-bundle" seLinuxMountContext="" Oct 29 22:33:39.301382 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.301005 5057 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="71d80eec-b5e7-41d0-b58d-8a8d9a6c769c" volumeName="kubernetes.io/secret/71d80eec-b5e7-41d0-b58d-8a8d9a6c769c-stats-auth" seLinuxMountContext="" Oct 29 22:33:39.301382 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.301017 5057 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f176efe0-febd-4652-94fc-91548ee3ab93" volumeName="kubernetes.io/configmap/f176efe0-febd-4652-94fc-91548ee3ab93-service-ca-bundle" seLinuxMountContext="" Oct 29 22:33:39.301382 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.301028 5057 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="04195c1d-9fb7-479b-8061-f92e3c88aed2" volumeName="kubernetes.io/secret/04195c1d-9fb7-479b-8061-f92e3c88aed2-ovn-node-metrics-cert" seLinuxMountContext="" Oct 29 22:33:39.301382 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.301039 5057 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="65c96dfb-06e9-4bea-bb83-f63b4c62d860" volumeName="kubernetes.io/secret/65c96dfb-06e9-4bea-bb83-f63b4c62d860-serving-cert" seLinuxMountContext="" Oct 29 22:33:39.301382 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.301052 5057 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="93dbae94-f92d-4157-a89b-d3c984a56013" volumeName="kubernetes.io/configmap/93dbae94-f92d-4157-a89b-d3c984a56013-serviceca" seLinuxMountContext="" Oct 29 22:33:39.301382 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.301063 5057 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2ab2aa22-99af-4451-8d33-9e3543c8d9cd" volumeName="kubernetes.io/configmap/2ab2aa22-99af-4451-8d33-9e3543c8d9cd-multus-daemon-config" seLinuxMountContext="" Oct 29 22:33:39.301382 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.301075 5057 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="641f9ec1-c8e2-4774-9d4f-6a7a2da97b6f" volumeName="kubernetes.io/projected/641f9ec1-c8e2-4774-9d4f-6a7a2da97b6f-kube-api-access-6sv4r" seLinuxMountContext="" Oct 29 22:33:39.301382 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.301088 5057 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6b34b3b1-84af-45f3-92ee-1ca720e8b263" volumeName="kubernetes.io/projected/6b34b3b1-84af-45f3-92ee-1ca720e8b263-kube-api-access-8gw9q" seLinuxMountContext="" Oct 29 22:33:39.301382 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.301100 5057 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aa6a190e-3283-458a-9dfd-2de7cb193fc5" volumeName="kubernetes.io/configmap/aa6a190e-3283-458a-9dfd-2de7cb193fc5-config" seLinuxMountContext="" Oct 29 22:33:39.301382 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.301112 5057 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aa6a190e-3283-458a-9dfd-2de7cb193fc5" volumeName="kubernetes.io/projected/aa6a190e-3283-458a-9dfd-2de7cb193fc5-kube-api-access-c4xwz" seLinuxMountContext="" Oct 29 22:33:39.301382 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.301124 5057 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b23027ca-0748-4125-9bc8-9d149a3418e0" volumeName="kubernetes.io/projected/b23027ca-0748-4125-9bc8-9d149a3418e0-kube-api-access-jqxx5" seLinuxMountContext="" Oct 29 22:33:39.301382 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.301136 5057 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c635d035-053b-49aa-9559-79c31e8e49ff" volumeName="kubernetes.io/secret/c635d035-053b-49aa-9559-79c31e8e49ff-serving-cert" seLinuxMountContext="" Oct 29 22:33:39.301382 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.301149 5057 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f176efe0-febd-4652-94fc-91548ee3ab93" volumeName="kubernetes.io/configmap/f176efe0-febd-4652-94fc-91548ee3ab93-trusted-ca-bundle" seLinuxMountContext="" Oct 29 22:33:39.301382 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.301162 5057 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0e1c2279-b6d4-44e0-84e6-94eb1206d2a0" volumeName="kubernetes.io/empty-dir/0e1c2279-b6d4-44e0-84e6-94eb1206d2a0-tmp-dir" seLinuxMountContext="" Oct 29 22:33:39.301382 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.301173 5057 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="49c67ce3-90fd-478d-9a15-e4267438ff69" volumeName="kubernetes.io/configmap/49c67ce3-90fd-478d-9a15-e4267438ff69-signing-cabundle" seLinuxMountContext="" Oct 29 22:33:39.301382 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.301184 5057 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="49c67ce3-90fd-478d-9a15-e4267438ff69" volumeName="kubernetes.io/projected/49c67ce3-90fd-478d-9a15-e4267438ff69-kube-api-access-9qd7k" seLinuxMountContext="" Oct 29 22:33:39.301382 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.301199 5057 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b5961d4b-f3c0-4f17-b1bc-2781cb3dcd6e" volumeName="kubernetes.io/projected/b5961d4b-f3c0-4f17-b1bc-2781cb3dcd6e-kube-api-access-hfdj9" seLinuxMountContext="" Oct 29 22:33:39.301382 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.301210 5057 reconstruct.go:97] "Volume reconstruction finished" Oct 29 22:33:39.301382 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.301219 5057 reconciler.go:29] "Reconciler: start to sync state" Oct 29 22:33:39.318281 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.318249 5057 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 29 22:33:39.319464 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.319441 5057 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 29 22:33:39.319464 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.319461 5057 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 29 22:33:39.319640 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.319479 5057 kubelet.go:2521] "Starting kubelet main sync loop" Oct 29 22:33:39.319640 ip-10-0-140-74 kubenswrapper[5057]: E1029 22:33:39.319511 5057 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Oct 29 22:33:39.324173 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.323711 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-vj6md_aa6a190e-3283-458a-9dfd-2de7cb193fc5/console-operator/1.log" Oct 29 22:33:39.325062 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.325046 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-vj6md_aa6a190e-3283-458a-9dfd-2de7cb193fc5/console-operator/1.log" Oct 29 22:33:39.341106 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.341082 5057 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Oct 29 22:33:39.342387 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.342367 5057 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 29 22:33:39.342876 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.342854 5057 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 29 22:33:39.347723 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.347699 5057 generic.go:356] "Generic (PLEG): container finished" podID="4c81f7639feb5d07aad0278bf28e49a4" containerID="cc1013fca8c05a300d6b735484f7ea18f02fa49a4188949ccb6a61dadee5670d" exitCode=0 Oct 29 22:33:39.352724 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.352664 5057 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 29 22:33:39.353389 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.353368 5057 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 29 22:33:39.378561 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.378539 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-vj6md_aa6a190e-3283-458a-9dfd-2de7cb193fc5/console-operator/1.log" Oct 29 22:33:39.378644 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.378573 5057 generic.go:356] "Generic (PLEG): container finished" podID="aa6a190e-3283-458a-9dfd-2de7cb193fc5" containerID="e5ab727c17d3a45185a300b3b2a3688775b919a83c6e93f3a510f87ee56a0fb7" exitCode=255 Oct 29 22:33:39.382212 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.382188 5057 generic.go:356] "Generic (PLEG): container finished" podID="8a972cc8-dce5-4f7c-b5d5-cb24ccf73bad" containerID="fca31e6edd11c11236b1a8598967db8b22f22b67bd81bf0a2579bccd8f0785f8" exitCode=0 Oct 29 22:33:39.382212 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.382209 5057 generic.go:356] "Generic (PLEG): container finished" podID="8a972cc8-dce5-4f7c-b5d5-cb24ccf73bad" containerID="8f8ee81ce02909c8bc39a0050b5296d599e92f270670edf383497f06f78ee26d" exitCode=0 Oct 29 22:33:39.382308 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.382217 5057 generic.go:356] "Generic (PLEG): container finished" podID="8a972cc8-dce5-4f7c-b5d5-cb24ccf73bad" containerID="63808e2ca3fe89606c5aec28d215f23b4bd9e746d7bdb945fa9c7792d2f18cbc" exitCode=0 Oct 29 22:33:39.382308 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.382225 5057 generic.go:356] "Generic (PLEG): container finished" podID="8a972cc8-dce5-4f7c-b5d5-cb24ccf73bad" containerID="509bf01b64666e56e89300f5782fa80e720d319c48f512372f2ff6dc7aae771d" exitCode=0 Oct 29 22:33:39.382308 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.382231 5057 generic.go:356] "Generic (PLEG): container finished" podID="8a972cc8-dce5-4f7c-b5d5-cb24ccf73bad" containerID="c9362cd1dc9fc8be11035beb2b37ad66e41e0d36082b49d8e4b5a8db4ed20470" exitCode=0 Oct 29 22:33:39.382308 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.382236 5057 generic.go:356] "Generic (PLEG): container finished" podID="8a972cc8-dce5-4f7c-b5d5-cb24ccf73bad" containerID="93cfdb05efe755447ad1112baf02fe611d673a0725f410122248e2005c6ec90e" exitCode=0 Oct 29 22:33:39.399272 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.399257 5057 manager.go:324] Recovery completed Oct 29 22:33:39.418392 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.418370 5057 cpu_manager.go:226] "Starting" policy="none" Oct 29 22:33:39.418392 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.418383 5057 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 29 22:33:39.418538 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.418398 5057 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 29 22:33:39.418538 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.418514 5057 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Oct 29 22:33:39.418964 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.418524 5057 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Oct 29 22:33:39.419010 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.418966 5057 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Oct 29 22:33:39.419010 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.418972 5057 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Oct 29 22:33:39.419010 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.418978 5057 policy_none.go:50] "Start" Oct 29 22:33:39.419010 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.418992 5057 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 29 22:33:39.419010 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.419004 5057 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 29 22:33:39.419140 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.419130 5057 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Oct 29 22:33:39.419169 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.419140 5057 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Oct 29 22:33:39.419169 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.419146 5057 policy_none.go:44] "Start" Oct 29 22:33:39.421491 ip-10-0-140-74 kubenswrapper[5057]: E1029 22:33:39.421462 5057 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet" Oct 29 22:33:39.425213 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.425186 5057 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 29 22:33:39.427756 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.427741 5057 manager.go:341] "Starting Device Plugin manager" Oct 29 22:33:39.427830 ip-10-0-140-74 kubenswrapper[5057]: E1029 22:33:39.427767 5057 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 29 22:33:39.427830 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.427776 5057 server.go:87] "Starting device plugin registration server" Oct 29 22:33:39.428025 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.428011 5057 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 29 22:33:39.428076 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.428026 5057 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 29 22:33:39.428133 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.428119 5057 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 29 22:33:39.428240 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.428216 5057 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 29 22:33:39.428291 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.428248 5057 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 29 22:33:39.428291 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.428255 5057 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 29 22:33:39.429480 ip-10-0-140-74 kubenswrapper[5057]: E1029 22:33:39.429460 5057 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 29 22:33:39.429546 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.429400 5057 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-29T22:33:39.42823902Z","UUID":"8cc35777-9015-4631-8f96-bd60082e2fdd","Handler":null,"Name":"","Endpoint":""} Oct 29 22:33:39.430787 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.430763 5057 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 29 22:33:39.430787 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.430782 5057 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 29 22:33:39.529088 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.529030 5057 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 29 22:33:39.530235 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.530220 5057 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-74.ec2.internal" event="NodeHasSufficientMemory" Oct 29 22:33:39.530290 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.530249 5057 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-74.ec2.internal" event="NodeHasNoDiskPressure" Oct 29 22:33:39.530290 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.530259 5057 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-74.ec2.internal" event="NodeHasSufficientPID" Oct 29 22:33:39.530400 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.530298 5057 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-140-74.ec2.internal" Oct 29 22:33:39.562238 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.562215 5057 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-140-74.ec2.internal" Oct 29 22:33:39.562304 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.562273 5057 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-140-74.ec2.internal" Oct 29 22:33:39.621812 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.621765 5057 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-74.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-140-74.ec2.internal"] Oct 29 22:33:39.623347 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.623275 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-74.ec2.internal" event={"ID":"4c81f7639feb5d07aad0278bf28e49a4","Type":"ContainerStarted","Data":"6757406edfe64af5cec51cc522da581f0e82635378a89ce2ec008829acfa9837"} Oct 29 22:33:39.623455 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.623359 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-74.ec2.internal" event={"ID":"4c81f7639feb5d07aad0278bf28e49a4","Type":"ContainerDied","Data":"cc1013fca8c05a300d6b735484f7ea18f02fa49a4188949ccb6a61dadee5670d"} Oct 29 22:33:39.623455 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.623389 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-74.ec2.internal" event={"ID":"4c81f7639feb5d07aad0278bf28e49a4","Type":"ContainerStarted","Data":"39746dc1a72c981c5e9698f1f0c72d7c28b892b2793847e6fcf516fb3f1c727f"} Oct 29 22:33:39.623551 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.623471 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-74.ec2.internal" event={"ID":"ef9ac0a87cdfc5ee9b8eaa2797bc55b1","Type":"ContainerStarted","Data":"5224a8a015dc5215e49834b8a31d3d856f4db56d46c58670903965e367d96af7"} Oct 29 22:33:39.623551 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.623489 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-74.ec2.internal" event={"ID":"ef9ac0a87cdfc5ee9b8eaa2797bc55b1","Type":"ContainerStarted","Data":"2c31fee02392017ba8b41b846564c28cb85ccd960603212936086d59f108fd8a"} Oct 29 22:33:39.623826 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.623810 5057 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-74.ec2.internal" Oct 29 22:33:39.624722 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.624704 5057 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-74.ec2.internal" Oct 29 22:33:39.647339 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.647318 5057 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 29 22:33:39.647409 ip-10-0-140-74 kubenswrapper[5057]: E1029 22:33:39.647374 5057 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-140-74.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-74.ec2.internal" Oct 29 22:33:39.648036 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.648024 5057 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 29 22:33:39.648081 ip-10-0-140-74 kubenswrapper[5057]: E1029 22:33:39.648050 5057 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-140-74.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-74.ec2.internal" Oct 29 22:33:39.742659 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.742591 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4c81f7639feb5d07aad0278bf28e49a4-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-74.ec2.internal\" (UID: \"4c81f7639feb5d07aad0278bf28e49a4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-74.ec2.internal" Oct 29 22:33:39.742756 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.742672 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4c81f7639feb5d07aad0278bf28e49a4-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-74.ec2.internal\" (UID: \"4c81f7639feb5d07aad0278bf28e49a4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-74.ec2.internal" Oct 29 22:33:39.742756 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.742695 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ef9ac0a87cdfc5ee9b8eaa2797bc55b1-config\") pod \"kube-apiserver-proxy-ip-10-0-140-74.ec2.internal\" (UID: \"ef9ac0a87cdfc5ee9b8eaa2797bc55b1\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-74.ec2.internal" Oct 29 22:33:39.843318 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.843259 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4c81f7639feb5d07aad0278bf28e49a4-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-74.ec2.internal\" (UID: \"4c81f7639feb5d07aad0278bf28e49a4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-74.ec2.internal" Oct 29 22:33:39.843318 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.843280 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4c81f7639feb5d07aad0278bf28e49a4-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-74.ec2.internal\" (UID: \"4c81f7639feb5d07aad0278bf28e49a4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-74.ec2.internal" Oct 29 22:33:39.843318 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.843306 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4c81f7639feb5d07aad0278bf28e49a4-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-74.ec2.internal\" (UID: \"4c81f7639feb5d07aad0278bf28e49a4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-74.ec2.internal" Oct 29 22:33:39.843504 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.843326 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ef9ac0a87cdfc5ee9b8eaa2797bc55b1-config\") pod \"kube-apiserver-proxy-ip-10-0-140-74.ec2.internal\" (UID: \"ef9ac0a87cdfc5ee9b8eaa2797bc55b1\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-74.ec2.internal" Oct 29 22:33:39.843504 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.843377 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ef9ac0a87cdfc5ee9b8eaa2797bc55b1-config\") pod \"kube-apiserver-proxy-ip-10-0-140-74.ec2.internal\" (UID: \"ef9ac0a87cdfc5ee9b8eaa2797bc55b1\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-74.ec2.internal" Oct 29 22:33:39.843504 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:39.843403 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4c81f7639feb5d07aad0278bf28e49a4-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-74.ec2.internal\" (UID: \"4c81f7639feb5d07aad0278bf28e49a4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-74.ec2.internal" Oct 29 22:33:40.281126 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.281101 5057 apiserver.go:52] "Watching apiserver" Oct 29 22:33:40.326182 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.326149 5057 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 29 22:33:40.326944 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.326907 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-mrbgh","openshift-multus/multus-additional-cni-plugins-llm78","openshift-monitoring/cluster-monitoring-operator-884576c57-8c5hr","openshift-network-diagnostics/network-check-target-7jv5g","openshift-dns/dns-default-jjz47","openshift-network-diagnostics/network-check-source-6f78d59f99-p9ds4","openshift-ingress-canary/ingress-canary-czfl9","kube-system/konnectivity-agent-54f9x","kube-system/kube-apiserver-proxy-ip-10-0-140-74.ec2.internal","openshift-network-console/networking-console-plugin-86c478bb88-6rr4f","openshift-ingress/router-default-5d8d7cc579-8cwjp","openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-sk5qq","openshift-insights/periodic-gathering-lnrkb-btcvq","openshift-multus/network-metrics-daemon-hzlmg","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c96pg","openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-w5d4m","openshift-insights/insights-runtime-extractor-d2dnn","openshift-insights/insights-operator-7c4986d449-b7bwd","openshift-service-ca/service-ca-5fffc97bdf-9xmdb","openshift-image-registry/node-ca-pcjl7","kube-system/global-pull-secret-syncer-bp46g","openshift-multus/multus-v4q29","openshift-network-operator/iptables-alerter-wpzm2","openshift-image-registry/image-registry-78db97464f-6ljzh","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-74.ec2.internal","openshift-service-ca-operator/service-ca-operator-544d749c8c-pk97l","openshift-image-registry/image-registry-84866f86bd-2v6d7","kube-system/kas-connection-checker-685895c7c9-5fjlz","openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-p2qv4","openshift-ovn-kubernetes/ovnkube-node-khlhx","openshift-kube-storage-version-migrator/migrator-66df5d496b-c7kcc","openshift-dns/node-resolver-dpsd9","kube-system/kas-connection-checker-685895c7c9-jgvr6","openshift-console-operator/console-operator-7cd56b8dbc-vj6md","openshift-kube-storage-version-migrator/migrator-66df5d496b-vjlkr","openshift-cluster-node-tuning-operator/tuned-jtlmz"] Oct 29 22:33:40.328870 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.328841 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-5d8d7cc579-8cwjp" Oct 29 22:33:40.329251 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.329233 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-p2qv4" Oct 29 22:33:40.330197 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.330180 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-8c5hr" Oct 29 22:33:40.331135 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.331118 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-78db97464f-6ljzh" Oct 29 22:33:40.331538 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.331522 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-6rr4f" Oct 29 22:33:40.340522 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.340505 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hzlmg" Oct 29 22:33:40.341494 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.341478 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-84866f86bd-2v6d7" Oct 29 22:33:40.342152 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.342131 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Oct 29 22:33:40.342227 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.342157 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Oct 29 22:33:40.342227 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.342204 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Oct 29 22:33:40.342311 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.342244 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 29 22:33:40.342311 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.342245 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 29 22:33:40.342311 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.342257 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Oct 29 22:33:40.342589 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.342571 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-wwqh2\"" Oct 29 22:33:40.342589 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.342579 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-6b9sj\"" Oct 29 22:33:40.342791 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.342775 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-jjz47" Oct 29 22:33:40.343942 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.343926 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-d2dnn" Oct 29 22:33:40.344750 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.344721 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Oct 29 22:33:40.344846 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.344787 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Oct 29 22:33:40.344965 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.344951 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Oct 29 22:33:40.345044 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.345028 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 29 22:33:40.345185 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.345161 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 29 22:33:40.345185 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.345179 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Oct 29 22:33:40.345324 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.345172 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Oct 29 22:33:40.345324 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.345170 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Oct 29 22:33:40.345324 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.345214 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Oct 29 22:33:40.345634 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.345614 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Oct 29 22:33:40.345720 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.345652 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Oct 29 22:33:40.345720 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.345654 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Oct 29 22:33:40.345720 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.345668 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Oct 29 22:33:40.345823 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.345784 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Oct 29 22:33:40.345823 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.345815 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Oct 29 22:33:40.345946 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.345924 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Oct 29 22:33:40.346079 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.346063 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-7zghq\"" Oct 29 22:33:40.346174 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.346157 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 29 22:33:40.346212 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.346187 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Oct 29 22:33:40.346241 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.346225 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-sllfs\"" Oct 29 22:33:40.346289 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.346278 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-5rnmj\"" Oct 29 22:33:40.346351 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.346325 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-647gd\"" Oct 29 22:33:40.346395 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.346387 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Oct 29 22:33:40.346461 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.346430 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Oct 29 22:33:40.346526 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.346510 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-s24md\"" Oct 29 22:33:40.346593 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.346580 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-z2g4l\"" Oct 29 22:33:40.346744 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.346730 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Oct 29 22:33:40.346834 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.346821 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Oct 29 22:33:40.346955 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.346943 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Oct 29 22:33:40.347072 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.347059 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Oct 29 22:33:40.347117 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.347070 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Oct 29 22:33:40.347594 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.347575 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Oct 29 22:33:40.352894 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.352874 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-6b58k\"" Oct 29 22:33:40.354751 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.354736 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-nr2md\"" Oct 29 22:33:40.356914 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.356897 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Oct 29 22:33:40.357194 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.357164 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 29 22:33:40.357278 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.357240 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Oct 29 22:33:40.357278 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.357244 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 29 22:33:40.357387 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.357308 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 29 22:33:40.357438 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.357423 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Oct 29 22:33:40.357490 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.357469 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 29 22:33:40.358316 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.358292 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Oct 29 22:33:40.359021 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.359006 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Oct 29 22:33:40.362528 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.362487 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Oct 29 22:33:40.365224 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.365206 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 29 22:33:40.365291 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.365231 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 29 22:33:40.365325 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.365301 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 29 22:33:40.365325 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.365306 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 29 22:33:40.365325 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.365302 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 29 22:33:40.365451 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.365302 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 29 22:33:40.366271 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.366257 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Oct 29 22:33:40.366271 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.366264 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 29 22:33:40.366379 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.366304 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 29 22:33:40.366501 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.366486 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-p6rt5\"" Oct 29 22:33:40.366552 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.366519 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 29 22:33:40.366552 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.366546 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 29 22:33:40.366644 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.366551 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 29 22:33:40.366644 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.366586 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 29 22:33:40.366644 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.366596 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 29 22:33:40.366644 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.366626 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 29 22:33:40.366814 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.366668 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-vd45d\"" Oct 29 22:33:40.366814 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.366669 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 29 22:33:40.366814 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.366681 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 29 22:33:40.366814 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.366712 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Oct 29 22:33:40.366814 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.366725 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Oct 29 22:33:40.366814 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.366742 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-wccb2\"" Oct 29 22:33:40.366814 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.366751 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-kr4qv\"" Oct 29 22:33:40.366814 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.366768 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 29 22:33:40.366814 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.366797 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 29 22:33:40.367228 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.366848 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 29 22:33:40.367228 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.366864 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-8l2jp\"" Oct 29 22:33:40.367228 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.366908 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 29 22:33:40.367228 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.366937 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-rgvrg\"" Oct 29 22:33:40.367228 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.366962 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 29 22:33:40.367228 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.366970 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-dvjrv\"" Oct 29 22:33:40.367228 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.366981 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 29 22:33:40.367228 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.366990 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-mhfn7\"" Oct 29 22:33:40.367228 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.366992 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-f9xcm\"" Oct 29 22:33:40.367228 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.367022 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 29 22:33:40.367228 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.367048 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-g9cd8\"" Oct 29 22:33:40.367228 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.367077 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-zrmx7\"" Oct 29 22:33:40.367228 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.367086 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 29 22:33:40.367228 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.367099 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 29 22:33:40.367228 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.367109 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 29 22:33:40.367228 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.367118 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 29 22:33:40.367228 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.367138 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 29 22:33:40.367228 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.367153 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 29 22:33:40.367228 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.367204 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-8pd7h\"" Oct 29 22:33:40.367902 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.367262 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 29 22:33:40.367902 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.367283 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 29 22:33:40.367902 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.367344 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-rfj6j\"" Oct 29 22:33:40.367902 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.367365 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-gg4pb\"" Oct 29 22:33:40.367902 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.367410 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 29 22:33:40.367902 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.367448 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Oct 29 22:33:40.367902 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.367456 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-xzwzj\"" Oct 29 22:33:40.367902 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.367411 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 29 22:33:40.367902 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.367480 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 29 22:33:40.367902 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.367513 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Oct 29 22:33:40.367902 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.367516 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-t9ks9\"" Oct 29 22:33:40.367902 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.367524 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 29 22:33:40.367902 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.367539 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Oct 29 22:33:40.367902 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.367563 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Oct 29 22:33:40.367902 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.367581 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-ct6ft\"" Oct 29 22:33:40.367902 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.367603 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Oct 29 22:33:40.367902 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.367632 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Oct 29 22:33:40.367902 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.367637 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-c4xfj\"" Oct 29 22:33:40.395126 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.395108 5057 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 29 22:33:40.399032 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.398999 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:33:40.445047 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.445028 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71d80eec-b5e7-41d0-b58d-8a8d9a6c769c-metrics-certs\") pod \"router-default-5d8d7cc579-8cwjp\" (UID: \"71d80eec-b5e7-41d0-b58d-8a8d9a6c769c\") " pod="openshift-ingress/router-default-5d8d7cc579-8cwjp" Oct 29 22:33:40.445124 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.445051 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b23027ca-0748-4125-9bc8-9d149a3418e0-etc-selinux\") pod \"aws-ebs-csi-driver-node-c96pg\" (UID: \"b23027ca-0748-4125-9bc8-9d149a3418e0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c96pg" Oct 29 22:33:40.445124 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.445067 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/04195c1d-9fb7-479b-8061-f92e3c88aed2-systemd-units\") pod \"ovnkube-node-khlhx\" (UID: \"04195c1d-9fb7-479b-8061-f92e3c88aed2\") " pod="openshift-ovn-kubernetes/ovnkube-node-khlhx" Oct 29 22:33:40.445124 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.445080 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f190a1a0-6f6a-44d0-aeaa-b053cff4bd9c-run\") pod \"tuned-jtlmz\" (UID: \"f190a1a0-6f6a-44d0-aeaa-b053cff4bd9c\") " pod="openshift-cluster-node-tuning-operator/tuned-jtlmz" Oct 29 22:33:40.445124 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.445104 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qd7k\" (UniqueName: \"kubernetes.io/projected/49c67ce3-90fd-478d-9a15-e4267438ff69-kube-api-access-9qd7k\") pod \"service-ca-5fffc97bdf-9xmdb\" (UID: \"49c67ce3-90fd-478d-9a15-e4267438ff69\") " pod="openshift-service-ca/service-ca-5fffc97bdf-9xmdb" Oct 29 22:33:40.445365 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.445143 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5296979a-7927-4ee2-84b9-a93780a2ad22-registry-certificates\") pod \"image-registry-78db97464f-6ljzh\" (UID: \"5296979a-7927-4ee2-84b9-a93780a2ad22\") " pod="openshift-image-registry/image-registry-78db97464f-6ljzh" Oct 29 22:33:40.445365 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.445219 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/04195c1d-9fb7-479b-8061-f92e3c88aed2-run-openvswitch\") pod \"ovnkube-node-khlhx\" (UID: \"04195c1d-9fb7-479b-8061-f92e3c88aed2\") " pod="openshift-ovn-kubernetes/ovnkube-node-khlhx" Oct 29 22:33:40.445365 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.445281 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/04195c1d-9fb7-479b-8061-f92e3c88aed2-node-log\") pod \"ovnkube-node-khlhx\" (UID: \"04195c1d-9fb7-479b-8061-f92e3c88aed2\") " pod="openshift-ovn-kubernetes/ovnkube-node-khlhx" Oct 29 22:33:40.445365 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.445322 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8d5d\" (UniqueName: \"kubernetes.io/projected/dd4e19ce-ef37-44e5-8b57-9dafc590685c-kube-api-access-l8d5d\") pod \"network-check-source-6f78d59f99-p9ds4\" (UID: \"dd4e19ce-ef37-44e5-8b57-9dafc590685c\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-p9ds4" Oct 29 22:33:40.445565 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.445380 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5296979a-7927-4ee2-84b9-a93780a2ad22-trusted-ca\") pod \"image-registry-78db97464f-6ljzh\" (UID: \"5296979a-7927-4ee2-84b9-a93780a2ad22\") " pod="openshift-image-registry/image-registry-78db97464f-6ljzh" Oct 29 22:33:40.445565 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.445488 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/71d80eec-b5e7-41d0-b58d-8a8d9a6c769c-stats-auth\") pod \"router-default-5d8d7cc579-8cwjp\" (UID: \"71d80eec-b5e7-41d0-b58d-8a8d9a6c769c\") " pod="openshift-ingress/router-default-5d8d7cc579-8cwjp" Oct 29 22:33:40.445664 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.445591 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/04ff31d0-a870-4527-a0c3-285737359f95-host-slash\") pod \"iptables-alerter-wpzm2\" (UID: \"04ff31d0-a870-4527-a0c3-285737359f95\") " pod="openshift-network-operator/iptables-alerter-wpzm2" Oct 29 22:33:40.445715 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.445691 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/641f9ec1-c8e2-4774-9d4f-6a7a2da97b6f-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-8c5hr\" (UID: \"641f9ec1-c8e2-4774-9d4f-6a7a2da97b6f\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-8c5hr" Oct 29 22:33:40.445775 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.445721 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2d2d7446-0741-4144-9b19-608854f8a04e-konnectivity-ca\") pod \"konnectivity-agent-54f9x\" (UID: \"2d2d7446-0741-4144-9b19-608854f8a04e\") " pod="kube-system/konnectivity-agent-54f9x" Oct 29 22:33:40.445775 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.445741 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f190a1a0-6f6a-44d0-aeaa-b053cff4bd9c-etc-sysctl-conf\") pod \"tuned-jtlmz\" (UID: \"f190a1a0-6f6a-44d0-aeaa-b053cff4bd9c\") " pod="openshift-cluster-node-tuning-operator/tuned-jtlmz" Oct 29 22:33:40.445870 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.445809 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2ab2aa22-99af-4451-8d33-9e3543c8d9cd-multus-cni-dir\") pod \"multus-v4q29\" (UID: \"2ab2aa22-99af-4451-8d33-9e3543c8d9cd\") " pod="openshift-multus/multus-v4q29" Oct 29 22:33:40.445870 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.445845 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5296979a-7927-4ee2-84b9-a93780a2ad22-trusted-ca\") pod \"image-registry-78db97464f-6ljzh\" (UID: \"5296979a-7927-4ee2-84b9-a93780a2ad22\") " pod="openshift-image-registry/image-registry-78db97464f-6ljzh" Oct 29 22:33:40.445933 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.445848 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/6b34b3b1-84af-45f3-92ee-1ca720e8b263-config-volume\") pod \"dns-default-jjz47\" (UID: \"6b34b3b1-84af-45f3-92ee-1ca720e8b263\") " pod="openshift-dns/dns-default-jjz47" Oct 29 22:33:40.445933 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.445914 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/989956e8-1893-4bbe-a4bf-6a2081509080-image-registry-private-configuration\") pod \"image-registry-84866f86bd-2v6d7\" (UID: \"989956e8-1893-4bbe-a4bf-6a2081509080\") " pod="openshift-image-registry/image-registry-84866f86bd-2v6d7" Oct 29 22:33:40.445987 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.445943 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/f176efe0-febd-4652-94fc-91548ee3ab93-archives-path\") pod \"periodic-gathering-lnrkb-btcvq\" (UID: \"f176efe0-febd-4652-94fc-91548ee3ab93\") " pod="openshift-insights/periodic-gathering-lnrkb-btcvq" Oct 29 22:33:40.445987 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.445969 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a53078b9-a6a8-423c-b852-34eececa7ab1-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-p2qv4\" (UID: \"a53078b9-a6a8-423c-b852-34eececa7ab1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-p2qv4" Oct 29 22:33:40.446047 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.445995 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/8a972cc8-dce5-4f7c-b5d5-cb24ccf73bad-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-llm78\" (UID: \"8a972cc8-dce5-4f7c-b5d5-cb24ccf73bad\") " pod="openshift-multus/multus-additional-cni-plugins-llm78" Oct 29 22:33:40.446047 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.446003 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5296979a-7927-4ee2-84b9-a93780a2ad22-registry-certificates\") pod \"image-registry-78db97464f-6ljzh\" (UID: \"5296979a-7927-4ee2-84b9-a93780a2ad22\") " pod="openshift-image-registry/image-registry-78db97464f-6ljzh" Oct 29 22:33:40.446047 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.446010 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/6b34b3b1-84af-45f3-92ee-1ca720e8b263-config-volume\") pod \"dns-default-jjz47\" (UID: \"6b34b3b1-84af-45f3-92ee-1ca720e8b263\") " pod="openshift-dns/dns-default-jjz47" Oct 29 22:33:40.446047 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.446019 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/989956e8-1893-4bbe-a4bf-6a2081509080-trusted-ca\") pod \"image-registry-84866f86bd-2v6d7\" (UID: \"989956e8-1893-4bbe-a4bf-6a2081509080\") " pod="openshift-image-registry/image-registry-84866f86bd-2v6d7" Oct 29 22:33:40.446047 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.446012 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/641f9ec1-c8e2-4774-9d4f-6a7a2da97b6f-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-8c5hr\" (UID: \"641f9ec1-c8e2-4774-9d4f-6a7a2da97b6f\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-8c5hr" Oct 29 22:33:40.446182 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.445812 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/71d80eec-b5e7-41d0-b58d-8a8d9a6c769c-stats-auth\") pod \"router-default-5d8d7cc579-8cwjp\" (UID: \"71d80eec-b5e7-41d0-b58d-8a8d9a6c769c\") " pod="openshift-ingress/router-default-5d8d7cc579-8cwjp" Oct 29 22:33:40.446182 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.446056 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2d2d7446-0741-4144-9b19-608854f8a04e-konnectivity-ca\") pod \"konnectivity-agent-54f9x\" (UID: \"2d2d7446-0741-4144-9b19-608854f8a04e\") " pod="kube-system/konnectivity-agent-54f9x" Oct 29 22:33:40.446182 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.446061 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9pvq\" (UniqueName: \"kubernetes.io/projected/f176efe0-febd-4652-94fc-91548ee3ab93-kube-api-access-p9pvq\") pod \"periodic-gathering-lnrkb-btcvq\" (UID: \"f176efe0-febd-4652-94fc-91548ee3ab93\") " pod="openshift-insights/periodic-gathering-lnrkb-btcvq" Oct 29 22:33:40.446182 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.446092 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/079ed4e4-bdc4-405c-90a1-a22b84fcf948-data-volume\") pod \"insights-runtime-extractor-d2dnn\" (UID: \"079ed4e4-bdc4-405c-90a1-a22b84fcf948\") " pod="openshift-insights/insights-runtime-extractor-d2dnn" Oct 29 22:33:40.446182 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.446120 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0e1c2279-b6d4-44e0-84e6-94eb1206d2a0-hosts-file\") pod \"node-resolver-dpsd9\" (UID: \"0e1c2279-b6d4-44e0-84e6-94eb1206d2a0\") " pod="openshift-dns/node-resolver-dpsd9" Oct 29 22:33:40.446182 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.446147 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8a972cc8-dce5-4f7c-b5d5-cb24ccf73bad-cni-binary-copy\") pod \"multus-additional-cni-plugins-llm78\" (UID: \"8a972cc8-dce5-4f7c-b5d5-cb24ccf73bad\") " pod="openshift-multus/multus-additional-cni-plugins-llm78" Oct 29 22:33:40.446182 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.446173 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f8f92bfb-5cef-412e-a6c1-4bd793c59fe7-metrics-certs\") pod \"network-metrics-daemon-hzlmg\" (UID: \"f8f92bfb-5cef-412e-a6c1-4bd793c59fe7\") " pod="openshift-multus/network-metrics-daemon-hzlmg" Oct 29 22:33:40.446397 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.446200 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8hb6\" (UniqueName: \"kubernetes.io/projected/4071b0c7-c4f3-419a-8d6e-6acc144fc9a4-kube-api-access-n8hb6\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-sk5qq\" (UID: \"4071b0c7-c4f3-419a-8d6e-6acc144fc9a4\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-sk5qq" Oct 29 22:33:40.446397 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.446229 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5296979a-7927-4ee2-84b9-a93780a2ad22-registry-tls\") pod \"image-registry-78db97464f-6ljzh\" (UID: \"5296979a-7927-4ee2-84b9-a93780a2ad22\") " pod="openshift-image-registry/image-registry-78db97464f-6ljzh" Oct 29 22:33:40.446397 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.446252 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/989956e8-1893-4bbe-a4bf-6a2081509080-image-registry-private-configuration\") pod \"image-registry-84866f86bd-2v6d7\" (UID: \"989956e8-1893-4bbe-a4bf-6a2081509080\") " pod="openshift-image-registry/image-registry-84866f86bd-2v6d7" Oct 29 22:33:40.446397 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.446268 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/04195c1d-9fb7-479b-8061-f92e3c88aed2-host-kubelet\") pod \"ovnkube-node-khlhx\" (UID: \"04195c1d-9fb7-479b-8061-f92e3c88aed2\") " pod="openshift-ovn-kubernetes/ovnkube-node-khlhx" Oct 29 22:33:40.446397 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.446232 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/8a972cc8-dce5-4f7c-b5d5-cb24ccf73bad-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-llm78\" (UID: \"8a972cc8-dce5-4f7c-b5d5-cb24ccf73bad\") " pod="openshift-multus/multus-additional-cni-plugins-llm78" Oct 29 22:33:40.446397 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.446297 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/04195c1d-9fb7-479b-8061-f92e3c88aed2-host-slash\") pod \"ovnkube-node-khlhx\" (UID: \"04195c1d-9fb7-479b-8061-f92e3c88aed2\") " pod="openshift-ovn-kubernetes/ovnkube-node-khlhx" Oct 29 22:33:40.446397 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.446367 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/989956e8-1893-4bbe-a4bf-6a2081509080-trusted-ca\") pod \"image-registry-84866f86bd-2v6d7\" (UID: \"989956e8-1893-4bbe-a4bf-6a2081509080\") " pod="openshift-image-registry/image-registry-84866f86bd-2v6d7" Oct 29 22:33:40.446397 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.446380 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8a972cc8-dce5-4f7c-b5d5-cb24ccf73bad-cni-binary-copy\") pod \"multus-additional-cni-plugins-llm78\" (UID: \"8a972cc8-dce5-4f7c-b5d5-cb24ccf73bad\") " pod="openshift-multus/multus-additional-cni-plugins-llm78" Oct 29 22:33:40.446397 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.446382 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/f176efe0-febd-4652-94fc-91548ee3ab93-archives-path\") pod \"periodic-gathering-lnrkb-btcvq\" (UID: \"f176efe0-febd-4652-94fc-91548ee3ab93\") " pod="openshift-insights/periodic-gathering-lnrkb-btcvq" Oct 29 22:33:40.446757 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.446408 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/93dbae94-f92d-4157-a89b-d3c984a56013-serviceca\") pod \"node-ca-pcjl7\" (UID: \"93dbae94-f92d-4157-a89b-d3c984a56013\") " pod="openshift-image-registry/node-ca-pcjl7" Oct 29 22:33:40.446757 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.446417 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/079ed4e4-bdc4-405c-90a1-a22b84fcf948-data-volume\") pod \"insights-runtime-extractor-d2dnn\" (UID: \"079ed4e4-bdc4-405c-90a1-a22b84fcf948\") " pod="openshift-insights/insights-runtime-extractor-d2dnn" Oct 29 22:33:40.446757 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.446536 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfk2t\" (UniqueName: \"kubernetes.io/projected/989956e8-1893-4bbe-a4bf-6a2081509080-kube-api-access-gfk2t\") pod \"image-registry-84866f86bd-2v6d7\" (UID: \"989956e8-1893-4bbe-a4bf-6a2081509080\") " pod="openshift-image-registry/image-registry-84866f86bd-2v6d7" Oct 29 22:33:40.446757 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.446621 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6sv4r\" (UniqueName: \"kubernetes.io/projected/641f9ec1-c8e2-4774-9d4f-6a7a2da97b6f-kube-api-access-6sv4r\") pod \"cluster-monitoring-operator-884576c57-8c5hr\" (UID: \"641f9ec1-c8e2-4774-9d4f-6a7a2da97b6f\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-8c5hr" Oct 29 22:33:40.446757 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.446719 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/04195c1d-9fb7-479b-8061-f92e3c88aed2-var-lib-openvswitch\") pod \"ovnkube-node-khlhx\" (UID: \"04195c1d-9fb7-479b-8061-f92e3c88aed2\") " pod="openshift-ovn-kubernetes/ovnkube-node-khlhx" Oct 29 22:33:40.446916 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.446798 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/a53078b9-a6a8-423c-b852-34eececa7ab1-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-p2qv4\" (UID: \"a53078b9-a6a8-423c-b852-34eececa7ab1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-p2qv4" Oct 29 22:33:40.446916 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.446839 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f190a1a0-6f6a-44d0-aeaa-b053cff4bd9c-etc-kubernetes\") pod \"tuned-jtlmz\" (UID: \"f190a1a0-6f6a-44d0-aeaa-b053cff4bd9c\") " pod="openshift-cluster-node-tuning-operator/tuned-jtlmz" Oct 29 22:33:40.446916 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.446880 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7fmsg\" (UniqueName: \"kubernetes.io/projected/0e1c2279-b6d4-44e0-84e6-94eb1206d2a0-kube-api-access-7fmsg\") pod \"node-resolver-dpsd9\" (UID: \"0e1c2279-b6d4-44e0-84e6-94eb1206d2a0\") " pod="openshift-dns/node-resolver-dpsd9" Oct 29 22:33:40.446916 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.446902 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/989956e8-1893-4bbe-a4bf-6a2081509080-bound-sa-token\") pod \"image-registry-84866f86bd-2v6d7\" (UID: \"989956e8-1893-4bbe-a4bf-6a2081509080\") " pod="openshift-image-registry/image-registry-84866f86bd-2v6d7" Oct 29 22:33:40.447058 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.446949 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/2accd775-2173-4bfe-a413-7d243bd0820f-nginx-conf\") pod \"networking-console-plugin-86c478bb88-6rr4f\" (UID: \"2accd775-2173-4bfe-a413-7d243bd0820f\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-6rr4f" Oct 29 22:33:40.447058 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.446976 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/a53078b9-a6a8-423c-b852-34eececa7ab1-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-p2qv4\" (UID: \"a53078b9-a6a8-423c-b852-34eececa7ab1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-p2qv4" Oct 29 22:33:40.447058 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.447021 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8a972cc8-dce5-4f7c-b5d5-cb24ccf73bad-cnibin\") pod \"multus-additional-cni-plugins-llm78\" (UID: \"8a972cc8-dce5-4f7c-b5d5-cb24ccf73bad\") " pod="openshift-multus/multus-additional-cni-plugins-llm78" Oct 29 22:33:40.447181 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.447041 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/93dbae94-f92d-4157-a89b-d3c984a56013-serviceca\") pod \"node-ca-pcjl7\" (UID: \"93dbae94-f92d-4157-a89b-d3c984a56013\") " pod="openshift-image-registry/node-ca-pcjl7" Oct 29 22:33:40.447181 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.447136 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvgj5\" (UniqueName: \"kubernetes.io/projected/8a972cc8-dce5-4f7c-b5d5-cb24ccf73bad-kube-api-access-jvgj5\") pod \"multus-additional-cni-plugins-llm78\" (UID: \"8a972cc8-dce5-4f7c-b5d5-cb24ccf73bad\") " pod="openshift-multus/multus-additional-cni-plugins-llm78" Oct 29 22:33:40.447181 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.447165 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/2accd775-2173-4bfe-a413-7d243bd0820f-nginx-conf\") pod \"networking-console-plugin-86c478bb88-6rr4f\" (UID: \"2accd775-2173-4bfe-a413-7d243bd0820f\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-6rr4f" Oct 29 22:33:40.447326 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.447231 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2ab2aa22-99af-4451-8d33-9e3543c8d9cd-multus-conf-dir\") pod \"multus-v4q29\" (UID: \"2ab2aa22-99af-4451-8d33-9e3543c8d9cd\") " pod="openshift-multus/multus-v4q29" Oct 29 22:33:40.447326 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.447252 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/989956e8-1893-4bbe-a4bf-6a2081509080-registry-tls\") pod \"image-registry-84866f86bd-2v6d7\" (UID: \"989956e8-1893-4bbe-a4bf-6a2081509080\") " pod="openshift-image-registry/image-registry-84866f86bd-2v6d7" Oct 29 22:33:40.447326 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.447276 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/49c67ce3-90fd-478d-9a15-e4267438ff69-signing-cabundle\") pod \"service-ca-5fffc97bdf-9xmdb\" (UID: \"49c67ce3-90fd-478d-9a15-e4267438ff69\") " pod="openshift-service-ca/service-ca-5fffc97bdf-9xmdb" Oct 29 22:33:40.447326 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.447292 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/04195c1d-9fb7-479b-8061-f92e3c88aed2-etc-openvswitch\") pod \"ovnkube-node-khlhx\" (UID: \"04195c1d-9fb7-479b-8061-f92e3c88aed2\") " pod="openshift-ovn-kubernetes/ovnkube-node-khlhx" Oct 29 22:33:40.447492 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.447389 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/04195c1d-9fb7-479b-8061-f92e3c88aed2-host-run-ovn-kubernetes\") pod \"ovnkube-node-khlhx\" (UID: \"04195c1d-9fb7-479b-8061-f92e3c88aed2\") " pod="openshift-ovn-kubernetes/ovnkube-node-khlhx" Oct 29 22:33:40.447492 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.447456 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9v2r9\" (UniqueName: \"kubernetes.io/projected/2ab2aa22-99af-4451-8d33-9e3543c8d9cd-kube-api-access-9v2r9\") pod \"multus-v4q29\" (UID: \"2ab2aa22-99af-4451-8d33-9e3543c8d9cd\") " pod="openshift-multus/multus-v4q29" Oct 29 22:33:40.447492 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.447477 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/71d80eec-b5e7-41d0-b58d-8a8d9a6c769c-service-ca-bundle\") pod \"router-default-5d8d7cc579-8cwjp\" (UID: \"71d80eec-b5e7-41d0-b58d-8a8d9a6c769c\") " pod="openshift-ingress/router-default-5d8d7cc579-8cwjp" Oct 29 22:33:40.447492 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.447491 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/93dbae94-f92d-4157-a89b-d3c984a56013-host\") pod \"node-ca-pcjl7\" (UID: \"93dbae94-f92d-4157-a89b-d3c984a56013\") " pod="openshift-image-registry/node-ca-pcjl7" Oct 29 22:33:40.447678 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.447513 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wf78m\" (UniqueName: \"kubernetes.io/projected/93dbae94-f92d-4157-a89b-d3c984a56013-kube-api-access-wf78m\") pod \"node-ca-pcjl7\" (UID: \"93dbae94-f92d-4157-a89b-d3c984a56013\") " pod="openshift-image-registry/node-ca-pcjl7" Oct 29 22:33:40.447678 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.447529 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fp2lh\" (UniqueName: \"kubernetes.io/projected/5296979a-7927-4ee2-84b9-a93780a2ad22-kube-api-access-fp2lh\") pod \"image-registry-78db97464f-6ljzh\" (UID: \"5296979a-7927-4ee2-84b9-a93780a2ad22\") " pod="openshift-image-registry/image-registry-78db97464f-6ljzh" Oct 29 22:33:40.447678 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.447544 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/65c96dfb-06e9-4bea-bb83-f63b4c62d860-serving-cert\") pod \"insights-operator-7c4986d449-b7bwd\" (UID: \"65c96dfb-06e9-4bea-bb83-f63b4c62d860\") " pod="openshift-insights/insights-operator-7c4986d449-b7bwd" Oct 29 22:33:40.447678 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.447575 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/49c67ce3-90fd-478d-9a15-e4267438ff69-signing-cabundle\") pod \"service-ca-5fffc97bdf-9xmdb\" (UID: \"49c67ce3-90fd-478d-9a15-e4267438ff69\") " pod="openshift-service-ca/service-ca-5fffc97bdf-9xmdb" Oct 29 22:33:40.447678 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.447606 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2ab2aa22-99af-4451-8d33-9e3543c8d9cd-multus-daemon-config\") pod \"multus-v4q29\" (UID: \"2ab2aa22-99af-4451-8d33-9e3543c8d9cd\") " pod="openshift-multus/multus-v4q29" Oct 29 22:33:40.447678 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.447645 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b23027ca-0748-4125-9bc8-9d149a3418e0-device-dir\") pod \"aws-ebs-csi-driver-node-c96pg\" (UID: \"b23027ca-0748-4125-9bc8-9d149a3418e0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c96pg" Oct 29 22:33:40.447940 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.447723 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/65c96dfb-06e9-4bea-bb83-f63b4c62d860-serving-cert\") pod \"insights-operator-7c4986d449-b7bwd\" (UID: \"65c96dfb-06e9-4bea-bb83-f63b4c62d860\") " pod="openshift-insights/insights-operator-7c4986d449-b7bwd" Oct 29 22:33:40.447940 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.447799 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9srv\" (UniqueName: \"kubernetes.io/projected/a4e1c52c-1eeb-4617-a0c9-d052521360d3-kube-api-access-l9srv\") pod \"network-check-target-7jv5g\" (UID: \"a4e1c52c-1eeb-4617-a0c9-d052521360d3\") " pod="openshift-network-diagnostics/network-check-target-7jv5g" Oct 29 22:33:40.447940 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.447826 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2ab2aa22-99af-4451-8d33-9e3543c8d9cd-multus-daemon-config\") pod \"multus-v4q29\" (UID: \"2ab2aa22-99af-4451-8d33-9e3543c8d9cd\") " pod="openshift-multus/multus-v4q29" Oct 29 22:33:40.447940 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.447822 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whjbn\" (UniqueName: \"kubernetes.io/projected/4ff372bf-82aa-4007-b74c-df0c779e5271-kube-api-access-whjbn\") pod \"kas-connection-checker-685895c7c9-mrbgh\" (UID: \"4ff372bf-82aa-4007-b74c-df0c779e5271\") " pod="kube-system/kas-connection-checker-685895c7c9-mrbgh" Oct 29 22:33:40.447940 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.447877 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/079ed4e4-bdc4-405c-90a1-a22b84fcf948-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-d2dnn\" (UID: \"079ed4e4-bdc4-405c-90a1-a22b84fcf948\") " pod="openshift-insights/insights-runtime-extractor-d2dnn" Oct 29 22:33:40.448081 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.448002 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f190a1a0-6f6a-44d0-aeaa-b053cff4bd9c-host\") pod \"tuned-jtlmz\" (UID: \"f190a1a0-6f6a-44d0-aeaa-b053cff4bd9c\") " pod="openshift-cluster-node-tuning-operator/tuned-jtlmz" Oct 29 22:33:40.448081 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.448023 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mxpd\" (UniqueName: \"kubernetes.io/projected/f190a1a0-6f6a-44d0-aeaa-b053cff4bd9c-kube-api-access-4mxpd\") pod \"tuned-jtlmz\" (UID: \"f190a1a0-6f6a-44d0-aeaa-b053cff4bd9c\") " pod="openshift-cluster-node-tuning-operator/tuned-jtlmz" Oct 29 22:33:40.448081 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.448040 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6f50ee57-a710-4325-bd45-4f861452ede4-original-pull-secret\") pod \"global-pull-secret-syncer-bp46g\" (UID: \"6f50ee57-a710-4325-bd45-4f861452ede4\") " pod="kube-system/global-pull-secret-syncer-bp46g" Oct 29 22:33:40.448081 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.448055 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f176efe0-febd-4652-94fc-91548ee3ab93-trusted-ca-bundle\") pod \"periodic-gathering-lnrkb-btcvq\" (UID: \"f176efe0-febd-4652-94fc-91548ee3ab93\") " pod="openshift-insights/periodic-gathering-lnrkb-btcvq" Oct 29 22:33:40.448081 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.448071 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/04195c1d-9fb7-479b-8061-f92e3c88aed2-ovnkube-config\") pod \"ovnkube-node-khlhx\" (UID: \"04195c1d-9fb7-479b-8061-f92e3c88aed2\") " pod="openshift-ovn-kubernetes/ovnkube-node-khlhx" Oct 29 22:33:40.448229 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.448088 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/079ed4e4-bdc4-405c-90a1-a22b84fcf948-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-d2dnn\" (UID: \"079ed4e4-bdc4-405c-90a1-a22b84fcf948\") " pod="openshift-insights/insights-runtime-extractor-d2dnn" Oct 29 22:33:40.448273 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.448261 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqxx5\" (UniqueName: \"kubernetes.io/projected/b23027ca-0748-4125-9bc8-9d149a3418e0-kube-api-access-jqxx5\") pod \"aws-ebs-csi-driver-node-c96pg\" (UID: \"b23027ca-0748-4125-9bc8-9d149a3418e0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c96pg" Oct 29 22:33:40.448321 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.448288 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/989956e8-1893-4bbe-a4bf-6a2081509080-registry-certificates\") pod \"image-registry-84866f86bd-2v6d7\" (UID: \"989956e8-1893-4bbe-a4bf-6a2081509080\") " pod="openshift-image-registry/image-registry-84866f86bd-2v6d7" Oct 29 22:33:40.448321 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.448304 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkxzs\" (UniqueName: \"kubernetes.io/projected/c635d035-053b-49aa-9559-79c31e8e49ff-kube-api-access-mkxzs\") pod \"service-ca-operator-544d749c8c-pk97l\" (UID: \"c635d035-053b-49aa-9559-79c31e8e49ff\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-pk97l" Oct 29 22:33:40.448453 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.448320 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4071b0c7-c4f3-419a-8d6e-6acc144fc9a4-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-sk5qq\" (UID: \"4071b0c7-c4f3-419a-8d6e-6acc144fc9a4\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-sk5qq" Oct 29 22:33:40.448453 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.448375 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5296979a-7927-4ee2-84b9-a93780a2ad22-image-registry-private-configuration\") pod \"image-registry-78db97464f-6ljzh\" (UID: \"5296979a-7927-4ee2-84b9-a93780a2ad22\") " pod="openshift-image-registry/image-registry-78db97464f-6ljzh" Oct 29 22:33:40.448453 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.448403 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f190a1a0-6f6a-44d0-aeaa-b053cff4bd9c-tmp\") pod \"tuned-jtlmz\" (UID: \"f190a1a0-6f6a-44d0-aeaa-b053cff4bd9c\") " pod="openshift-cluster-node-tuning-operator/tuned-jtlmz" Oct 29 22:33:40.448453 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.448324 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6f50ee57-a710-4325-bd45-4f861452ede4-original-pull-secret\") pod \"global-pull-secret-syncer-bp46g\" (UID: \"6f50ee57-a710-4325-bd45-4f861452ede4\") " pod="kube-system/global-pull-secret-syncer-bp46g" Oct 29 22:33:40.448453 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.448426 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8406bb9f-3a4e-42f3-a822-f754d9b8e7a8-cert\") pod \"ingress-canary-czfl9\" (UID: \"8406bb9f-3a4e-42f3-a822-f754d9b8e7a8\") " pod="openshift-ingress-canary/ingress-canary-czfl9" Oct 29 22:33:40.448686 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.448457 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/04195c1d-9fb7-479b-8061-f92e3c88aed2-ovnkube-config\") pod \"ovnkube-node-khlhx\" (UID: \"04195c1d-9fb7-479b-8061-f92e3c88aed2\") " pod="openshift-ovn-kubernetes/ovnkube-node-khlhx" Oct 29 22:33:40.448686 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.448539 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f190a1a0-6f6a-44d0-aeaa-b053cff4bd9c-tmp\") pod \"tuned-jtlmz\" (UID: \"f190a1a0-6f6a-44d0-aeaa-b053cff4bd9c\") " pod="openshift-cluster-node-tuning-operator/tuned-jtlmz" Oct 29 22:33:40.448686 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.448565 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f176efe0-febd-4652-94fc-91548ee3ab93-trusted-ca-bundle\") pod \"periodic-gathering-lnrkb-btcvq\" (UID: \"f176efe0-febd-4652-94fc-91548ee3ab93\") " pod="openshift-insights/periodic-gathering-lnrkb-btcvq" Oct 29 22:33:40.448686 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.448574 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/641f9ec1-c8e2-4774-9d4f-6a7a2da97b6f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-8c5hr\" (UID: \"641f9ec1-c8e2-4774-9d4f-6a7a2da97b6f\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-8c5hr" Oct 29 22:33:40.448686 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.448629 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4071b0c7-c4f3-419a-8d6e-6acc144fc9a4-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-sk5qq\" (UID: \"4071b0c7-c4f3-419a-8d6e-6acc144fc9a4\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-sk5qq" Oct 29 22:33:40.448686 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.448642 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8406bb9f-3a4e-42f3-a822-f754d9b8e7a8-cert\") pod \"ingress-canary-czfl9\" (UID: \"8406bb9f-3a4e-42f3-a822-f754d9b8e7a8\") " pod="openshift-ingress-canary/ingress-canary-czfl9" Oct 29 22:33:40.448686 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.448615 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f190a1a0-6f6a-44d0-aeaa-b053cff4bd9c-lib-modules\") pod \"tuned-jtlmz\" (UID: \"f190a1a0-6f6a-44d0-aeaa-b053cff4bd9c\") " pod="openshift-cluster-node-tuning-operator/tuned-jtlmz" Oct 29 22:33:40.448686 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.448644 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5296979a-7927-4ee2-84b9-a93780a2ad22-image-registry-private-configuration\") pod \"image-registry-78db97464f-6ljzh\" (UID: \"5296979a-7927-4ee2-84b9-a93780a2ad22\") " pod="openshift-image-registry/image-registry-78db97464f-6ljzh" Oct 29 22:33:40.448686 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.448679 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2ab2aa22-99af-4451-8d33-9e3543c8d9cd-host-run-k8s-cni-cncf-io\") pod \"multus-v4q29\" (UID: \"2ab2aa22-99af-4451-8d33-9e3543c8d9cd\") " pod="openshift-multus/multus-v4q29" Oct 29 22:33:40.448970 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.448728 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/04195c1d-9fb7-479b-8061-f92e3c88aed2-run-systemd\") pod \"ovnkube-node-khlhx\" (UID: \"04195c1d-9fb7-479b-8061-f92e3c88aed2\") " pod="openshift-ovn-kubernetes/ovnkube-node-khlhx" Oct 29 22:33:40.448970 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.448744 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/04195c1d-9fb7-479b-8061-f92e3c88aed2-ovn-node-metrics-cert\") pod \"ovnkube-node-khlhx\" (UID: \"04195c1d-9fb7-479b-8061-f92e3c88aed2\") " pod="openshift-ovn-kubernetes/ovnkube-node-khlhx" Oct 29 22:33:40.448970 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.448760 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2ab2aa22-99af-4451-8d33-9e3543c8d9cd-system-cni-dir\") pod \"multus-v4q29\" (UID: \"2ab2aa22-99af-4451-8d33-9e3543c8d9cd\") " pod="openshift-multus/multus-v4q29" Oct 29 22:33:40.448970 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.448774 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b23027ca-0748-4125-9bc8-9d149a3418e0-kubelet-dir\") pod \"aws-ebs-csi-driver-node-c96pg\" (UID: \"b23027ca-0748-4125-9bc8-9d149a3418e0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c96pg" Oct 29 22:33:40.448970 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.448807 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b23027ca-0748-4125-9bc8-9d149a3418e0-sys-fs\") pod \"aws-ebs-csi-driver-node-c96pg\" (UID: \"b23027ca-0748-4125-9bc8-9d149a3418e0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c96pg" Oct 29 22:33:40.448970 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.448891 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfdj9\" (UniqueName: \"kubernetes.io/projected/b5961d4b-f3c0-4f17-b1bc-2781cb3dcd6e-kube-api-access-hfdj9\") pod \"migrator-66df5d496b-c7kcc\" (UID: \"b5961d4b-f3c0-4f17-b1bc-2781cb3dcd6e\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-c7kcc" Oct 29 22:33:40.448970 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.448928 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/989956e8-1893-4bbe-a4bf-6a2081509080-registry-certificates\") pod \"image-registry-84866f86bd-2v6d7\" (UID: \"989956e8-1893-4bbe-a4bf-6a2081509080\") " pod="openshift-image-registry/image-registry-84866f86bd-2v6d7" Oct 29 22:33:40.449196 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.448983 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2ab2aa22-99af-4451-8d33-9e3543c8d9cd-host-var-lib-cni-multus\") pod \"multus-v4q29\" (UID: \"2ab2aa22-99af-4451-8d33-9e3543c8d9cd\") " pod="openshift-multus/multus-v4q29" Oct 29 22:33:40.449196 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.449029 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2ab2aa22-99af-4451-8d33-9e3543c8d9cd-hostroot\") pod \"multus-v4q29\" (UID: \"2ab2aa22-99af-4451-8d33-9e3543c8d9cd\") " pod="openshift-multus/multus-v4q29" Oct 29 22:33:40.449196 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.449035 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/04195c1d-9fb7-479b-8061-f92e3c88aed2-ovn-node-metrics-cert\") pod \"ovnkube-node-khlhx\" (UID: \"04195c1d-9fb7-479b-8061-f92e3c88aed2\") " pod="openshift-ovn-kubernetes/ovnkube-node-khlhx" Oct 29 22:33:40.449196 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.449071 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4nczb\" (UniqueName: \"kubernetes.io/projected/f8f92bfb-5cef-412e-a6c1-4bd793c59fe7-kube-api-access-4nczb\") pod \"network-metrics-daemon-hzlmg\" (UID: \"f8f92bfb-5cef-412e-a6c1-4bd793c59fe7\") " pod="openshift-multus/network-metrics-daemon-hzlmg" Oct 29 22:33:40.449196 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.449101 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/aa6a190e-3283-458a-9dfd-2de7cb193fc5-config\") pod \"console-operator-7cd56b8dbc-vj6md\" (UID: \"aa6a190e-3283-458a-9dfd-2de7cb193fc5\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-vj6md" Oct 29 22:33:40.449196 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.449118 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/04195c1d-9fb7-479b-8061-f92e3c88aed2-log-socket\") pod \"ovnkube-node-khlhx\" (UID: \"04195c1d-9fb7-479b-8061-f92e3c88aed2\") " pod="openshift-ovn-kubernetes/ovnkube-node-khlhx" Oct 29 22:33:40.449196 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.449133 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/65c96dfb-06e9-4bea-bb83-f63b4c62d860-snapshots\") pod \"insights-operator-7c4986d449-b7bwd\" (UID: \"65c96dfb-06e9-4bea-bb83-f63b4c62d860\") " pod="openshift-insights/insights-operator-7c4986d449-b7bwd" Oct 29 22:33:40.449196 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.449156 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2d2d7446-0741-4144-9b19-608854f8a04e-agent-certs\") pod \"konnectivity-agent-54f9x\" (UID: \"2d2d7446-0741-4144-9b19-608854f8a04e\") " pod="kube-system/konnectivity-agent-54f9x" Oct 29 22:33:40.449196 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.449189 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2ab2aa22-99af-4451-8d33-9e3543c8d9cd-host-run-netns\") pod \"multus-v4q29\" (UID: \"2ab2aa22-99af-4451-8d33-9e3543c8d9cd\") " pod="openshift-multus/multus-v4q29" Oct 29 22:33:40.449550 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.449254 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2ab2aa22-99af-4451-8d33-9e3543c8d9cd-host-run-multus-certs\") pod \"multus-v4q29\" (UID: \"2ab2aa22-99af-4451-8d33-9e3543c8d9cd\") " pod="openshift-multus/multus-v4q29" Oct 29 22:33:40.449550 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.449258 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/65c96dfb-06e9-4bea-bb83-f63b4c62d860-snapshots\") pod \"insights-operator-7c4986d449-b7bwd\" (UID: \"65c96dfb-06e9-4bea-bb83-f63b4c62d860\") " pod="openshift-insights/insights-operator-7c4986d449-b7bwd" Oct 29 22:33:40.449550 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.449312 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/aa6a190e-3283-458a-9dfd-2de7cb193fc5-config\") pod \"console-operator-7cd56b8dbc-vj6md\" (UID: \"aa6a190e-3283-458a-9dfd-2de7cb193fc5\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-vj6md" Oct 29 22:33:40.449550 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.449364 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/71d80eec-b5e7-41d0-b58d-8a8d9a6c769c-default-certificate\") pod \"router-default-5d8d7cc579-8cwjp\" (UID: \"71d80eec-b5e7-41d0-b58d-8a8d9a6c769c\") " pod="openshift-ingress/router-default-5d8d7cc579-8cwjp" Oct 29 22:33:40.449550 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.449397 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b23027ca-0748-4125-9bc8-9d149a3418e0-registration-dir\") pod \"aws-ebs-csi-driver-node-c96pg\" (UID: \"b23027ca-0748-4125-9bc8-9d149a3418e0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c96pg" Oct 29 22:33:40.449550 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.449420 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjwwn\" (UniqueName: \"kubernetes.io/projected/587893f5-e85a-46a9-829f-545a42c10056-kube-api-access-bjwwn\") pod \"kas-connection-checker-685895c7c9-jgvr6\" (UID: \"587893f5-e85a-46a9-829f-545a42c10056\") " pod="kube-system/kas-connection-checker-685895c7c9-jgvr6" Oct 29 22:33:40.449550 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.449425 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2d2d7446-0741-4144-9b19-608854f8a04e-agent-certs\") pod \"konnectivity-agent-54f9x\" (UID: \"2d2d7446-0741-4144-9b19-608854f8a04e\") " pod="kube-system/konnectivity-agent-54f9x" Oct 29 22:33:40.449550 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.449437 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4071b0c7-c4f3-419a-8d6e-6acc144fc9a4-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-sk5qq\" (UID: \"4071b0c7-c4f3-419a-8d6e-6acc144fc9a4\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-sk5qq" Oct 29 22:33:40.449881 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.449609 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/65c96dfb-06e9-4bea-bb83-f63b4c62d860-service-ca-bundle\") pod \"insights-operator-7c4986d449-b7bwd\" (UID: \"65c96dfb-06e9-4bea-bb83-f63b4c62d860\") " pod="openshift-insights/insights-operator-7c4986d449-b7bwd" Oct 29 22:33:40.449881 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.449626 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/71d80eec-b5e7-41d0-b58d-8a8d9a6c769c-default-certificate\") pod \"router-default-5d8d7cc579-8cwjp\" (UID: \"71d80eec-b5e7-41d0-b58d-8a8d9a6c769c\") " pod="openshift-ingress/router-default-5d8d7cc579-8cwjp" Oct 29 22:33:40.449881 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.449652 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2tw5\" (UniqueName: \"kubernetes.io/projected/65c96dfb-06e9-4bea-bb83-f63b4c62d860-kube-api-access-z2tw5\") pod \"insights-operator-7c4986d449-b7bwd\" (UID: \"65c96dfb-06e9-4bea-bb83-f63b4c62d860\") " pod="openshift-insights/insights-operator-7c4986d449-b7bwd" Oct 29 22:33:40.449881 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.449738 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4071b0c7-c4f3-419a-8d6e-6acc144fc9a4-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-sk5qq\" (UID: \"4071b0c7-c4f3-419a-8d6e-6acc144fc9a4\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-sk5qq" Oct 29 22:33:40.449881 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.449752 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2ab2aa22-99af-4451-8d33-9e3543c8d9cd-multus-socket-dir-parent\") pod \"multus-v4q29\" (UID: \"2ab2aa22-99af-4451-8d33-9e3543c8d9cd\") " pod="openshift-multus/multus-v4q29" Oct 29 22:33:40.449881 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.449774 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/65c96dfb-06e9-4bea-bb83-f63b4c62d860-service-ca-bundle\") pod \"insights-operator-7c4986d449-b7bwd\" (UID: \"65c96dfb-06e9-4bea-bb83-f63b4c62d860\") " pod="openshift-insights/insights-operator-7c4986d449-b7bwd" Oct 29 22:33:40.449881 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.449839 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tdnl\" (UniqueName: \"kubernetes.io/projected/71d80eec-b5e7-41d0-b58d-8a8d9a6c769c-kube-api-access-2tdnl\") pod \"router-default-5d8d7cc579-8cwjp\" (UID: \"71d80eec-b5e7-41d0-b58d-8a8d9a6c769c\") " pod="openshift-ingress/router-default-5d8d7cc579-8cwjp" Oct 29 22:33:40.449881 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.449862 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/989956e8-1893-4bbe-a4bf-6a2081509080-installation-pull-secrets\") pod \"image-registry-84866f86bd-2v6d7\" (UID: \"989956e8-1893-4bbe-a4bf-6a2081509080\") " pod="openshift-image-registry/image-registry-84866f86bd-2v6d7" Oct 29 22:33:40.449881 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.449878 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5296979a-7927-4ee2-84b9-a93780a2ad22-bound-sa-token\") pod \"image-registry-78db97464f-6ljzh\" (UID: \"5296979a-7927-4ee2-84b9-a93780a2ad22\") " pod="openshift-image-registry/image-registry-78db97464f-6ljzh" Oct 29 22:33:40.450176 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.449893 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-698ds\" (UniqueName: \"kubernetes.io/projected/a53078b9-a6a8-423c-b852-34eececa7ab1-kube-api-access-698ds\") pod \"cluster-samples-operator-6ddb6c8b94-p2qv4\" (UID: \"a53078b9-a6a8-423c-b852-34eececa7ab1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-p2qv4" Oct 29 22:33:40.450176 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.449909 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f190a1a0-6f6a-44d0-aeaa-b053cff4bd9c-etc-sysconfig\") pod \"tuned-jtlmz\" (UID: \"f190a1a0-6f6a-44d0-aeaa-b053cff4bd9c\") " pod="openshift-cluster-node-tuning-operator/tuned-jtlmz" Oct 29 22:33:40.450176 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.449926 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f190a1a0-6f6a-44d0-aeaa-b053cff4bd9c-etc-sysctl-d\") pod \"tuned-jtlmz\" (UID: \"f190a1a0-6f6a-44d0-aeaa-b053cff4bd9c\") " pod="openshift-cluster-node-tuning-operator/tuned-jtlmz" Oct 29 22:33:40.450176 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.449975 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b23027ca-0748-4125-9bc8-9d149a3418e0-socket-dir\") pod \"aws-ebs-csi-driver-node-c96pg\" (UID: \"b23027ca-0748-4125-9bc8-9d149a3418e0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c96pg" Oct 29 22:33:40.450176 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.450031 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/989956e8-1893-4bbe-a4bf-6a2081509080-installation-pull-secrets\") pod \"image-registry-84866f86bd-2v6d7\" (UID: \"989956e8-1893-4bbe-a4bf-6a2081509080\") " pod="openshift-image-registry/image-registry-84866f86bd-2v6d7" Oct 29 22:33:40.450176 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.450068 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tlxfr\" (UniqueName: \"kubernetes.io/projected/04195c1d-9fb7-479b-8061-f92e3c88aed2-kube-api-access-tlxfr\") pod \"ovnkube-node-khlhx\" (UID: \"04195c1d-9fb7-479b-8061-f92e3c88aed2\") " pod="openshift-ovn-kubernetes/ovnkube-node-khlhx" Oct 29 22:33:40.450176 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.450102 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/65c96dfb-06e9-4bea-bb83-f63b4c62d860-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-b7bwd\" (UID: \"65c96dfb-06e9-4bea-bb83-f63b4c62d860\") " pod="openshift-insights/insights-operator-7c4986d449-b7bwd" Oct 29 22:33:40.450176 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.450176 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/079ed4e4-bdc4-405c-90a1-a22b84fcf948-crio-socket\") pod \"insights-runtime-extractor-d2dnn\" (UID: \"079ed4e4-bdc4-405c-90a1-a22b84fcf948\") " pod="openshift-insights/insights-runtime-extractor-d2dnn" Oct 29 22:33:40.450406 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.450263 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f190a1a0-6f6a-44d0-aeaa-b053cff4bd9c-etc-systemd\") pod \"tuned-jtlmz\" (UID: \"f190a1a0-6f6a-44d0-aeaa-b053cff4bd9c\") " pod="openshift-cluster-node-tuning-operator/tuned-jtlmz" Oct 29 22:33:40.450406 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.450278 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/04195c1d-9fb7-479b-8061-f92e3c88aed2-host-cni-netd\") pod \"ovnkube-node-khlhx\" (UID: \"04195c1d-9fb7-479b-8061-f92e3c88aed2\") " pod="openshift-ovn-kubernetes/ovnkube-node-khlhx" Oct 29 22:33:40.450406 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.450300 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6b34b3b1-84af-45f3-92ee-1ca720e8b263-metrics-tls\") pod \"dns-default-jjz47\" (UID: \"6b34b3b1-84af-45f3-92ee-1ca720e8b263\") " pod="openshift-dns/dns-default-jjz47" Oct 29 22:33:40.450406 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.450326 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jxgn\" (UniqueName: \"kubernetes.io/projected/04ff31d0-a870-4527-a0c3-285737359f95-kube-api-access-8jxgn\") pod \"iptables-alerter-wpzm2\" (UID: \"04ff31d0-a870-4527-a0c3-285737359f95\") " pod="openshift-network-operator/iptables-alerter-wpzm2" Oct 29 22:33:40.450406 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.450364 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvmbg\" (UniqueName: \"kubernetes.io/projected/8406bb9f-3a4e-42f3-a822-f754d9b8e7a8-kube-api-access-xvmbg\") pod \"ingress-canary-czfl9\" (UID: \"8406bb9f-3a4e-42f3-a822-f754d9b8e7a8\") " pod="openshift-ingress-canary/ingress-canary-czfl9" Oct 29 22:33:40.450563 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.450443 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5296979a-7927-4ee2-84b9-a93780a2ad22-installation-pull-secrets\") pod \"image-registry-78db97464f-6ljzh\" (UID: \"5296979a-7927-4ee2-84b9-a93780a2ad22\") " pod="openshift-image-registry/image-registry-78db97464f-6ljzh" Oct 29 22:33:40.450563 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.450469 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/04195c1d-9fb7-479b-8061-f92e3c88aed2-ovnkube-script-lib\") pod \"ovnkube-node-khlhx\" (UID: \"04195c1d-9fb7-479b-8061-f92e3c88aed2\") " pod="openshift-ovn-kubernetes/ovnkube-node-khlhx" Oct 29 22:33:40.450563 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.450494 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f190a1a0-6f6a-44d0-aeaa-b053cff4bd9c-etc-modprobe-d\") pod \"tuned-jtlmz\" (UID: \"f190a1a0-6f6a-44d0-aeaa-b053cff4bd9c\") " pod="openshift-cluster-node-tuning-operator/tuned-jtlmz" Oct 29 22:33:40.450563 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.450520 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8gw9q\" (UniqueName: \"kubernetes.io/projected/6b34b3b1-84af-45f3-92ee-1ca720e8b263-kube-api-access-8gw9q\") pod \"dns-default-jjz47\" (UID: \"6b34b3b1-84af-45f3-92ee-1ca720e8b263\") " pod="openshift-dns/dns-default-jjz47" Oct 29 22:33:40.450707 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.450569 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f190a1a0-6f6a-44d0-aeaa-b053cff4bd9c-var-lib-kubelet\") pod \"tuned-jtlmz\" (UID: \"f190a1a0-6f6a-44d0-aeaa-b053cff4bd9c\") " pod="openshift-cluster-node-tuning-operator/tuned-jtlmz" Oct 29 22:33:40.450707 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.450605 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e1c2279-b6d4-44e0-84e6-94eb1206d2a0-tmp-dir\") pod \"node-resolver-dpsd9\" (UID: \"0e1c2279-b6d4-44e0-84e6-94eb1206d2a0\") " pod="openshift-dns/node-resolver-dpsd9" Oct 29 22:33:40.450707 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.450642 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfd8g\" (UniqueName: \"kubernetes.io/projected/652bedff-87f1-4b2f-917f-571a37dd2f3f-kube-api-access-mfd8g\") pod \"migrator-66df5d496b-vjlkr\" (UID: \"652bedff-87f1-4b2f-917f-571a37dd2f3f\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-vjlkr" Oct 29 22:33:40.450707 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.450669 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5296979a-7927-4ee2-84b9-a93780a2ad22-installation-pull-secrets\") pod \"image-registry-78db97464f-6ljzh\" (UID: \"5296979a-7927-4ee2-84b9-a93780a2ad22\") " pod="openshift-image-registry/image-registry-78db97464f-6ljzh" Oct 29 22:33:40.450707 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.450702 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/04195c1d-9fb7-479b-8061-f92e3c88aed2-run-ovn\") pod \"ovnkube-node-khlhx\" (UID: \"04195c1d-9fb7-479b-8061-f92e3c88aed2\") " pod="openshift-ovn-kubernetes/ovnkube-node-khlhx" Oct 29 22:33:40.450843 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.450717 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e1c2279-b6d4-44e0-84e6-94eb1206d2a0-tmp-dir\") pod \"node-resolver-dpsd9\" (UID: \"0e1c2279-b6d4-44e0-84e6-94eb1206d2a0\") " pod="openshift-dns/node-resolver-dpsd9" Oct 29 22:33:40.450843 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.450732 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2accd775-2173-4bfe-a413-7d243bd0820f-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-6rr4f\" (UID: \"2accd775-2173-4bfe-a413-7d243bd0820f\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-6rr4f" Oct 29 22:33:40.450843 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.450770 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/04195c1d-9fb7-479b-8061-f92e3c88aed2-ovnkube-script-lib\") pod \"ovnkube-node-khlhx\" (UID: \"04195c1d-9fb7-479b-8061-f92e3c88aed2\") " pod="openshift-ovn-kubernetes/ovnkube-node-khlhx" Oct 29 22:33:40.450843 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.450790 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c635d035-053b-49aa-9559-79c31e8e49ff-serving-cert\") pod \"service-ca-operator-544d749c8c-pk97l\" (UID: \"c635d035-053b-49aa-9559-79c31e8e49ff\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-pk97l" Oct 29 22:33:40.450843 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.450824 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b34b3b1-84af-45f3-92ee-1ca720e8b263-tmp-dir\") pod \"dns-default-jjz47\" (UID: \"6b34b3b1-84af-45f3-92ee-1ca720e8b263\") " pod="openshift-dns/dns-default-jjz47" Oct 29 22:33:40.450977 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.450849 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/04195c1d-9fb7-479b-8061-f92e3c88aed2-host-cni-bin\") pod \"ovnkube-node-khlhx\" (UID: \"04195c1d-9fb7-479b-8061-f92e3c88aed2\") " pod="openshift-ovn-kubernetes/ovnkube-node-khlhx" Oct 29 22:33:40.450977 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.450867 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f190a1a0-6f6a-44d0-aeaa-b053cff4bd9c-etc-tuned\") pod \"tuned-jtlmz\" (UID: \"f190a1a0-6f6a-44d0-aeaa-b053cff4bd9c\") " pod="openshift-cluster-node-tuning-operator/tuned-jtlmz" Oct 29 22:33:40.450977 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.450884 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8a972cc8-dce5-4f7c-b5d5-cb24ccf73bad-tuning-conf-dir\") pod \"multus-additional-cni-plugins-llm78\" (UID: \"8a972cc8-dce5-4f7c-b5d5-cb24ccf73bad\") " pod="openshift-multus/multus-additional-cni-plugins-llm78" Oct 29 22:33:40.450977 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.450886 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/65c96dfb-06e9-4bea-bb83-f63b4c62d860-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-b7bwd\" (UID: \"65c96dfb-06e9-4bea-bb83-f63b4c62d860\") " pod="openshift-insights/insights-operator-7c4986d449-b7bwd" Oct 29 22:33:40.450977 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.450908 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2ab2aa22-99af-4451-8d33-9e3543c8d9cd-host-var-lib-kubelet\") pod \"multus-v4q29\" (UID: \"2ab2aa22-99af-4451-8d33-9e3543c8d9cd\") " pod="openshift-multus/multus-v4q29" Oct 29 22:33:40.450977 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.450946 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2ab2aa22-99af-4451-8d33-9e3543c8d9cd-etc-kubernetes\") pod \"multus-v4q29\" (UID: \"2ab2aa22-99af-4451-8d33-9e3543c8d9cd\") " pod="openshift-multus/multus-v4q29" Oct 29 22:33:40.451176 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.450988 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b34b3b1-84af-45f3-92ee-1ca720e8b263-tmp-dir\") pod \"dns-default-jjz47\" (UID: \"6b34b3b1-84af-45f3-92ee-1ca720e8b263\") " pod="openshift-dns/dns-default-jjz47" Oct 29 22:33:40.451176 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.450998 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f190a1a0-6f6a-44d0-aeaa-b053cff4bd9c-etc-tuned\") pod \"tuned-jtlmz\" (UID: \"f190a1a0-6f6a-44d0-aeaa-b053cff4bd9c\") " pod="openshift-cluster-node-tuning-operator/tuned-jtlmz" Oct 29 22:33:40.451176 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.451029 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/aa6a190e-3283-458a-9dfd-2de7cb193fc5-serving-cert\") pod \"console-operator-7cd56b8dbc-vj6md\" (UID: \"aa6a190e-3283-458a-9dfd-2de7cb193fc5\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-vj6md" Oct 29 22:33:40.451176 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.451089 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c635d035-053b-49aa-9559-79c31e8e49ff-serving-cert\") pod \"service-ca-operator-544d749c8c-pk97l\" (UID: \"c635d035-053b-49aa-9559-79c31e8e49ff\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-pk97l" Oct 29 22:33:40.451176 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.451135 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/04195c1d-9fb7-479b-8061-f92e3c88aed2-host-run-netns\") pod \"ovnkube-node-khlhx\" (UID: \"04195c1d-9fb7-479b-8061-f92e3c88aed2\") " pod="openshift-ovn-kubernetes/ovnkube-node-khlhx" Oct 29 22:33:40.451176 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.451143 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/aa6a190e-3283-458a-9dfd-2de7cb193fc5-serving-cert\") pod \"console-operator-7cd56b8dbc-vj6md\" (UID: \"aa6a190e-3283-458a-9dfd-2de7cb193fc5\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-vj6md" Oct 29 22:33:40.451438 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.451188 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ln6mw\" (UniqueName: \"kubernetes.io/projected/079ed4e4-bdc4-405c-90a1-a22b84fcf948-kube-api-access-ln6mw\") pod \"insights-runtime-extractor-d2dnn\" (UID: \"079ed4e4-bdc4-405c-90a1-a22b84fcf948\") " pod="openshift-insights/insights-runtime-extractor-d2dnn" Oct 29 22:33:40.451438 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.451231 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2ab2aa22-99af-4451-8d33-9e3543c8d9cd-cnibin\") pod \"multus-v4q29\" (UID: \"2ab2aa22-99af-4451-8d33-9e3543c8d9cd\") " pod="openshift-multus/multus-v4q29" Oct 29 22:33:40.451438 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.451293 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2ab2aa22-99af-4451-8d33-9e3543c8d9cd-cni-binary-copy\") pod \"multus-v4q29\" (UID: \"2ab2aa22-99af-4451-8d33-9e3543c8d9cd\") " pod="openshift-multus/multus-v4q29" Oct 29 22:33:40.451438 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.451322 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6f50ee57-a710-4325-bd45-4f861452ede4-dbus\") pod \"global-pull-secret-syncer-bp46g\" (UID: \"6f50ee57-a710-4325-bd45-4f861452ede4\") " pod="kube-system/global-pull-secret-syncer-bp46g" Oct 29 22:33:40.451438 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.451426 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/65c96dfb-06e9-4bea-bb83-f63b4c62d860-tmp\") pod \"insights-operator-7c4986d449-b7bwd\" (UID: \"65c96dfb-06e9-4bea-bb83-f63b4c62d860\") " pod="openshift-insights/insights-operator-7c4986d449-b7bwd" Oct 29 22:33:40.451632 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.451456 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c635d035-053b-49aa-9559-79c31e8e49ff-config\") pod \"service-ca-operator-544d749c8c-pk97l\" (UID: \"c635d035-053b-49aa-9559-79c31e8e49ff\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-pk97l" Oct 29 22:33:40.451632 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.451491 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8a972cc8-dce5-4f7c-b5d5-cb24ccf73bad-system-cni-dir\") pod \"multus-additional-cni-plugins-llm78\" (UID: \"8a972cc8-dce5-4f7c-b5d5-cb24ccf73bad\") " pod="openshift-multus/multus-additional-cni-plugins-llm78" Oct 29 22:33:40.451632 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.451498 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2ab2aa22-99af-4451-8d33-9e3543c8d9cd-cni-binary-copy\") pod \"multus-v4q29\" (UID: \"2ab2aa22-99af-4451-8d33-9e3543c8d9cd\") " pod="openshift-multus/multus-v4q29" Oct 29 22:33:40.451632 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.451525 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/65c96dfb-06e9-4bea-bb83-f63b4c62d860-tmp\") pod \"insights-operator-7c4986d449-b7bwd\" (UID: \"65c96dfb-06e9-4bea-bb83-f63b4c62d860\") " pod="openshift-insights/insights-operator-7c4986d449-b7bwd" Oct 29 22:33:40.451632 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.451556 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/8a972cc8-dce5-4f7c-b5d5-cb24ccf73bad-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-llm78\" (UID: \"8a972cc8-dce5-4f7c-b5d5-cb24ccf73bad\") " pod="openshift-multus/multus-additional-cni-plugins-llm78" Oct 29 22:33:40.451632 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.451604 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/989956e8-1893-4bbe-a4bf-6a2081509080-ca-trust-extracted\") pod \"image-registry-84866f86bd-2v6d7\" (UID: \"989956e8-1893-4bbe-a4bf-6a2081509080\") " pod="openshift-image-registry/image-registry-84866f86bd-2v6d7" Oct 29 22:33:40.451632 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.451620 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c635d035-053b-49aa-9559-79c31e8e49ff-config\") pod \"service-ca-operator-544d749c8c-pk97l\" (UID: \"c635d035-053b-49aa-9559-79c31e8e49ff\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-pk97l" Oct 29 22:33:40.451916 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.451662 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f176efe0-febd-4652-94fc-91548ee3ab93-service-ca-bundle\") pod \"periodic-gathering-lnrkb-btcvq\" (UID: \"f176efe0-febd-4652-94fc-91548ee3ab93\") " pod="openshift-insights/periodic-gathering-lnrkb-btcvq" Oct 29 22:33:40.451916 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.451682 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/989956e8-1893-4bbe-a4bf-6a2081509080-ca-trust-extracted\") pod \"image-registry-84866f86bd-2v6d7\" (UID: \"989956e8-1893-4bbe-a4bf-6a2081509080\") " pod="openshift-image-registry/image-registry-84866f86bd-2v6d7" Oct 29 22:33:40.451916 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.451700 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/04195c1d-9fb7-479b-8061-f92e3c88aed2-env-overrides\") pod \"ovnkube-node-khlhx\" (UID: \"04195c1d-9fb7-479b-8061-f92e3c88aed2\") " pod="openshift-ovn-kubernetes/ovnkube-node-khlhx" Oct 29 22:33:40.451916 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.451691 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/8a972cc8-dce5-4f7c-b5d5-cb24ccf73bad-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-llm78\" (UID: \"8a972cc8-dce5-4f7c-b5d5-cb24ccf73bad\") " pod="openshift-multus/multus-additional-cni-plugins-llm78" Oct 29 22:33:40.451916 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.451726 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vv2vp\" (UniqueName: \"kubernetes.io/projected/79ad3230-163e-447d-9f04-a072479ff957-kube-api-access-vv2vp\") pod \"kas-connection-checker-685895c7c9-5fjlz\" (UID: \"79ad3230-163e-447d-9f04-a072479ff957\") " pod="kube-system/kas-connection-checker-685895c7c9-5fjlz" Oct 29 22:33:40.451916 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.451825 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6f50ee57-a710-4325-bd45-4f861452ede4-kubelet-config\") pod \"global-pull-secret-syncer-bp46g\" (UID: \"6f50ee57-a710-4325-bd45-4f861452ede4\") " pod="kube-system/global-pull-secret-syncer-bp46g" Oct 29 22:33:40.451916 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.451846 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f176efe0-febd-4652-94fc-91548ee3ab93-service-ca-bundle\") pod \"periodic-gathering-lnrkb-btcvq\" (UID: \"f176efe0-febd-4652-94fc-91548ee3ab93\") " pod="openshift-insights/periodic-gathering-lnrkb-btcvq" Oct 29 22:33:40.452237 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.451947 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/aa6a190e-3283-458a-9dfd-2de7cb193fc5-trusted-ca\") pod \"console-operator-7cd56b8dbc-vj6md\" (UID: \"aa6a190e-3283-458a-9dfd-2de7cb193fc5\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-vj6md" Oct 29 22:33:40.452237 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.451974 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4xwz\" (UniqueName: \"kubernetes.io/projected/aa6a190e-3283-458a-9dfd-2de7cb193fc5-kube-api-access-c4xwz\") pod \"console-operator-7cd56b8dbc-vj6md\" (UID: \"aa6a190e-3283-458a-9dfd-2de7cb193fc5\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-vj6md" Oct 29 22:33:40.452237 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.451986 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/04195c1d-9fb7-479b-8061-f92e3c88aed2-env-overrides\") pod \"ovnkube-node-khlhx\" (UID: \"04195c1d-9fb7-479b-8061-f92e3c88aed2\") " pod="openshift-ovn-kubernetes/ovnkube-node-khlhx" Oct 29 22:33:40.452237 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.451998 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/04195c1d-9fb7-479b-8061-f92e3c88aed2-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-khlhx\" (UID: \"04195c1d-9fb7-479b-8061-f92e3c88aed2\") " pod="openshift-ovn-kubernetes/ovnkube-node-khlhx" Oct 29 22:33:40.452237 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.452030 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7h8fj\" (UniqueName: \"kubernetes.io/projected/c54ef754-c91e-4bc5-8e83-08523aa09d37-kube-api-access-7h8fj\") pod \"volume-data-source-validator-58c9f5f46d-w5d4m\" (UID: \"c54ef754-c91e-4bc5-8e83-08523aa09d37\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-w5d4m" Oct 29 22:33:40.452237 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.452066 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f190a1a0-6f6a-44d0-aeaa-b053cff4bd9c-sys\") pod \"tuned-jtlmz\" (UID: \"f190a1a0-6f6a-44d0-aeaa-b053cff4bd9c\") " pod="openshift-cluster-node-tuning-operator/tuned-jtlmz" Oct 29 22:33:40.452237 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.452113 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/04ff31d0-a870-4527-a0c3-285737359f95-iptables-alerter-script\") pod \"iptables-alerter-wpzm2\" (UID: \"04ff31d0-a870-4527-a0c3-285737359f95\") " pod="openshift-network-operator/iptables-alerter-wpzm2" Oct 29 22:33:40.452237 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.452235 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/aa6a190e-3283-458a-9dfd-2de7cb193fc5-trusted-ca\") pod \"console-operator-7cd56b8dbc-vj6md\" (UID: \"aa6a190e-3283-458a-9dfd-2de7cb193fc5\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-vj6md" Oct 29 22:33:40.452555 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.452239 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5296979a-7927-4ee2-84b9-a93780a2ad22-ca-trust-extracted\") pod \"image-registry-78db97464f-6ljzh\" (UID: \"5296979a-7927-4ee2-84b9-a93780a2ad22\") " pod="openshift-image-registry/image-registry-78db97464f-6ljzh" Oct 29 22:33:40.452555 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.452277 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2ab2aa22-99af-4451-8d33-9e3543c8d9cd-host-var-lib-cni-bin\") pod \"multus-v4q29\" (UID: \"2ab2aa22-99af-4451-8d33-9e3543c8d9cd\") " pod="openshift-multus/multus-v4q29" Oct 29 22:33:40.452555 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.452295 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/49c67ce3-90fd-478d-9a15-e4267438ff69-signing-key\") pod \"service-ca-5fffc97bdf-9xmdb\" (UID: \"49c67ce3-90fd-478d-9a15-e4267438ff69\") " pod="openshift-service-ca/service-ca-5fffc97bdf-9xmdb" Oct 29 22:33:40.452555 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.452293 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5296979a-7927-4ee2-84b9-a93780a2ad22-ca-trust-extracted\") pod \"image-registry-78db97464f-6ljzh\" (UID: \"5296979a-7927-4ee2-84b9-a93780a2ad22\") " pod="openshift-image-registry/image-registry-78db97464f-6ljzh" Oct 29 22:33:40.452555 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.452515 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/49c67ce3-90fd-478d-9a15-e4267438ff69-signing-key\") pod \"service-ca-5fffc97bdf-9xmdb\" (UID: \"49c67ce3-90fd-478d-9a15-e4267438ff69\") " pod="openshift-service-ca/service-ca-5fffc97bdf-9xmdb" Oct 29 22:33:40.452828 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.452403 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/04ff31d0-a870-4527-a0c3-285737359f95-iptables-alerter-script\") pod \"iptables-alerter-wpzm2\" (UID: \"04ff31d0-a870-4527-a0c3-285737359f95\") " pod="openshift-network-operator/iptables-alerter-wpzm2" Oct 29 22:33:40.464198 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.464179 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qd7k\" (UniqueName: \"kubernetes.io/projected/49c67ce3-90fd-478d-9a15-e4267438ff69-kube-api-access-9qd7k\") pod \"service-ca-5fffc97bdf-9xmdb\" (UID: \"49c67ce3-90fd-478d-9a15-e4267438ff69\") " pod="openshift-service-ca/service-ca-5fffc97bdf-9xmdb" Oct 29 22:33:40.464291 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.464275 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8d5d\" (UniqueName: \"kubernetes.io/projected/dd4e19ce-ef37-44e5-8b57-9dafc590685c-kube-api-access-l8d5d\") pod \"network-check-source-6f78d59f99-p9ds4\" (UID: \"dd4e19ce-ef37-44e5-8b57-9dafc590685c\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-p9ds4" Oct 29 22:33:40.464773 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.464759 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8hb6\" (UniqueName: \"kubernetes.io/projected/4071b0c7-c4f3-419a-8d6e-6acc144fc9a4-kube-api-access-n8hb6\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-sk5qq\" (UID: \"4071b0c7-c4f3-419a-8d6e-6acc144fc9a4\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-sk5qq" Oct 29 22:33:40.465785 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.465758 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfk2t\" (UniqueName: \"kubernetes.io/projected/989956e8-1893-4bbe-a4bf-6a2081509080-kube-api-access-gfk2t\") pod \"image-registry-84866f86bd-2v6d7\" (UID: \"989956e8-1893-4bbe-a4bf-6a2081509080\") " pod="openshift-image-registry/image-registry-84866f86bd-2v6d7" Oct 29 22:33:40.465994 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.465979 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6sv4r\" (UniqueName: \"kubernetes.io/projected/641f9ec1-c8e2-4774-9d4f-6a7a2da97b6f-kube-api-access-6sv4r\") pod \"cluster-monitoring-operator-884576c57-8c5hr\" (UID: \"641f9ec1-c8e2-4774-9d4f-6a7a2da97b6f\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-8c5hr" Oct 29 22:33:40.467694 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.467678 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9pvq\" (UniqueName: \"kubernetes.io/projected/f176efe0-febd-4652-94fc-91548ee3ab93-kube-api-access-p9pvq\") pod \"periodic-gathering-lnrkb-btcvq\" (UID: \"f176efe0-febd-4652-94fc-91548ee3ab93\") " pod="openshift-insights/periodic-gathering-lnrkb-btcvq" Oct 29 22:33:40.468735 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.468721 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/989956e8-1893-4bbe-a4bf-6a2081509080-bound-sa-token\") pod \"image-registry-84866f86bd-2v6d7\" (UID: \"989956e8-1893-4bbe-a4bf-6a2081509080\") " pod="openshift-image-registry/image-registry-84866f86bd-2v6d7" Oct 29 22:33:40.470970 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.470951 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7fmsg\" (UniqueName: \"kubernetes.io/projected/0e1c2279-b6d4-44e0-84e6-94eb1206d2a0-kube-api-access-7fmsg\") pod \"node-resolver-dpsd9\" (UID: \"0e1c2279-b6d4-44e0-84e6-94eb1206d2a0\") " pod="openshift-dns/node-resolver-dpsd9" Oct 29 22:33:40.471121 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.471099 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mxpd\" (UniqueName: \"kubernetes.io/projected/f190a1a0-6f6a-44d0-aeaa-b053cff4bd9c-kube-api-access-4mxpd\") pod \"tuned-jtlmz\" (UID: \"f190a1a0-6f6a-44d0-aeaa-b053cff4bd9c\") " pod="openshift-cluster-node-tuning-operator/tuned-jtlmz" Oct 29 22:33:40.471197 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.471147 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wf78m\" (UniqueName: \"kubernetes.io/projected/93dbae94-f92d-4157-a89b-d3c984a56013-kube-api-access-wf78m\") pod \"node-ca-pcjl7\" (UID: \"93dbae94-f92d-4157-a89b-d3c984a56013\") " pod="openshift-image-registry/node-ca-pcjl7" Oct 29 22:33:40.471250 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.471224 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvgj5\" (UniqueName: \"kubernetes.io/projected/8a972cc8-dce5-4f7c-b5d5-cb24ccf73bad-kube-api-access-jvgj5\") pod \"multus-additional-cni-plugins-llm78\" (UID: \"8a972cc8-dce5-4f7c-b5d5-cb24ccf73bad\") " pod="openshift-multus/multus-additional-cni-plugins-llm78" Oct 29 22:33:40.471379 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.471363 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9v2r9\" (UniqueName: \"kubernetes.io/projected/2ab2aa22-99af-4451-8d33-9e3543c8d9cd-kube-api-access-9v2r9\") pod \"multus-v4q29\" (UID: \"2ab2aa22-99af-4451-8d33-9e3543c8d9cd\") " pod="openshift-multus/multus-v4q29" Oct 29 22:33:40.471656 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.471641 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkxzs\" (UniqueName: \"kubernetes.io/projected/c635d035-053b-49aa-9559-79c31e8e49ff-kube-api-access-mkxzs\") pod \"service-ca-operator-544d749c8c-pk97l\" (UID: \"c635d035-053b-49aa-9559-79c31e8e49ff\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-pk97l" Oct 29 22:33:40.472497 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.472479 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9srv\" (UniqueName: \"kubernetes.io/projected/a4e1c52c-1eeb-4617-a0c9-d052521360d3-kube-api-access-l9srv\") pod \"network-check-target-7jv5g\" (UID: \"a4e1c52c-1eeb-4617-a0c9-d052521360d3\") " pod="openshift-network-diagnostics/network-check-target-7jv5g" Oct 29 22:33:40.473173 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.473157 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fp2lh\" (UniqueName: \"kubernetes.io/projected/5296979a-7927-4ee2-84b9-a93780a2ad22-kube-api-access-fp2lh\") pod \"image-registry-78db97464f-6ljzh\" (UID: \"5296979a-7927-4ee2-84b9-a93780a2ad22\") " pod="openshift-image-registry/image-registry-78db97464f-6ljzh" Oct 29 22:33:40.474393 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.474375 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-whjbn\" (UniqueName: \"kubernetes.io/projected/4ff372bf-82aa-4007-b74c-df0c779e5271-kube-api-access-whjbn\") pod \"kas-connection-checker-685895c7c9-mrbgh\" (UID: \"4ff372bf-82aa-4007-b74c-df0c779e5271\") " pod="kube-system/kas-connection-checker-685895c7c9-mrbgh" Oct 29 22:33:40.478764 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.478745 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4nczb\" (UniqueName: \"kubernetes.io/projected/f8f92bfb-5cef-412e-a6c1-4bd793c59fe7-kube-api-access-4nczb\") pod \"network-metrics-daemon-hzlmg\" (UID: \"f8f92bfb-5cef-412e-a6c1-4bd793c59fe7\") " pod="openshift-multus/network-metrics-daemon-hzlmg" Oct 29 22:33:40.480202 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.480182 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2tw5\" (UniqueName: \"kubernetes.io/projected/65c96dfb-06e9-4bea-bb83-f63b4c62d860-kube-api-access-z2tw5\") pod \"insights-operator-7c4986d449-b7bwd\" (UID: \"65c96dfb-06e9-4bea-bb83-f63b4c62d860\") " pod="openshift-insights/insights-operator-7c4986d449-b7bwd" Oct 29 22:33:40.480550 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.480533 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jxgn\" (UniqueName: \"kubernetes.io/projected/04ff31d0-a870-4527-a0c3-285737359f95-kube-api-access-8jxgn\") pod \"iptables-alerter-wpzm2\" (UID: \"04ff31d0-a870-4527-a0c3-285737359f95\") " pod="openshift-network-operator/iptables-alerter-wpzm2" Oct 29 22:33:40.480588 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.480562 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5296979a-7927-4ee2-84b9-a93780a2ad22-bound-sa-token\") pod \"image-registry-78db97464f-6ljzh\" (UID: \"5296979a-7927-4ee2-84b9-a93780a2ad22\") " pod="openshift-image-registry/image-registry-78db97464f-6ljzh" Oct 29 22:33:40.483226 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.483207 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7h8fj\" (UniqueName: \"kubernetes.io/projected/c54ef754-c91e-4bc5-8e83-08523aa09d37-kube-api-access-7h8fj\") pod \"volume-data-source-validator-58c9f5f46d-w5d4m\" (UID: \"c54ef754-c91e-4bc5-8e83-08523aa09d37\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-w5d4m" Oct 29 22:33:40.483432 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.483412 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tlxfr\" (UniqueName: \"kubernetes.io/projected/04195c1d-9fb7-479b-8061-f92e3c88aed2-kube-api-access-tlxfr\") pod \"ovnkube-node-khlhx\" (UID: \"04195c1d-9fb7-479b-8061-f92e3c88aed2\") " pod="openshift-ovn-kubernetes/ovnkube-node-khlhx" Oct 29 22:33:40.483498 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.483435 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8gw9q\" (UniqueName: \"kubernetes.io/projected/6b34b3b1-84af-45f3-92ee-1ca720e8b263-kube-api-access-8gw9q\") pod \"dns-default-jjz47\" (UID: \"6b34b3b1-84af-45f3-92ee-1ca720e8b263\") " pod="openshift-dns/dns-default-jjz47" Oct 29 22:33:40.483558 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.483504 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqxx5\" (UniqueName: \"kubernetes.io/projected/b23027ca-0748-4125-9bc8-9d149a3418e0-kube-api-access-jqxx5\") pod \"aws-ebs-csi-driver-node-c96pg\" (UID: \"b23027ca-0748-4125-9bc8-9d149a3418e0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c96pg" Oct 29 22:33:40.483778 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.483741 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ln6mw\" (UniqueName: \"kubernetes.io/projected/079ed4e4-bdc4-405c-90a1-a22b84fcf948-kube-api-access-ln6mw\") pod \"insights-runtime-extractor-d2dnn\" (UID: \"079ed4e4-bdc4-405c-90a1-a22b84fcf948\") " pod="openshift-insights/insights-runtime-extractor-d2dnn" Oct 29 22:33:40.483842 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.483825 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vv2vp\" (UniqueName: \"kubernetes.io/projected/79ad3230-163e-447d-9f04-a072479ff957-kube-api-access-vv2vp\") pod \"kas-connection-checker-685895c7c9-5fjlz\" (UID: \"79ad3230-163e-447d-9f04-a072479ff957\") " pod="kube-system/kas-connection-checker-685895c7c9-5fjlz" Oct 29 22:33:40.483968 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.483952 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfd8g\" (UniqueName: \"kubernetes.io/projected/652bedff-87f1-4b2f-917f-571a37dd2f3f-kube-api-access-mfd8g\") pod \"migrator-66df5d496b-vjlkr\" (UID: \"652bedff-87f1-4b2f-917f-571a37dd2f3f\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-vjlkr" Oct 29 22:33:40.484365 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.484324 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2tdnl\" (UniqueName: \"kubernetes.io/projected/71d80eec-b5e7-41d0-b58d-8a8d9a6c769c-kube-api-access-2tdnl\") pod \"router-default-5d8d7cc579-8cwjp\" (UID: \"71d80eec-b5e7-41d0-b58d-8a8d9a6c769c\") " pod="openshift-ingress/router-default-5d8d7cc579-8cwjp" Oct 29 22:33:40.485445 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.485426 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvmbg\" (UniqueName: \"kubernetes.io/projected/8406bb9f-3a4e-42f3-a822-f754d9b8e7a8-kube-api-access-xvmbg\") pod \"ingress-canary-czfl9\" (UID: \"8406bb9f-3a4e-42f3-a822-f754d9b8e7a8\") " pod="openshift-ingress-canary/ingress-canary-czfl9" Oct 29 22:33:40.485734 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.485716 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-698ds\" (UniqueName: \"kubernetes.io/projected/a53078b9-a6a8-423c-b852-34eececa7ab1-kube-api-access-698ds\") pod \"cluster-samples-operator-6ddb6c8b94-p2qv4\" (UID: \"a53078b9-a6a8-423c-b852-34eececa7ab1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-p2qv4" Oct 29 22:33:40.486234 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.486218 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4xwz\" (UniqueName: \"kubernetes.io/projected/aa6a190e-3283-458a-9dfd-2de7cb193fc5-kube-api-access-c4xwz\") pod \"console-operator-7cd56b8dbc-vj6md\" (UID: \"aa6a190e-3283-458a-9dfd-2de7cb193fc5\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-vj6md" Oct 29 22:33:40.488403 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.488380 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjwwn\" (UniqueName: \"kubernetes.io/projected/587893f5-e85a-46a9-829f-545a42c10056-kube-api-access-bjwwn\") pod \"kas-connection-checker-685895c7c9-jgvr6\" (UID: \"587893f5-e85a-46a9-829f-545a42c10056\") " pod="kube-system/kas-connection-checker-685895c7c9-jgvr6" Oct 29 22:33:40.488766 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.488742 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hfdj9\" (UniqueName: \"kubernetes.io/projected/b5961d4b-f3c0-4f17-b1bc-2781cb3dcd6e-kube-api-access-hfdj9\") pod \"migrator-66df5d496b-c7kcc\" (UID: \"b5961d4b-f3c0-4f17-b1bc-2781cb3dcd6e\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-c7kcc" Oct 29 22:33:40.536921 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.536853 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:33:40.553089 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.553065 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2ab2aa22-99af-4451-8d33-9e3543c8d9cd-host-var-lib-cni-bin\") pod \"multus-v4q29\" (UID: \"2ab2aa22-99af-4451-8d33-9e3543c8d9cd\") " pod="openshift-multus/multus-v4q29" Oct 29 22:33:40.553186 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.553092 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71d80eec-b5e7-41d0-b58d-8a8d9a6c769c-metrics-certs\") pod \"router-default-5d8d7cc579-8cwjp\" (UID: \"71d80eec-b5e7-41d0-b58d-8a8d9a6c769c\") " pod="openshift-ingress/router-default-5d8d7cc579-8cwjp" Oct 29 22:33:40.553186 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.553109 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b23027ca-0748-4125-9bc8-9d149a3418e0-etc-selinux\") pod \"aws-ebs-csi-driver-node-c96pg\" (UID: \"b23027ca-0748-4125-9bc8-9d149a3418e0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c96pg" Oct 29 22:33:40.553186 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.553123 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/04195c1d-9fb7-479b-8061-f92e3c88aed2-systemd-units\") pod \"ovnkube-node-khlhx\" (UID: \"04195c1d-9fb7-479b-8061-f92e3c88aed2\") " pod="openshift-ovn-kubernetes/ovnkube-node-khlhx" Oct 29 22:33:40.553186 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.553137 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f190a1a0-6f6a-44d0-aeaa-b053cff4bd9c-run\") pod \"tuned-jtlmz\" (UID: \"f190a1a0-6f6a-44d0-aeaa-b053cff4bd9c\") " pod="openshift-cluster-node-tuning-operator/tuned-jtlmz" Oct 29 22:33:40.553186 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.553157 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/04195c1d-9fb7-479b-8061-f92e3c88aed2-run-openvswitch\") pod \"ovnkube-node-khlhx\" (UID: \"04195c1d-9fb7-479b-8061-f92e3c88aed2\") " pod="openshift-ovn-kubernetes/ovnkube-node-khlhx" Oct 29 22:33:40.553452 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.553202 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/04195c1d-9fb7-479b-8061-f92e3c88aed2-run-openvswitch\") pod \"ovnkube-node-khlhx\" (UID: \"04195c1d-9fb7-479b-8061-f92e3c88aed2\") " pod="openshift-ovn-kubernetes/ovnkube-node-khlhx" Oct 29 22:33:40.553452 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.553202 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/04195c1d-9fb7-479b-8061-f92e3c88aed2-systemd-units\") pod \"ovnkube-node-khlhx\" (UID: \"04195c1d-9fb7-479b-8061-f92e3c88aed2\") " pod="openshift-ovn-kubernetes/ovnkube-node-khlhx" Oct 29 22:33:40.553452 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.553207 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b23027ca-0748-4125-9bc8-9d149a3418e0-etc-selinux\") pod \"aws-ebs-csi-driver-node-c96pg\" (UID: \"b23027ca-0748-4125-9bc8-9d149a3418e0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c96pg" Oct 29 22:33:40.553452 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.553234 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/04195c1d-9fb7-479b-8061-f92e3c88aed2-node-log\") pod \"ovnkube-node-khlhx\" (UID: \"04195c1d-9fb7-479b-8061-f92e3c88aed2\") " pod="openshift-ovn-kubernetes/ovnkube-node-khlhx" Oct 29 22:33:40.553452 ip-10-0-140-74 kubenswrapper[5057]: E1029 22:33:40.553232 5057 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 29 22:33:40.553452 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.553255 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2ab2aa22-99af-4451-8d33-9e3543c8d9cd-host-var-lib-cni-bin\") pod \"multus-v4q29\" (UID: \"2ab2aa22-99af-4451-8d33-9e3543c8d9cd\") " pod="openshift-multus/multus-v4q29" Oct 29 22:33:40.553452 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.553235 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f190a1a0-6f6a-44d0-aeaa-b053cff4bd9c-run\") pod \"tuned-jtlmz\" (UID: \"f190a1a0-6f6a-44d0-aeaa-b053cff4bd9c\") " pod="openshift-cluster-node-tuning-operator/tuned-jtlmz" Oct 29 22:33:40.553452 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.553280 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/04195c1d-9fb7-479b-8061-f92e3c88aed2-node-log\") pod \"ovnkube-node-khlhx\" (UID: \"04195c1d-9fb7-479b-8061-f92e3c88aed2\") " pod="openshift-ovn-kubernetes/ovnkube-node-khlhx" Oct 29 22:33:40.553452 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.553286 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/04ff31d0-a870-4527-a0c3-285737359f95-host-slash\") pod \"iptables-alerter-wpzm2\" (UID: \"04ff31d0-a870-4527-a0c3-285737359f95\") " pod="openshift-network-operator/iptables-alerter-wpzm2" Oct 29 22:33:40.553452 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.553313 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/04ff31d0-a870-4527-a0c3-285737359f95-host-slash\") pod \"iptables-alerter-wpzm2\" (UID: \"04ff31d0-a870-4527-a0c3-285737359f95\") " pod="openshift-network-operator/iptables-alerter-wpzm2" Oct 29 22:33:40.553452 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.553315 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f190a1a0-6f6a-44d0-aeaa-b053cff4bd9c-etc-sysctl-conf\") pod \"tuned-jtlmz\" (UID: \"f190a1a0-6f6a-44d0-aeaa-b053cff4bd9c\") " pod="openshift-cluster-node-tuning-operator/tuned-jtlmz" Oct 29 22:33:40.553452 ip-10-0-140-74 kubenswrapper[5057]: E1029 22:33:40.553356 5057 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71d80eec-b5e7-41d0-b58d-8a8d9a6c769c-metrics-certs podName:71d80eec-b5e7-41d0-b58d-8a8d9a6c769c nodeName:}" failed. No retries permitted until 2026-10-29 22:33:41.053321235 +0000 UTC m=+1.836534654 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/71d80eec-b5e7-41d0-b58d-8a8d9a6c769c-metrics-certs") pod "router-default-5d8d7cc579-8cwjp" (UID: "71d80eec-b5e7-41d0-b58d-8a8d9a6c769c") : secret "router-metrics-certs-default" not found Oct 29 22:33:40.553452 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.553382 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2ab2aa22-99af-4451-8d33-9e3543c8d9cd-multus-cni-dir\") pod \"multus-v4q29\" (UID: \"2ab2aa22-99af-4451-8d33-9e3543c8d9cd\") " pod="openshift-multus/multus-v4q29" Oct 29 22:33:40.553452 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.553411 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a53078b9-a6a8-423c-b852-34eececa7ab1-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-p2qv4\" (UID: \"a53078b9-a6a8-423c-b852-34eececa7ab1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-p2qv4" Oct 29 22:33:40.553452 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.553431 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f190a1a0-6f6a-44d0-aeaa-b053cff4bd9c-etc-sysctl-conf\") pod \"tuned-jtlmz\" (UID: \"f190a1a0-6f6a-44d0-aeaa-b053cff4bd9c\") " pod="openshift-cluster-node-tuning-operator/tuned-jtlmz" Oct 29 22:33:40.554107 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.553462 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0e1c2279-b6d4-44e0-84e6-94eb1206d2a0-hosts-file\") pod \"node-resolver-dpsd9\" (UID: \"0e1c2279-b6d4-44e0-84e6-94eb1206d2a0\") " pod="openshift-dns/node-resolver-dpsd9" Oct 29 22:33:40.554107 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.553480 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2ab2aa22-99af-4451-8d33-9e3543c8d9cd-multus-cni-dir\") pod \"multus-v4q29\" (UID: \"2ab2aa22-99af-4451-8d33-9e3543c8d9cd\") " pod="openshift-multus/multus-v4q29" Oct 29 22:33:40.554107 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.553487 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f8f92bfb-5cef-412e-a6c1-4bd793c59fe7-metrics-certs\") pod \"network-metrics-daemon-hzlmg\" (UID: \"f8f92bfb-5cef-412e-a6c1-4bd793c59fe7\") " pod="openshift-multus/network-metrics-daemon-hzlmg" Oct 29 22:33:40.554107 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.553516 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5296979a-7927-4ee2-84b9-a93780a2ad22-registry-tls\") pod \"image-registry-78db97464f-6ljzh\" (UID: \"5296979a-7927-4ee2-84b9-a93780a2ad22\") " pod="openshift-image-registry/image-registry-78db97464f-6ljzh" Oct 29 22:33:40.554107 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.553523 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0e1c2279-b6d4-44e0-84e6-94eb1206d2a0-hosts-file\") pod \"node-resolver-dpsd9\" (UID: \"0e1c2279-b6d4-44e0-84e6-94eb1206d2a0\") " pod="openshift-dns/node-resolver-dpsd9" Oct 29 22:33:40.554107 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.553540 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/04195c1d-9fb7-479b-8061-f92e3c88aed2-host-kubelet\") pod \"ovnkube-node-khlhx\" (UID: \"04195c1d-9fb7-479b-8061-f92e3c88aed2\") " pod="openshift-ovn-kubernetes/ovnkube-node-khlhx" Oct 29 22:33:40.554107 ip-10-0-140-74 kubenswrapper[5057]: E1029 22:33:40.553561 5057 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 29 22:33:40.554107 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.553591 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/04195c1d-9fb7-479b-8061-f92e3c88aed2-host-slash\") pod \"ovnkube-node-khlhx\" (UID: \"04195c1d-9fb7-479b-8061-f92e3c88aed2\") " pod="openshift-ovn-kubernetes/ovnkube-node-khlhx" Oct 29 22:33:40.554107 ip-10-0-140-74 kubenswrapper[5057]: E1029 22:33:40.553595 5057 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 29 22:33:40.554107 ip-10-0-140-74 kubenswrapper[5057]: E1029 22:33:40.553611 5057 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-78db97464f-6ljzh: secret "image-registry-tls" not found Oct 29 22:33:40.554107 ip-10-0-140-74 kubenswrapper[5057]: E1029 22:33:40.553616 5057 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f8f92bfb-5cef-412e-a6c1-4bd793c59fe7-metrics-certs podName:f8f92bfb-5cef-412e-a6c1-4bd793c59fe7 nodeName:}" failed. No retries permitted until 2026-10-29 22:33:41.053600715 +0000 UTC m=+1.836814122 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f8f92bfb-5cef-412e-a6c1-4bd793c59fe7-metrics-certs") pod "network-metrics-daemon-hzlmg" (UID: "f8f92bfb-5cef-412e-a6c1-4bd793c59fe7") : secret "metrics-daemon-secret" not found Oct 29 22:33:40.554107 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.553626 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/04195c1d-9fb7-479b-8061-f92e3c88aed2-host-kubelet\") pod \"ovnkube-node-khlhx\" (UID: \"04195c1d-9fb7-479b-8061-f92e3c88aed2\") " pod="openshift-ovn-kubernetes/ovnkube-node-khlhx" Oct 29 22:33:40.554107 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.553562 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/04195c1d-9fb7-479b-8061-f92e3c88aed2-host-slash\") pod \"ovnkube-node-khlhx\" (UID: \"04195c1d-9fb7-479b-8061-f92e3c88aed2\") " pod="openshift-ovn-kubernetes/ovnkube-node-khlhx" Oct 29 22:33:40.554107 ip-10-0-140-74 kubenswrapper[5057]: E1029 22:33:40.553655 5057 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5296979a-7927-4ee2-84b9-a93780a2ad22-registry-tls podName:5296979a-7927-4ee2-84b9-a93780a2ad22 nodeName:}" failed. No retries permitted until 2026-10-29 22:33:41.053648002 +0000 UTC m=+1.836861406 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5296979a-7927-4ee2-84b9-a93780a2ad22-registry-tls") pod "image-registry-78db97464f-6ljzh" (UID: "5296979a-7927-4ee2-84b9-a93780a2ad22") : secret "image-registry-tls" not found Oct 29 22:33:40.554107 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.553680 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/04195c1d-9fb7-479b-8061-f92e3c88aed2-var-lib-openvswitch\") pod \"ovnkube-node-khlhx\" (UID: \"04195c1d-9fb7-479b-8061-f92e3c88aed2\") " pod="openshift-ovn-kubernetes/ovnkube-node-khlhx" Oct 29 22:33:40.554107 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.553703 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f190a1a0-6f6a-44d0-aeaa-b053cff4bd9c-etc-kubernetes\") pod \"tuned-jtlmz\" (UID: \"f190a1a0-6f6a-44d0-aeaa-b053cff4bd9c\") " pod="openshift-cluster-node-tuning-operator/tuned-jtlmz" Oct 29 22:33:40.554107 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.553721 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8a972cc8-dce5-4f7c-b5d5-cb24ccf73bad-cnibin\") pod \"multus-additional-cni-plugins-llm78\" (UID: \"8a972cc8-dce5-4f7c-b5d5-cb24ccf73bad\") " pod="openshift-multus/multus-additional-cni-plugins-llm78" Oct 29 22:33:40.554107 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.553737 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2ab2aa22-99af-4451-8d33-9e3543c8d9cd-multus-conf-dir\") pod \"multus-v4q29\" (UID: \"2ab2aa22-99af-4451-8d33-9e3543c8d9cd\") " pod="openshift-multus/multus-v4q29" Oct 29 22:33:40.554107 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.553755 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/04195c1d-9fb7-479b-8061-f92e3c88aed2-var-lib-openvswitch\") pod \"ovnkube-node-khlhx\" (UID: \"04195c1d-9fb7-479b-8061-f92e3c88aed2\") " pod="openshift-ovn-kubernetes/ovnkube-node-khlhx" Oct 29 22:33:40.554107 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.553757 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/989956e8-1893-4bbe-a4bf-6a2081509080-registry-tls\") pod \"image-registry-84866f86bd-2v6d7\" (UID: \"989956e8-1893-4bbe-a4bf-6a2081509080\") " pod="openshift-image-registry/image-registry-84866f86bd-2v6d7" Oct 29 22:33:40.554107 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.553761 5057 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 29 22:33:40.554107 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.553784 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8a972cc8-dce5-4f7c-b5d5-cb24ccf73bad-cnibin\") pod \"multus-additional-cni-plugins-llm78\" (UID: \"8a972cc8-dce5-4f7c-b5d5-cb24ccf73bad\") " pod="openshift-multus/multus-additional-cni-plugins-llm78" Oct 29 22:33:40.554107 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.553790 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/04195c1d-9fb7-479b-8061-f92e3c88aed2-etc-openvswitch\") pod \"ovnkube-node-khlhx\" (UID: \"04195c1d-9fb7-479b-8061-f92e3c88aed2\") " pod="openshift-ovn-kubernetes/ovnkube-node-khlhx" Oct 29 22:33:40.554107 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.553799 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2ab2aa22-99af-4451-8d33-9e3543c8d9cd-multus-conf-dir\") pod \"multus-v4q29\" (UID: \"2ab2aa22-99af-4451-8d33-9e3543c8d9cd\") " pod="openshift-multus/multus-v4q29" Oct 29 22:33:40.554107 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.553815 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/04195c1d-9fb7-479b-8061-f92e3c88aed2-host-run-ovn-kubernetes\") pod \"ovnkube-node-khlhx\" (UID: \"04195c1d-9fb7-479b-8061-f92e3c88aed2\") " pod="openshift-ovn-kubernetes/ovnkube-node-khlhx" Oct 29 22:33:40.554107 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.553810 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f190a1a0-6f6a-44d0-aeaa-b053cff4bd9c-etc-kubernetes\") pod \"tuned-jtlmz\" (UID: \"f190a1a0-6f6a-44d0-aeaa-b053cff4bd9c\") " pod="openshift-cluster-node-tuning-operator/tuned-jtlmz" Oct 29 22:33:40.554107 ip-10-0-140-74 kubenswrapper[5057]: E1029 22:33:40.553833 5057 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 29 22:33:40.554107 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.553848 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/04195c1d-9fb7-479b-8061-f92e3c88aed2-etc-openvswitch\") pod \"ovnkube-node-khlhx\" (UID: \"04195c1d-9fb7-479b-8061-f92e3c88aed2\") " pod="openshift-ovn-kubernetes/ovnkube-node-khlhx" Oct 29 22:33:40.554107 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.553849 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/04195c1d-9fb7-479b-8061-f92e3c88aed2-host-run-ovn-kubernetes\") pod \"ovnkube-node-khlhx\" (UID: \"04195c1d-9fb7-479b-8061-f92e3c88aed2\") " pod="openshift-ovn-kubernetes/ovnkube-node-khlhx" Oct 29 22:33:40.554107 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.553853 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/71d80eec-b5e7-41d0-b58d-8a8d9a6c769c-service-ca-bundle\") pod \"router-default-5d8d7cc579-8cwjp\" (UID: \"71d80eec-b5e7-41d0-b58d-8a8d9a6c769c\") " pod="openshift-ingress/router-default-5d8d7cc579-8cwjp" Oct 29 22:33:40.554107 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.553900 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/93dbae94-f92d-4157-a89b-d3c984a56013-host\") pod \"node-ca-pcjl7\" (UID: \"93dbae94-f92d-4157-a89b-d3c984a56013\") " pod="openshift-image-registry/node-ca-pcjl7" Oct 29 22:33:40.554107 ip-10-0-140-74 kubenswrapper[5057]: E1029 22:33:40.553858 5057 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-84866f86bd-2v6d7: secret "image-registry-tls" not found Oct 29 22:33:40.554107 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.553927 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b23027ca-0748-4125-9bc8-9d149a3418e0-device-dir\") pod \"aws-ebs-csi-driver-node-c96pg\" (UID: \"b23027ca-0748-4125-9bc8-9d149a3418e0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c96pg" Oct 29 22:33:40.554107 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.553935 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/93dbae94-f92d-4157-a89b-d3c984a56013-host\") pod \"node-ca-pcjl7\" (UID: \"93dbae94-f92d-4157-a89b-d3c984a56013\") " pod="openshift-image-registry/node-ca-pcjl7" Oct 29 22:33:40.554107 ip-10-0-140-74 kubenswrapper[5057]: E1029 22:33:40.553940 5057 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/71d80eec-b5e7-41d0-b58d-8a8d9a6c769c-service-ca-bundle podName:71d80eec-b5e7-41d0-b58d-8a8d9a6c769c nodeName:}" failed. No retries permitted until 2026-10-29 22:33:41.05392586 +0000 UTC m=+1.837139289 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/71d80eec-b5e7-41d0-b58d-8a8d9a6c769c-service-ca-bundle") pod "router-default-5d8d7cc579-8cwjp" (UID: "71d80eec-b5e7-41d0-b58d-8a8d9a6c769c") : configmap references non-existent config key: service-ca.crt Oct 29 22:33:40.554107 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.553962 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b23027ca-0748-4125-9bc8-9d149a3418e0-device-dir\") pod \"aws-ebs-csi-driver-node-c96pg\" (UID: \"b23027ca-0748-4125-9bc8-9d149a3418e0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c96pg" Oct 29 22:33:40.554107 ip-10-0-140-74 kubenswrapper[5057]: E1029 22:33:40.553991 5057 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/989956e8-1893-4bbe-a4bf-6a2081509080-registry-tls podName:989956e8-1893-4bbe-a4bf-6a2081509080 nodeName:}" failed. No retries permitted until 2026-10-29 22:33:41.053959411 +0000 UTC m=+1.837172815 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/989956e8-1893-4bbe-a4bf-6a2081509080-registry-tls") pod "image-registry-84866f86bd-2v6d7" (UID: "989956e8-1893-4bbe-a4bf-6a2081509080") : secret "image-registry-tls" not found Oct 29 22:33:40.554107 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.554019 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/079ed4e4-bdc4-405c-90a1-a22b84fcf948-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-d2dnn\" (UID: \"079ed4e4-bdc4-405c-90a1-a22b84fcf948\") " pod="openshift-insights/insights-runtime-extractor-d2dnn" Oct 29 22:33:40.554107 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.554035 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f190a1a0-6f6a-44d0-aeaa-b053cff4bd9c-host\") pod \"tuned-jtlmz\" (UID: \"f190a1a0-6f6a-44d0-aeaa-b053cff4bd9c\") " pod="openshift-cluster-node-tuning-operator/tuned-jtlmz" Oct 29 22:33:40.554107 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.554063 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/079ed4e4-bdc4-405c-90a1-a22b84fcf948-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-d2dnn\" (UID: \"079ed4e4-bdc4-405c-90a1-a22b84fcf948\") " pod="openshift-insights/insights-runtime-extractor-d2dnn" Oct 29 22:33:40.554107 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.554092 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/641f9ec1-c8e2-4774-9d4f-6a7a2da97b6f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-8c5hr\" (UID: \"641f9ec1-c8e2-4774-9d4f-6a7a2da97b6f\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-8c5hr" Oct 29 22:33:40.555919 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.554140 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f190a1a0-6f6a-44d0-aeaa-b053cff4bd9c-host\") pod \"tuned-jtlmz\" (UID: \"f190a1a0-6f6a-44d0-aeaa-b053cff4bd9c\") " pod="openshift-cluster-node-tuning-operator/tuned-jtlmz" Oct 29 22:33:40.555919 ip-10-0-140-74 kubenswrapper[5057]: E1029 22:33:40.554166 5057 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 29 22:33:40.555919 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.554174 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f190a1a0-6f6a-44d0-aeaa-b053cff4bd9c-lib-modules\") pod \"tuned-jtlmz\" (UID: \"f190a1a0-6f6a-44d0-aeaa-b053cff4bd9c\") " pod="openshift-cluster-node-tuning-operator/tuned-jtlmz" Oct 29 22:33:40.555919 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.554204 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2ab2aa22-99af-4451-8d33-9e3543c8d9cd-host-run-k8s-cni-cncf-io\") pod \"multus-v4q29\" (UID: \"2ab2aa22-99af-4451-8d33-9e3543c8d9cd\") " pod="openshift-multus/multus-v4q29" Oct 29 22:33:40.555919 ip-10-0-140-74 kubenswrapper[5057]: E1029 22:33:40.554216 5057 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/079ed4e4-bdc4-405c-90a1-a22b84fcf948-insights-runtime-extractor-tls podName:079ed4e4-bdc4-405c-90a1-a22b84fcf948 nodeName:}" failed. No retries permitted until 2026-10-29 22:33:41.054203135 +0000 UTC m=+1.837416549 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/079ed4e4-bdc4-405c-90a1-a22b84fcf948-insights-runtime-extractor-tls") pod "insights-runtime-extractor-d2dnn" (UID: "079ed4e4-bdc4-405c-90a1-a22b84fcf948") : secret "insights-runtime-extractor-tls" not found Oct 29 22:33:40.555919 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.554244 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/04195c1d-9fb7-479b-8061-f92e3c88aed2-run-systemd\") pod \"ovnkube-node-khlhx\" (UID: \"04195c1d-9fb7-479b-8061-f92e3c88aed2\") " pod="openshift-ovn-kubernetes/ovnkube-node-khlhx" Oct 29 22:33:40.555919 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.554253 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2ab2aa22-99af-4451-8d33-9e3543c8d9cd-host-run-k8s-cni-cncf-io\") pod \"multus-v4q29\" (UID: \"2ab2aa22-99af-4451-8d33-9e3543c8d9cd\") " pod="openshift-multus/multus-v4q29" Oct 29 22:33:40.555919 ip-10-0-140-74 kubenswrapper[5057]: E1029 22:33:40.554166 5057 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 29 22:33:40.555919 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.554283 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/04195c1d-9fb7-479b-8061-f92e3c88aed2-run-systemd\") pod \"ovnkube-node-khlhx\" (UID: \"04195c1d-9fb7-479b-8061-f92e3c88aed2\") " pod="openshift-ovn-kubernetes/ovnkube-node-khlhx" Oct 29 22:33:40.555919 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.554267 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2ab2aa22-99af-4451-8d33-9e3543c8d9cd-system-cni-dir\") pod \"multus-v4q29\" (UID: \"2ab2aa22-99af-4451-8d33-9e3543c8d9cd\") " pod="openshift-multus/multus-v4q29" Oct 29 22:33:40.555919 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.554308 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2ab2aa22-99af-4451-8d33-9e3543c8d9cd-system-cni-dir\") pod \"multus-v4q29\" (UID: \"2ab2aa22-99af-4451-8d33-9e3543c8d9cd\") " pod="openshift-multus/multus-v4q29" Oct 29 22:33:40.555919 ip-10-0-140-74 kubenswrapper[5057]: E1029 22:33:40.554314 5057 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/641f9ec1-c8e2-4774-9d4f-6a7a2da97b6f-cluster-monitoring-operator-tls podName:641f9ec1-c8e2-4774-9d4f-6a7a2da97b6f nodeName:}" failed. No retries permitted until 2026-10-29 22:33:41.054299753 +0000 UTC m=+1.837513166 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/641f9ec1-c8e2-4774-9d4f-6a7a2da97b6f-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-8c5hr" (UID: "641f9ec1-c8e2-4774-9d4f-6a7a2da97b6f") : secret "cluster-monitoring-operator-tls" not found Oct 29 22:33:40.555919 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.554325 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f190a1a0-6f6a-44d0-aeaa-b053cff4bd9c-lib-modules\") pod \"tuned-jtlmz\" (UID: \"f190a1a0-6f6a-44d0-aeaa-b053cff4bd9c\") " pod="openshift-cluster-node-tuning-operator/tuned-jtlmz" Oct 29 22:33:40.555919 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.554376 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b23027ca-0748-4125-9bc8-9d149a3418e0-kubelet-dir\") pod \"aws-ebs-csi-driver-node-c96pg\" (UID: \"b23027ca-0748-4125-9bc8-9d149a3418e0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c96pg" Oct 29 22:33:40.555919 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.554413 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b23027ca-0748-4125-9bc8-9d149a3418e0-sys-fs\") pod \"aws-ebs-csi-driver-node-c96pg\" (UID: \"b23027ca-0748-4125-9bc8-9d149a3418e0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c96pg" Oct 29 22:33:40.555919 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.554440 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2ab2aa22-99af-4451-8d33-9e3543c8d9cd-host-var-lib-cni-multus\") pod \"multus-v4q29\" (UID: \"2ab2aa22-99af-4451-8d33-9e3543c8d9cd\") " pod="openshift-multus/multus-v4q29" Oct 29 22:33:40.555919 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.554455 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b23027ca-0748-4125-9bc8-9d149a3418e0-kubelet-dir\") pod \"aws-ebs-csi-driver-node-c96pg\" (UID: \"b23027ca-0748-4125-9bc8-9d149a3418e0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c96pg" Oct 29 22:33:40.555919 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.554463 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2ab2aa22-99af-4451-8d33-9e3543c8d9cd-hostroot\") pod \"multus-v4q29\" (UID: \"2ab2aa22-99af-4451-8d33-9e3543c8d9cd\") " pod="openshift-multus/multus-v4q29" Oct 29 22:33:40.555919 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.554496 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/04195c1d-9fb7-479b-8061-f92e3c88aed2-log-socket\") pod \"ovnkube-node-khlhx\" (UID: \"04195c1d-9fb7-479b-8061-f92e3c88aed2\") " pod="openshift-ovn-kubernetes/ovnkube-node-khlhx" Oct 29 22:33:40.555919 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.554501 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2ab2aa22-99af-4451-8d33-9e3543c8d9cd-hostroot\") pod \"multus-v4q29\" (UID: \"2ab2aa22-99af-4451-8d33-9e3543c8d9cd\") " pod="openshift-multus/multus-v4q29" Oct 29 22:33:40.555919 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.554528 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2ab2aa22-99af-4451-8d33-9e3543c8d9cd-host-var-lib-cni-multus\") pod \"multus-v4q29\" (UID: \"2ab2aa22-99af-4451-8d33-9e3543c8d9cd\") " pod="openshift-multus/multus-v4q29" Oct 29 22:33:40.555919 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.554545 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2ab2aa22-99af-4451-8d33-9e3543c8d9cd-host-run-netns\") pod \"multus-v4q29\" (UID: \"2ab2aa22-99af-4451-8d33-9e3543c8d9cd\") " pod="openshift-multus/multus-v4q29" Oct 29 22:33:40.555919 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.554530 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b23027ca-0748-4125-9bc8-9d149a3418e0-sys-fs\") pod \"aws-ebs-csi-driver-node-c96pg\" (UID: \"b23027ca-0748-4125-9bc8-9d149a3418e0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c96pg" Oct 29 22:33:40.555919 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.554568 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2ab2aa22-99af-4451-8d33-9e3543c8d9cd-host-run-multus-certs\") pod \"multus-v4q29\" (UID: \"2ab2aa22-99af-4451-8d33-9e3543c8d9cd\") " pod="openshift-multus/multus-v4q29" Oct 29 22:33:40.555919 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.554575 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/04195c1d-9fb7-479b-8061-f92e3c88aed2-log-socket\") pod \"ovnkube-node-khlhx\" (UID: \"04195c1d-9fb7-479b-8061-f92e3c88aed2\") " pod="openshift-ovn-kubernetes/ovnkube-node-khlhx" Oct 29 22:33:40.555919 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.554603 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b23027ca-0748-4125-9bc8-9d149a3418e0-registration-dir\") pod \"aws-ebs-csi-driver-node-c96pg\" (UID: \"b23027ca-0748-4125-9bc8-9d149a3418e0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c96pg" Oct 29 22:33:40.555919 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.554615 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2ab2aa22-99af-4451-8d33-9e3543c8d9cd-host-run-netns\") pod \"multus-v4q29\" (UID: \"2ab2aa22-99af-4451-8d33-9e3543c8d9cd\") " pod="openshift-multus/multus-v4q29" Oct 29 22:33:40.555919 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.554639 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/079ed4e4-bdc4-405c-90a1-a22b84fcf948-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-d2dnn\" (UID: \"079ed4e4-bdc4-405c-90a1-a22b84fcf948\") " pod="openshift-insights/insights-runtime-extractor-d2dnn" Oct 29 22:33:40.555919 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.554655 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2ab2aa22-99af-4451-8d33-9e3543c8d9cd-multus-socket-dir-parent\") pod \"multus-v4q29\" (UID: \"2ab2aa22-99af-4451-8d33-9e3543c8d9cd\") " pod="openshift-multus/multus-v4q29" Oct 29 22:33:40.555919 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.554698 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f190a1a0-6f6a-44d0-aeaa-b053cff4bd9c-etc-sysconfig\") pod \"tuned-jtlmz\" (UID: \"f190a1a0-6f6a-44d0-aeaa-b053cff4bd9c\") " pod="openshift-cluster-node-tuning-operator/tuned-jtlmz" Oct 29 22:33:40.555919 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.554703 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b23027ca-0748-4125-9bc8-9d149a3418e0-registration-dir\") pod \"aws-ebs-csi-driver-node-c96pg\" (UID: \"b23027ca-0748-4125-9bc8-9d149a3418e0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c96pg" Oct 29 22:33:40.555919 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.554703 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2ab2aa22-99af-4451-8d33-9e3543c8d9cd-multus-socket-dir-parent\") pod \"multus-v4q29\" (UID: \"2ab2aa22-99af-4451-8d33-9e3543c8d9cd\") " pod="openshift-multus/multus-v4q29" Oct 29 22:33:40.555919 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.554661 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2ab2aa22-99af-4451-8d33-9e3543c8d9cd-host-run-multus-certs\") pod \"multus-v4q29\" (UID: \"2ab2aa22-99af-4451-8d33-9e3543c8d9cd\") " pod="openshift-multus/multus-v4q29" Oct 29 22:33:40.555919 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.554723 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f190a1a0-6f6a-44d0-aeaa-b053cff4bd9c-etc-sysctl-d\") pod \"tuned-jtlmz\" (UID: \"f190a1a0-6f6a-44d0-aeaa-b053cff4bd9c\") " pod="openshift-cluster-node-tuning-operator/tuned-jtlmz" Oct 29 22:33:40.555919 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.554753 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f190a1a0-6f6a-44d0-aeaa-b053cff4bd9c-etc-sysconfig\") pod \"tuned-jtlmz\" (UID: \"f190a1a0-6f6a-44d0-aeaa-b053cff4bd9c\") " pod="openshift-cluster-node-tuning-operator/tuned-jtlmz" Oct 29 22:33:40.555919 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.554755 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b23027ca-0748-4125-9bc8-9d149a3418e0-socket-dir\") pod \"aws-ebs-csi-driver-node-c96pg\" (UID: \"b23027ca-0748-4125-9bc8-9d149a3418e0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c96pg" Oct 29 22:33:40.555919 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.554774 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/079ed4e4-bdc4-405c-90a1-a22b84fcf948-crio-socket\") pod \"insights-runtime-extractor-d2dnn\" (UID: \"079ed4e4-bdc4-405c-90a1-a22b84fcf948\") " pod="openshift-insights/insights-runtime-extractor-d2dnn" Oct 29 22:33:40.555919 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.554788 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f190a1a0-6f6a-44d0-aeaa-b053cff4bd9c-etc-systemd\") pod \"tuned-jtlmz\" (UID: \"f190a1a0-6f6a-44d0-aeaa-b053cff4bd9c\") " pod="openshift-cluster-node-tuning-operator/tuned-jtlmz" Oct 29 22:33:40.555919 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.554800 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b23027ca-0748-4125-9bc8-9d149a3418e0-socket-dir\") pod \"aws-ebs-csi-driver-node-c96pg\" (UID: \"b23027ca-0748-4125-9bc8-9d149a3418e0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c96pg" Oct 29 22:33:40.555919 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.554809 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/04195c1d-9fb7-479b-8061-f92e3c88aed2-host-cni-netd\") pod \"ovnkube-node-khlhx\" (UID: \"04195c1d-9fb7-479b-8061-f92e3c88aed2\") " pod="openshift-ovn-kubernetes/ovnkube-node-khlhx" Oct 29 22:33:40.555919 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.554834 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f190a1a0-6f6a-44d0-aeaa-b053cff4bd9c-etc-sysctl-d\") pod \"tuned-jtlmz\" (UID: \"f190a1a0-6f6a-44d0-aeaa-b053cff4bd9c\") " pod="openshift-cluster-node-tuning-operator/tuned-jtlmz" Oct 29 22:33:40.555919 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.554825 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6b34b3b1-84af-45f3-92ee-1ca720e8b263-metrics-tls\") pod \"dns-default-jjz47\" (UID: \"6b34b3b1-84af-45f3-92ee-1ca720e8b263\") " pod="openshift-dns/dns-default-jjz47" Oct 29 22:33:40.555919 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.554862 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f190a1a0-6f6a-44d0-aeaa-b053cff4bd9c-etc-systemd\") pod \"tuned-jtlmz\" (UID: \"f190a1a0-6f6a-44d0-aeaa-b053cff4bd9c\") " pod="openshift-cluster-node-tuning-operator/tuned-jtlmz" Oct 29 22:33:40.555919 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.554884 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f190a1a0-6f6a-44d0-aeaa-b053cff4bd9c-etc-modprobe-d\") pod \"tuned-jtlmz\" (UID: \"f190a1a0-6f6a-44d0-aeaa-b053cff4bd9c\") " pod="openshift-cluster-node-tuning-operator/tuned-jtlmz" Oct 29 22:33:40.555919 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.554889 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/04195c1d-9fb7-479b-8061-f92e3c88aed2-host-cni-netd\") pod \"ovnkube-node-khlhx\" (UID: \"04195c1d-9fb7-479b-8061-f92e3c88aed2\") " pod="openshift-ovn-kubernetes/ovnkube-node-khlhx" Oct 29 22:33:40.555919 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.554843 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/079ed4e4-bdc4-405c-90a1-a22b84fcf948-crio-socket\") pod \"insights-runtime-extractor-d2dnn\" (UID: \"079ed4e4-bdc4-405c-90a1-a22b84fcf948\") " pod="openshift-insights/insights-runtime-extractor-d2dnn" Oct 29 22:33:40.555919 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.554907 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f190a1a0-6f6a-44d0-aeaa-b053cff4bd9c-var-lib-kubelet\") pod \"tuned-jtlmz\" (UID: \"f190a1a0-6f6a-44d0-aeaa-b053cff4bd9c\") " pod="openshift-cluster-node-tuning-operator/tuned-jtlmz" Oct 29 22:33:40.555919 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.554939 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/04195c1d-9fb7-479b-8061-f92e3c88aed2-run-ovn\") pod \"ovnkube-node-khlhx\" (UID: \"04195c1d-9fb7-479b-8061-f92e3c88aed2\") " pod="openshift-ovn-kubernetes/ovnkube-node-khlhx" Oct 29 22:33:40.555919 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.554951 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f190a1a0-6f6a-44d0-aeaa-b053cff4bd9c-var-lib-kubelet\") pod \"tuned-jtlmz\" (UID: \"f190a1a0-6f6a-44d0-aeaa-b053cff4bd9c\") " pod="openshift-cluster-node-tuning-operator/tuned-jtlmz" Oct 29 22:33:40.555919 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.554964 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2accd775-2173-4bfe-a413-7d243bd0820f-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-6rr4f\" (UID: \"2accd775-2173-4bfe-a413-7d243bd0820f\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-6rr4f" Oct 29 22:33:40.555919 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.554973 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f190a1a0-6f6a-44d0-aeaa-b053cff4bd9c-etc-modprobe-d\") pod \"tuned-jtlmz\" (UID: \"f190a1a0-6f6a-44d0-aeaa-b053cff4bd9c\") " pod="openshift-cluster-node-tuning-operator/tuned-jtlmz" Oct 29 22:33:40.555919 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.554988 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/04195c1d-9fb7-479b-8061-f92e3c88aed2-run-ovn\") pod \"ovnkube-node-khlhx\" (UID: \"04195c1d-9fb7-479b-8061-f92e3c88aed2\") " pod="openshift-ovn-kubernetes/ovnkube-node-khlhx" Oct 29 22:33:40.555919 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.554991 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/04195c1d-9fb7-479b-8061-f92e3c88aed2-host-cni-bin\") pod \"ovnkube-node-khlhx\" (UID: \"04195c1d-9fb7-479b-8061-f92e3c88aed2\") " pod="openshift-ovn-kubernetes/ovnkube-node-khlhx" Oct 29 22:33:40.555919 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.555021 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8a972cc8-dce5-4f7c-b5d5-cb24ccf73bad-tuning-conf-dir\") pod \"multus-additional-cni-plugins-llm78\" (UID: \"8a972cc8-dce5-4f7c-b5d5-cb24ccf73bad\") " pod="openshift-multus/multus-additional-cni-plugins-llm78" Oct 29 22:33:40.555919 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.555026 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/04195c1d-9fb7-479b-8061-f92e3c88aed2-host-cni-bin\") pod \"ovnkube-node-khlhx\" (UID: \"04195c1d-9fb7-479b-8061-f92e3c88aed2\") " pod="openshift-ovn-kubernetes/ovnkube-node-khlhx" Oct 29 22:33:40.555919 ip-10-0-140-74 kubenswrapper[5057]: E1029 22:33:40.555030 5057 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 29 22:33:40.555919 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.555072 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8a972cc8-dce5-4f7c-b5d5-cb24ccf73bad-tuning-conf-dir\") pod \"multus-additional-cni-plugins-llm78\" (UID: \"8a972cc8-dce5-4f7c-b5d5-cb24ccf73bad\") " pod="openshift-multus/multus-additional-cni-plugins-llm78" Oct 29 22:33:40.555919 ip-10-0-140-74 kubenswrapper[5057]: E1029 22:33:40.555095 5057 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2accd775-2173-4bfe-a413-7d243bd0820f-networking-console-plugin-cert podName:2accd775-2173-4bfe-a413-7d243bd0820f nodeName:}" failed. No retries permitted until 2026-10-29 22:33:41.055081478 +0000 UTC m=+1.838294891 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/2accd775-2173-4bfe-a413-7d243bd0820f-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-6rr4f" (UID: "2accd775-2173-4bfe-a413-7d243bd0820f") : secret "networking-console-plugin-cert" not found Oct 29 22:33:40.555919 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.555114 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2ab2aa22-99af-4451-8d33-9e3543c8d9cd-host-var-lib-kubelet\") pod \"multus-v4q29\" (UID: \"2ab2aa22-99af-4451-8d33-9e3543c8d9cd\") " pod="openshift-multus/multus-v4q29" Oct 29 22:33:40.555919 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.555141 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2ab2aa22-99af-4451-8d33-9e3543c8d9cd-etc-kubernetes\") pod \"multus-v4q29\" (UID: \"2ab2aa22-99af-4451-8d33-9e3543c8d9cd\") " pod="openshift-multus/multus-v4q29" Oct 29 22:33:40.555919 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.555163 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/04195c1d-9fb7-479b-8061-f92e3c88aed2-host-run-netns\") pod \"ovnkube-node-khlhx\" (UID: \"04195c1d-9fb7-479b-8061-f92e3c88aed2\") " pod="openshift-ovn-kubernetes/ovnkube-node-khlhx" Oct 29 22:33:40.555919 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.555177 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2ab2aa22-99af-4451-8d33-9e3543c8d9cd-etc-kubernetes\") pod \"multus-v4q29\" (UID: \"2ab2aa22-99af-4451-8d33-9e3543c8d9cd\") " pod="openshift-multus/multus-v4q29" Oct 29 22:33:40.555919 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.555180 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2ab2aa22-99af-4451-8d33-9e3543c8d9cd-host-var-lib-kubelet\") pod \"multus-v4q29\" (UID: \"2ab2aa22-99af-4451-8d33-9e3543c8d9cd\") " pod="openshift-multus/multus-v4q29" Oct 29 22:33:40.555919 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.555201 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2ab2aa22-99af-4451-8d33-9e3543c8d9cd-cnibin\") pod \"multus-v4q29\" (UID: \"2ab2aa22-99af-4451-8d33-9e3543c8d9cd\") " pod="openshift-multus/multus-v4q29" Oct 29 22:33:40.555919 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.555225 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6f50ee57-a710-4325-bd45-4f861452ede4-dbus\") pod \"global-pull-secret-syncer-bp46g\" (UID: \"6f50ee57-a710-4325-bd45-4f861452ede4\") " pod="kube-system/global-pull-secret-syncer-bp46g" Oct 29 22:33:40.555919 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.555232 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/04195c1d-9fb7-479b-8061-f92e3c88aed2-host-run-netns\") pod \"ovnkube-node-khlhx\" (UID: \"04195c1d-9fb7-479b-8061-f92e3c88aed2\") " pod="openshift-ovn-kubernetes/ovnkube-node-khlhx" Oct 29 22:33:40.555919 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.555250 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8a972cc8-dce5-4f7c-b5d5-cb24ccf73bad-system-cni-dir\") pod \"multus-additional-cni-plugins-llm78\" (UID: \"8a972cc8-dce5-4f7c-b5d5-cb24ccf73bad\") " pod="openshift-multus/multus-additional-cni-plugins-llm78" Oct 29 22:33:40.555919 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.555257 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2ab2aa22-99af-4451-8d33-9e3543c8d9cd-cnibin\") pod \"multus-v4q29\" (UID: \"2ab2aa22-99af-4451-8d33-9e3543c8d9cd\") " pod="openshift-multus/multus-v4q29" Oct 29 22:33:40.555919 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.555277 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6f50ee57-a710-4325-bd45-4f861452ede4-kubelet-config\") pod \"global-pull-secret-syncer-bp46g\" (UID: \"6f50ee57-a710-4325-bd45-4f861452ede4\") " pod="kube-system/global-pull-secret-syncer-bp46g" Oct 29 22:33:40.555919 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.555283 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6f50ee57-a710-4325-bd45-4f861452ede4-dbus\") pod \"global-pull-secret-syncer-bp46g\" (UID: \"6f50ee57-a710-4325-bd45-4f861452ede4\") " pod="kube-system/global-pull-secret-syncer-bp46g" Oct 29 22:33:40.555919 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.555297 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8a972cc8-dce5-4f7c-b5d5-cb24ccf73bad-system-cni-dir\") pod \"multus-additional-cni-plugins-llm78\" (UID: \"8a972cc8-dce5-4f7c-b5d5-cb24ccf73bad\") " pod="openshift-multus/multus-additional-cni-plugins-llm78" Oct 29 22:33:40.555919 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.555304 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/04195c1d-9fb7-479b-8061-f92e3c88aed2-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-khlhx\" (UID: \"04195c1d-9fb7-479b-8061-f92e3c88aed2\") " pod="openshift-ovn-kubernetes/ovnkube-node-khlhx" Oct 29 22:33:40.555919 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.555341 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6f50ee57-a710-4325-bd45-4f861452ede4-kubelet-config\") pod \"global-pull-secret-syncer-bp46g\" (UID: \"6f50ee57-a710-4325-bd45-4f861452ede4\") " pod="kube-system/global-pull-secret-syncer-bp46g" Oct 29 22:33:40.555919 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.555352 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f190a1a0-6f6a-44d0-aeaa-b053cff4bd9c-sys\") pod \"tuned-jtlmz\" (UID: \"f190a1a0-6f6a-44d0-aeaa-b053cff4bd9c\") " pod="openshift-cluster-node-tuning-operator/tuned-jtlmz" Oct 29 22:33:40.555919 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.555370 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/04195c1d-9fb7-479b-8061-f92e3c88aed2-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-khlhx\" (UID: \"04195c1d-9fb7-479b-8061-f92e3c88aed2\") " pod="openshift-ovn-kubernetes/ovnkube-node-khlhx" Oct 29 22:33:40.555919 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.555439 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f190a1a0-6f6a-44d0-aeaa-b053cff4bd9c-sys\") pod \"tuned-jtlmz\" (UID: \"f190a1a0-6f6a-44d0-aeaa-b053cff4bd9c\") " pod="openshift-cluster-node-tuning-operator/tuned-jtlmz" Oct 29 22:33:40.559531 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.556383 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a53078b9-a6a8-423c-b852-34eececa7ab1-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-p2qv4\" (UID: \"a53078b9-a6a8-423c-b852-34eececa7ab1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-p2qv4" Oct 29 22:33:40.559531 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.557043 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6b34b3b1-84af-45f3-92ee-1ca720e8b263-metrics-tls\") pod \"dns-default-jjz47\" (UID: \"6b34b3b1-84af-45f3-92ee-1ca720e8b263\") " pod="openshift-dns/dns-default-jjz47" Oct 29 22:33:40.630396 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.630375 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-p2qv4" Oct 29 22:33:40.631517 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.631500 5057 scope.go:122] "RemoveContainer" containerID="e5ab727c17d3a45185a300b3b2a3688775b919a83c6e93f3a510f87ee56a0fb7" Oct 29 22:33:40.643889 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.643866 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-jjz47" Oct 29 22:33:40.648463 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.648440 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:33:40.759087 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.759057 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:33:40.957657 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.957628 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-p2qv4"] Oct 29 22:33:40.987777 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.987723 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-jjz47"] Oct 29 22:33:40.991638 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:40.991612 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b34b3b1_84af_45f3_92ee_1ca720e8b263.slice/crio-3665792a7a041193ffcb2881b0f0a576b7043fd51efe89f7905b783e68c9b293 WatchSource:0}: Error finding container 3665792a7a041193ffcb2881b0f0a576b7043fd51efe89f7905b783e68c9b293: Status 404 returned error can't find the container with id 3665792a7a041193ffcb2881b0f0a576b7043fd51efe89f7905b783e68c9b293 Oct 29 22:33:40.992368 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:40.992352 5057 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 29 22:33:41.058526 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:41.058471 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/079ed4e4-bdc4-405c-90a1-a22b84fcf948-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-d2dnn\" (UID: \"079ed4e4-bdc4-405c-90a1-a22b84fcf948\") " pod="openshift-insights/insights-runtime-extractor-d2dnn" Oct 29 22:33:41.058526 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:41.058503 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/641f9ec1-c8e2-4774-9d4f-6a7a2da97b6f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-8c5hr\" (UID: \"641f9ec1-c8e2-4774-9d4f-6a7a2da97b6f\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-8c5hr" Oct 29 22:33:41.058637 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:41.058538 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2accd775-2173-4bfe-a413-7d243bd0820f-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-6rr4f\" (UID: \"2accd775-2173-4bfe-a413-7d243bd0820f\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-6rr4f" Oct 29 22:33:41.058637 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:41.058576 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71d80eec-b5e7-41d0-b58d-8a8d9a6c769c-metrics-certs\") pod \"router-default-5d8d7cc579-8cwjp\" (UID: \"71d80eec-b5e7-41d0-b58d-8a8d9a6c769c\") " pod="openshift-ingress/router-default-5d8d7cc579-8cwjp" Oct 29 22:33:41.058637 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:41.058603 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f8f92bfb-5cef-412e-a6c1-4bd793c59fe7-metrics-certs\") pod \"network-metrics-daemon-hzlmg\" (UID: \"f8f92bfb-5cef-412e-a6c1-4bd793c59fe7\") " pod="openshift-multus/network-metrics-daemon-hzlmg" Oct 29 22:33:41.058637 ip-10-0-140-74 kubenswrapper[5057]: E1029 22:33:41.058620 5057 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 29 22:33:41.058815 ip-10-0-140-74 kubenswrapper[5057]: E1029 22:33:41.058644 5057 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 29 22:33:41.058815 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:41.058624 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5296979a-7927-4ee2-84b9-a93780a2ad22-registry-tls\") pod \"image-registry-78db97464f-6ljzh\" (UID: \"5296979a-7927-4ee2-84b9-a93780a2ad22\") " pod="openshift-image-registry/image-registry-78db97464f-6ljzh" Oct 29 22:33:41.058815 ip-10-0-140-74 kubenswrapper[5057]: E1029 22:33:41.058681 5057 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/641f9ec1-c8e2-4774-9d4f-6a7a2da97b6f-cluster-monitoring-operator-tls podName:641f9ec1-c8e2-4774-9d4f-6a7a2da97b6f nodeName:}" failed. No retries permitted until 2026-10-29 22:33:42.058662514 +0000 UTC m=+2.841875941 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/641f9ec1-c8e2-4774-9d4f-6a7a2da97b6f-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-8c5hr" (UID: "641f9ec1-c8e2-4774-9d4f-6a7a2da97b6f") : secret "cluster-monitoring-operator-tls" not found Oct 29 22:33:41.058815 ip-10-0-140-74 kubenswrapper[5057]: E1029 22:33:41.058688 5057 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 29 22:33:41.058815 ip-10-0-140-74 kubenswrapper[5057]: E1029 22:33:41.058692 5057 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 29 22:33:41.058815 ip-10-0-140-74 kubenswrapper[5057]: E1029 22:33:41.058708 5057 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-78db97464f-6ljzh: secret "image-registry-tls" not found Oct 29 22:33:41.058815 ip-10-0-140-74 kubenswrapper[5057]: E1029 22:33:41.058727 5057 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 29 22:33:41.058815 ip-10-0-140-74 kubenswrapper[5057]: E1029 22:33:41.058740 5057 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2accd775-2173-4bfe-a413-7d243bd0820f-networking-console-plugin-cert podName:2accd775-2173-4bfe-a413-7d243bd0820f nodeName:}" failed. No retries permitted until 2026-10-29 22:33:42.058707151 +0000 UTC m=+2.841920565 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/2accd775-2173-4bfe-a413-7d243bd0820f-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-6rr4f" (UID: "2accd775-2173-4bfe-a413-7d243bd0820f") : secret "networking-console-plugin-cert" not found Oct 29 22:33:41.058815 ip-10-0-140-74 kubenswrapper[5057]: E1029 22:33:41.058765 5057 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/079ed4e4-bdc4-405c-90a1-a22b84fcf948-insights-runtime-extractor-tls podName:079ed4e4-bdc4-405c-90a1-a22b84fcf948 nodeName:}" failed. No retries permitted until 2026-10-29 22:33:42.058754472 +0000 UTC m=+2.841967875 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/079ed4e4-bdc4-405c-90a1-a22b84fcf948-insights-runtime-extractor-tls") pod "insights-runtime-extractor-d2dnn" (UID: "079ed4e4-bdc4-405c-90a1-a22b84fcf948") : secret "insights-runtime-extractor-tls" not found Oct 29 22:33:41.058815 ip-10-0-140-74 kubenswrapper[5057]: E1029 22:33:41.058783 5057 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5296979a-7927-4ee2-84b9-a93780a2ad22-registry-tls podName:5296979a-7927-4ee2-84b9-a93780a2ad22 nodeName:}" failed. No retries permitted until 2026-10-29 22:33:42.058774439 +0000 UTC m=+2.841987844 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5296979a-7927-4ee2-84b9-a93780a2ad22-registry-tls") pod "image-registry-78db97464f-6ljzh" (UID: "5296979a-7927-4ee2-84b9-a93780a2ad22") : secret "image-registry-tls" not found Oct 29 22:33:41.058815 ip-10-0-140-74 kubenswrapper[5057]: E1029 22:33:41.058801 5057 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f8f92bfb-5cef-412e-a6c1-4bd793c59fe7-metrics-certs podName:f8f92bfb-5cef-412e-a6c1-4bd793c59fe7 nodeName:}" failed. No retries permitted until 2026-10-29 22:33:42.058792659 +0000 UTC m=+2.842006068 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f8f92bfb-5cef-412e-a6c1-4bd793c59fe7-metrics-certs") pod "network-metrics-daemon-hzlmg" (UID: "f8f92bfb-5cef-412e-a6c1-4bd793c59fe7") : secret "metrics-daemon-secret" not found Oct 29 22:33:41.059181 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:41.058846 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/989956e8-1893-4bbe-a4bf-6a2081509080-registry-tls\") pod \"image-registry-84866f86bd-2v6d7\" (UID: \"989956e8-1893-4bbe-a4bf-6a2081509080\") " pod="openshift-image-registry/image-registry-84866f86bd-2v6d7" Oct 29 22:33:41.059181 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:41.058881 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/71d80eec-b5e7-41d0-b58d-8a8d9a6c769c-service-ca-bundle\") pod \"router-default-5d8d7cc579-8cwjp\" (UID: \"71d80eec-b5e7-41d0-b58d-8a8d9a6c769c\") " pod="openshift-ingress/router-default-5d8d7cc579-8cwjp" Oct 29 22:33:41.059181 ip-10-0-140-74 kubenswrapper[5057]: E1029 22:33:41.058914 5057 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 29 22:33:41.059181 ip-10-0-140-74 kubenswrapper[5057]: E1029 22:33:41.058923 5057 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-84866f86bd-2v6d7: secret "image-registry-tls" not found Oct 29 22:33:41.059181 ip-10-0-140-74 kubenswrapper[5057]: E1029 22:33:41.058946 5057 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/989956e8-1893-4bbe-a4bf-6a2081509080-registry-tls podName:989956e8-1893-4bbe-a4bf-6a2081509080 nodeName:}" failed. No retries permitted until 2026-10-29 22:33:42.058937059 +0000 UTC m=+2.842150480 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/989956e8-1893-4bbe-a4bf-6a2081509080-registry-tls") pod "image-registry-84866f86bd-2v6d7" (UID: "989956e8-1893-4bbe-a4bf-6a2081509080") : secret "image-registry-tls" not found Oct 29 22:33:41.059181 ip-10-0-140-74 kubenswrapper[5057]: E1029 22:33:41.058989 5057 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/71d80eec-b5e7-41d0-b58d-8a8d9a6c769c-service-ca-bundle podName:71d80eec-b5e7-41d0-b58d-8a8d9a6c769c nodeName:}" failed. No retries permitted until 2026-10-29 22:33:42.058982264 +0000 UTC m=+2.842195668 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/71d80eec-b5e7-41d0-b58d-8a8d9a6c769c-service-ca-bundle") pod "router-default-5d8d7cc579-8cwjp" (UID: "71d80eec-b5e7-41d0-b58d-8a8d9a6c769c") : configmap references non-existent config key: service-ca.crt Oct 29 22:33:41.060719 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:41.060703 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71d80eec-b5e7-41d0-b58d-8a8d9a6c769c-metrics-certs\") pod \"router-default-5d8d7cc579-8cwjp\" (UID: \"71d80eec-b5e7-41d0-b58d-8a8d9a6c769c\") " pod="openshift-ingress/router-default-5d8d7cc579-8cwjp" Oct 29 22:33:41.391193 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:41.391096 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-jjz47" event={"ID":"6b34b3b1-84af-45f3-92ee-1ca720e8b263","Type":"ContainerStarted","Data":"3665792a7a041193ffcb2881b0f0a576b7043fd51efe89f7905b783e68c9b293"} Oct 29 22:33:41.393566 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:41.393534 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-vj6md_aa6a190e-3283-458a-9dfd-2de7cb193fc5/console-operator/2.log" Oct 29 22:33:41.394030 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:41.394009 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-vj6md_aa6a190e-3283-458a-9dfd-2de7cb193fc5/console-operator/1.log" Oct 29 22:33:41.394127 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:41.394048 5057 generic.go:356] "Generic (PLEG): container finished" podID="aa6a190e-3283-458a-9dfd-2de7cb193fc5" containerID="0d5a2e4675b4b9cb622cb6f909929ab1e98c8fce31ad1e68771db98245eb9f63" exitCode=255 Oct 29 22:33:41.394197 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:41.394118 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-vj6md" event={"ID":"aa6a190e-3283-458a-9dfd-2de7cb193fc5","Type":"ContainerDied","Data":"0d5a2e4675b4b9cb622cb6f909929ab1e98c8fce31ad1e68771db98245eb9f63"} Oct 29 22:33:41.394197 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:41.394169 5057 scope.go:122] "RemoveContainer" containerID="e5ab727c17d3a45185a300b3b2a3688775b919a83c6e93f3a510f87ee56a0fb7" Oct 29 22:33:41.394608 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:41.394590 5057 scope.go:122] "RemoveContainer" containerID="0d5a2e4675b4b9cb622cb6f909929ab1e98c8fce31ad1e68771db98245eb9f63" Oct 29 22:33:41.394954 ip-10-0-140-74 kubenswrapper[5057]: E1029 22:33:41.394930 5057 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-vj6md_openshift-console-operator(aa6a190e-3283-458a-9dfd-2de7cb193fc5)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-vj6md" podUID="aa6a190e-3283-458a-9dfd-2de7cb193fc5" Oct 29 22:33:41.395895 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:41.395870 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-p2qv4" event={"ID":"a53078b9-a6a8-423c-b852-34eececa7ab1","Type":"ContainerStarted","Data":"1e6d29537f616afb4e09e32766859e33336b4bfe76391609b0f0e7f54563afd1"} Oct 29 22:33:41.959600 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:41.959548 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:33:42.068513 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:42.068466 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2accd775-2173-4bfe-a413-7d243bd0820f-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-6rr4f\" (UID: \"2accd775-2173-4bfe-a413-7d243bd0820f\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-6rr4f" Oct 29 22:33:42.068690 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:42.068564 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f8f92bfb-5cef-412e-a6c1-4bd793c59fe7-metrics-certs\") pod \"network-metrics-daemon-hzlmg\" (UID: \"f8f92bfb-5cef-412e-a6c1-4bd793c59fe7\") " pod="openshift-multus/network-metrics-daemon-hzlmg" Oct 29 22:33:42.068690 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:42.068599 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5296979a-7927-4ee2-84b9-a93780a2ad22-registry-tls\") pod \"image-registry-78db97464f-6ljzh\" (UID: \"5296979a-7927-4ee2-84b9-a93780a2ad22\") " pod="openshift-image-registry/image-registry-78db97464f-6ljzh" Oct 29 22:33:42.068690 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:42.068623 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/989956e8-1893-4bbe-a4bf-6a2081509080-registry-tls\") pod \"image-registry-84866f86bd-2v6d7\" (UID: \"989956e8-1893-4bbe-a4bf-6a2081509080\") " pod="openshift-image-registry/image-registry-84866f86bd-2v6d7" Oct 29 22:33:42.068690 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:42.068643 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/71d80eec-b5e7-41d0-b58d-8a8d9a6c769c-service-ca-bundle\") pod \"router-default-5d8d7cc579-8cwjp\" (UID: \"71d80eec-b5e7-41d0-b58d-8a8d9a6c769c\") " pod="openshift-ingress/router-default-5d8d7cc579-8cwjp" Oct 29 22:33:42.068690 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:42.068666 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/079ed4e4-bdc4-405c-90a1-a22b84fcf948-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-d2dnn\" (UID: \"079ed4e4-bdc4-405c-90a1-a22b84fcf948\") " pod="openshift-insights/insights-runtime-extractor-d2dnn" Oct 29 22:33:42.068690 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:42.068687 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/641f9ec1-c8e2-4774-9d4f-6a7a2da97b6f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-8c5hr\" (UID: \"641f9ec1-c8e2-4774-9d4f-6a7a2da97b6f\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-8c5hr" Oct 29 22:33:42.068995 ip-10-0-140-74 kubenswrapper[5057]: E1029 22:33:42.068800 5057 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 29 22:33:42.068995 ip-10-0-140-74 kubenswrapper[5057]: E1029 22:33:42.068852 5057 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/641f9ec1-c8e2-4774-9d4f-6a7a2da97b6f-cluster-monitoring-operator-tls podName:641f9ec1-c8e2-4774-9d4f-6a7a2da97b6f nodeName:}" failed. No retries permitted until 2026-10-29 22:33:44.068841373 +0000 UTC m=+4.852054777 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/641f9ec1-c8e2-4774-9d4f-6a7a2da97b6f-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-8c5hr" (UID: "641f9ec1-c8e2-4774-9d4f-6a7a2da97b6f") : secret "cluster-monitoring-operator-tls" not found Oct 29 22:33:42.069099 ip-10-0-140-74 kubenswrapper[5057]: E1029 22:33:42.069088 5057 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 29 22:33:42.069152 ip-10-0-140-74 kubenswrapper[5057]: E1029 22:33:42.069097 5057 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 29 22:33:42.069152 ip-10-0-140-74 kubenswrapper[5057]: E1029 22:33:42.069130 5057 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2accd775-2173-4bfe-a413-7d243bd0820f-networking-console-plugin-cert podName:2accd775-2173-4bfe-a413-7d243bd0820f nodeName:}" failed. No retries permitted until 2026-10-29 22:33:44.06911836 +0000 UTC m=+4.852331764 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/2accd775-2173-4bfe-a413-7d243bd0820f-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-6rr4f" (UID: "2accd775-2173-4bfe-a413-7d243bd0820f") : secret "networking-console-plugin-cert" not found Oct 29 22:33:42.069252 ip-10-0-140-74 kubenswrapper[5057]: E1029 22:33:42.069155 5057 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/079ed4e4-bdc4-405c-90a1-a22b84fcf948-insights-runtime-extractor-tls podName:079ed4e4-bdc4-405c-90a1-a22b84fcf948 nodeName:}" failed. No retries permitted until 2026-10-29 22:33:44.069138392 +0000 UTC m=+4.852351814 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/079ed4e4-bdc4-405c-90a1-a22b84fcf948-insights-runtime-extractor-tls") pod "insights-runtime-extractor-d2dnn" (UID: "079ed4e4-bdc4-405c-90a1-a22b84fcf948") : secret "insights-runtime-extractor-tls" not found Oct 29 22:33:42.069252 ip-10-0-140-74 kubenswrapper[5057]: E1029 22:33:42.069208 5057 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 29 22:33:42.069252 ip-10-0-140-74 kubenswrapper[5057]: E1029 22:33:42.069219 5057 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-84866f86bd-2v6d7: secret "image-registry-tls" not found Oct 29 22:33:42.069252 ip-10-0-140-74 kubenswrapper[5057]: E1029 22:33:42.069252 5057 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/989956e8-1893-4bbe-a4bf-6a2081509080-registry-tls podName:989956e8-1893-4bbe-a4bf-6a2081509080 nodeName:}" failed. No retries permitted until 2026-10-29 22:33:44.069241259 +0000 UTC m=+4.852454682 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/989956e8-1893-4bbe-a4bf-6a2081509080-registry-tls") pod "image-registry-84866f86bd-2v6d7" (UID: "989956e8-1893-4bbe-a4bf-6a2081509080") : secret "image-registry-tls" not found Oct 29 22:33:42.069472 ip-10-0-140-74 kubenswrapper[5057]: E1029 22:33:42.069309 5057 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/71d80eec-b5e7-41d0-b58d-8a8d9a6c769c-service-ca-bundle podName:71d80eec-b5e7-41d0-b58d-8a8d9a6c769c nodeName:}" failed. No retries permitted until 2026-10-29 22:33:44.069300675 +0000 UTC m=+4.852514111 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/71d80eec-b5e7-41d0-b58d-8a8d9a6c769c-service-ca-bundle") pod "router-default-5d8d7cc579-8cwjp" (UID: "71d80eec-b5e7-41d0-b58d-8a8d9a6c769c") : configmap references non-existent config key: service-ca.crt Oct 29 22:33:42.069472 ip-10-0-140-74 kubenswrapper[5057]: E1029 22:33:42.069404 5057 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 29 22:33:42.069472 ip-10-0-140-74 kubenswrapper[5057]: E1029 22:33:42.069418 5057 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-78db97464f-6ljzh: secret "image-registry-tls" not found Oct 29 22:33:42.069472 ip-10-0-140-74 kubenswrapper[5057]: E1029 22:33:42.069453 5057 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5296979a-7927-4ee2-84b9-a93780a2ad22-registry-tls podName:5296979a-7927-4ee2-84b9-a93780a2ad22 nodeName:}" failed. No retries permitted until 2026-10-29 22:33:44.069441434 +0000 UTC m=+4.852654863 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5296979a-7927-4ee2-84b9-a93780a2ad22-registry-tls") pod "image-registry-78db97464f-6ljzh" (UID: "5296979a-7927-4ee2-84b9-a93780a2ad22") : secret "image-registry-tls" not found Oct 29 22:33:42.070063 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:42.070030 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:33:42.074713 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:42.074669 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f8f92bfb-5cef-412e-a6c1-4bd793c59fe7-metrics-certs\") pod \"network-metrics-daemon-hzlmg\" (UID: \"f8f92bfb-5cef-412e-a6c1-4bd793c59fe7\") " pod="openshift-multus/network-metrics-daemon-hzlmg" Oct 29 22:33:42.141672 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:42.141641 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hzlmg" Oct 29 22:33:42.177613 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:42.177583 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:33:42.358846 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:42.358804 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:33:42.401638 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:42.401612 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-vj6md_aa6a190e-3283-458a-9dfd-2de7cb193fc5/console-operator/2.log" Oct 29 22:33:42.448880 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:42.448843 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-hzlmg"] Oct 29 22:33:42.454775 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:42.454736 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8f92bfb_5cef_412e_a6c1_4bd793c59fe7.slice/crio-3d09ec398a8e36c6a49b448f1b1e44b6c7084868527df67881f813bae01f40b2 WatchSource:0}: Error finding container 3d09ec398a8e36c6a49b448f1b1e44b6c7084868527df67881f813bae01f40b2: Status 404 returned error can't find the container with id 3d09ec398a8e36c6a49b448f1b1e44b6c7084868527df67881f813bae01f40b2 Oct 29 22:33:42.559551 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:42.559473 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:33:42.767784 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:42.767759 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:33:43.161621 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:43.161595 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-khlhx" Oct 29 22:33:43.339093 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:43.339053 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:33:43.406858 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:43.406829 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-p2qv4" event={"ID":"a53078b9-a6a8-423c-b852-34eececa7ab1","Type":"ContainerStarted","Data":"22af55dcdf884edc9271f36536826204356c73c7a0e8f4935239d8a4a4892cc5"} Oct 29 22:33:43.407253 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:43.406885 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-p2qv4" event={"ID":"a53078b9-a6a8-423c-b852-34eececa7ab1","Type":"ContainerStarted","Data":"470a80d46261e6d08680cbb4e40857f1a7c1ecbbcb8b9328a28ea994b2db2d61"} Oct 29 22:33:43.408714 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:43.408679 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-jjz47" event={"ID":"6b34b3b1-84af-45f3-92ee-1ca720e8b263","Type":"ContainerStarted","Data":"77ea9f369cb6a4a0676eca55021433df1a31c489707e34948bb4f2c53af32bcd"} Oct 29 22:33:43.408824 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:43.408732 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-jjz47" event={"ID":"6b34b3b1-84af-45f3-92ee-1ca720e8b263","Type":"ContainerStarted","Data":"24b186f6cb20bac1240b6a9f780cc17d640cf22cec3ffe53a26dd60f8902fad5"} Oct 29 22:33:43.408824 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:43.408795 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-jjz47" Oct 29 22:33:43.409992 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:43.409972 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-hzlmg" event={"ID":"f8f92bfb-5cef-412e-a6c1-4bd793c59fe7","Type":"ContainerStarted","Data":"3d09ec398a8e36c6a49b448f1b1e44b6c7084868527df67881f813bae01f40b2"} Oct 29 22:33:43.959184 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:43.959145 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:33:44.090216 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:44.090180 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2accd775-2173-4bfe-a413-7d243bd0820f-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-6rr4f\" (UID: \"2accd775-2173-4bfe-a413-7d243bd0820f\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-6rr4f" Oct 29 22:33:44.090412 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:44.090233 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5296979a-7927-4ee2-84b9-a93780a2ad22-registry-tls\") pod \"image-registry-78db97464f-6ljzh\" (UID: \"5296979a-7927-4ee2-84b9-a93780a2ad22\") " pod="openshift-image-registry/image-registry-78db97464f-6ljzh" Oct 29 22:33:44.090412 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:44.090255 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/989956e8-1893-4bbe-a4bf-6a2081509080-registry-tls\") pod \"image-registry-84866f86bd-2v6d7\" (UID: \"989956e8-1893-4bbe-a4bf-6a2081509080\") " pod="openshift-image-registry/image-registry-84866f86bd-2v6d7" Oct 29 22:33:44.090412 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:44.090272 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/71d80eec-b5e7-41d0-b58d-8a8d9a6c769c-service-ca-bundle\") pod \"router-default-5d8d7cc579-8cwjp\" (UID: \"71d80eec-b5e7-41d0-b58d-8a8d9a6c769c\") " pod="openshift-ingress/router-default-5d8d7cc579-8cwjp" Oct 29 22:33:44.090412 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:44.090293 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/079ed4e4-bdc4-405c-90a1-a22b84fcf948-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-d2dnn\" (UID: \"079ed4e4-bdc4-405c-90a1-a22b84fcf948\") " pod="openshift-insights/insights-runtime-extractor-d2dnn" Oct 29 22:33:44.090412 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:44.090311 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/641f9ec1-c8e2-4774-9d4f-6a7a2da97b6f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-8c5hr\" (UID: \"641f9ec1-c8e2-4774-9d4f-6a7a2da97b6f\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-8c5hr" Oct 29 22:33:44.090709 ip-10-0-140-74 kubenswrapper[5057]: E1029 22:33:44.090428 5057 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 29 22:33:44.090709 ip-10-0-140-74 kubenswrapper[5057]: E1029 22:33:44.090428 5057 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 29 22:33:44.090709 ip-10-0-140-74 kubenswrapper[5057]: E1029 22:33:44.090449 5057 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/71d80eec-b5e7-41d0-b58d-8a8d9a6c769c-service-ca-bundle podName:71d80eec-b5e7-41d0-b58d-8a8d9a6c769c nodeName:}" failed. No retries permitted until 2026-10-29 22:33:48.090429363 +0000 UTC m=+8.873642769 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/71d80eec-b5e7-41d0-b58d-8a8d9a6c769c-service-ca-bundle") pod "router-default-5d8d7cc579-8cwjp" (UID: "71d80eec-b5e7-41d0-b58d-8a8d9a6c769c") : configmap references non-existent config key: service-ca.crt Oct 29 22:33:44.090709 ip-10-0-140-74 kubenswrapper[5057]: E1029 22:33:44.090562 5057 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/079ed4e4-bdc4-405c-90a1-a22b84fcf948-insights-runtime-extractor-tls podName:079ed4e4-bdc4-405c-90a1-a22b84fcf948 nodeName:}" failed. No retries permitted until 2026-10-29 22:33:48.090545859 +0000 UTC m=+8.873759263 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/079ed4e4-bdc4-405c-90a1-a22b84fcf948-insights-runtime-extractor-tls") pod "insights-runtime-extractor-d2dnn" (UID: "079ed4e4-bdc4-405c-90a1-a22b84fcf948") : secret "insights-runtime-extractor-tls" not found Oct 29 22:33:44.090709 ip-10-0-140-74 kubenswrapper[5057]: E1029 22:33:44.090578 5057 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/641f9ec1-c8e2-4774-9d4f-6a7a2da97b6f-cluster-monitoring-operator-tls podName:641f9ec1-c8e2-4774-9d4f-6a7a2da97b6f nodeName:}" failed. No retries permitted until 2026-10-29 22:33:48.090568916 +0000 UTC m=+8.873782322 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/641f9ec1-c8e2-4774-9d4f-6a7a2da97b6f-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-8c5hr" (UID: "641f9ec1-c8e2-4774-9d4f-6a7a2da97b6f") : secret "cluster-monitoring-operator-tls" not found Oct 29 22:33:44.092949 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:44.092918 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5296979a-7927-4ee2-84b9-a93780a2ad22-registry-tls\") pod \"image-registry-78db97464f-6ljzh\" (UID: \"5296979a-7927-4ee2-84b9-a93780a2ad22\") " pod="openshift-image-registry/image-registry-78db97464f-6ljzh" Oct 29 22:33:44.093305 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:44.093275 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2accd775-2173-4bfe-a413-7d243bd0820f-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-6rr4f\" (UID: \"2accd775-2173-4bfe-a413-7d243bd0820f\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-6rr4f" Oct 29 22:33:44.093425 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:44.093304 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/989956e8-1893-4bbe-a4bf-6a2081509080-registry-tls\") pod \"image-registry-84866f86bd-2v6d7\" (UID: \"989956e8-1893-4bbe-a4bf-6a2081509080\") " pod="openshift-image-registry/image-registry-84866f86bd-2v6d7" Oct 29 22:33:44.159721 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:44.159685 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:33:44.231916 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:44.231856 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-6rr4f" Oct 29 22:33:44.231916 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:44.231886 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-78db97464f-6ljzh" Oct 29 22:33:44.242254 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:44.242233 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-84866f86bd-2v6d7" Oct 29 22:33:44.368645 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:44.368614 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:33:44.420187 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:44.420152 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-hzlmg" event={"ID":"f8f92bfb-5cef-412e-a6c1-4bd793c59fe7","Type":"ContainerStarted","Data":"62dff03e7ca6e81ad161fc1929bc752fb3cba208a5e35b6f9fbc951785041864"} Oct 29 22:33:44.420603 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:44.420245 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-hzlmg" event={"ID":"f8f92bfb-5cef-412e-a6c1-4bd793c59fe7","Type":"ContainerStarted","Data":"98d5303afd3c1910440452ba4357133d5cc3490384d3be2f74533d1c6aed6f69"} Oct 29 22:33:44.421637 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:44.421615 5057 generic.go:356] "Generic (PLEG): container finished" podID="c635d035-053b-49aa-9559-79c31e8e49ff" containerID="a3fb0a65f00623c0a7fbc591def44660ef1faa56e5f686dc9671c0cebfecf081" exitCode=0 Oct 29 22:33:44.421729 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:44.421693 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-pk97l" event={"ID":"c635d035-053b-49aa-9559-79c31e8e49ff","Type":"ContainerDied","Data":"a3fb0a65f00623c0a7fbc591def44660ef1faa56e5f686dc9671c0cebfecf081"} Oct 29 22:33:44.422003 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:44.421987 5057 scope.go:122] "RemoveContainer" containerID="a3fb0a65f00623c0a7fbc591def44660ef1faa56e5f686dc9671c0cebfecf081" Oct 29 22:33:44.423202 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:44.423181 5057 generic.go:356] "Generic (PLEG): container finished" podID="4071b0c7-c4f3-419a-8d6e-6acc144fc9a4" containerID="1eb5a229ae3b60cb6a6342d3cfa649361c63bcc57b0708eb0d314b868634e43d" exitCode=0 Oct 29 22:33:44.423305 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:44.423243 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-sk5qq" event={"ID":"4071b0c7-c4f3-419a-8d6e-6acc144fc9a4","Type":"ContainerDied","Data":"1eb5a229ae3b60cb6a6342d3cfa649361c63bcc57b0708eb0d314b868634e43d"} Oct 29 22:33:44.423871 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:44.423853 5057 scope.go:122] "RemoveContainer" containerID="1eb5a229ae3b60cb6a6342d3cfa649361c63bcc57b0708eb0d314b868634e43d" Oct 29 22:33:44.518403 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:44.513896 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:33:44.550428 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:44.550400 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-6rr4f"] Oct 29 22:33:44.554729 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:44.554704 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-78db97464f-6ljzh"] Oct 29 22:33:44.555198 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:44.555172 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2accd775_2173_4bfe_a413_7d243bd0820f.slice/crio-1c0a8827c03e75848dc0f45110108a66ac03611fa709b35ea75b0797b897ada0 WatchSource:0}: Error finding container 1c0a8827c03e75848dc0f45110108a66ac03611fa709b35ea75b0797b897ada0: Status 404 returned error can't find the container with id 1c0a8827c03e75848dc0f45110108a66ac03611fa709b35ea75b0797b897ada0 Oct 29 22:33:44.557511 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:44.557481 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-84866f86bd-2v6d7"] Oct 29 22:33:44.560562 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:44.560540 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5296979a_7927_4ee2_84b9_a93780a2ad22.slice/crio-aa9cd79eac96497d75021725e7d6f4a8c7a012f3c8fd7e520590844af17e72b2 WatchSource:0}: Error finding container aa9cd79eac96497d75021725e7d6f4a8c7a012f3c8fd7e520590844af17e72b2: Status 404 returned error can't find the container with id aa9cd79eac96497d75021725e7d6f4a8c7a012f3c8fd7e520590844af17e72b2 Oct 29 22:33:44.565663 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:44.565089 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod989956e8_1893_4bbe_a4bf_6a2081509080.slice/crio-1a573d2d443f71339edf3b0caa40e76794c0ede1c2c257700a55fd6a80fe572d WatchSource:0}: Error finding container 1a573d2d443f71339edf3b0caa40e76794c0ede1c2c257700a55fd6a80fe572d: Status 404 returned error can't find the container with id 1a573d2d443f71339edf3b0caa40e76794c0ede1c2c257700a55fd6a80fe572d Oct 29 22:33:44.710834 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:44.710755 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:33:44.821084 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:44.821001 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:33:45.428994 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:45.428959 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-sk5qq" event={"ID":"4071b0c7-c4f3-419a-8d6e-6acc144fc9a4","Type":"ContainerStarted","Data":"11f81ba477f65cad62ee86d7a1aeb5967e0f4441be573874d76242a0cb0e6b6c"} Oct 29 22:33:45.430782 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:45.430757 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-pk97l" event={"ID":"c635d035-053b-49aa-9559-79c31e8e49ff","Type":"ContainerStarted","Data":"0f39e42e482ffef5f8ca9a78ddacb82a3c8cb84f9c9df056363eb234c6e051dc"} Oct 29 22:33:45.432203 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:45.432175 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-84866f86bd-2v6d7" event={"ID":"989956e8-1893-4bbe-a4bf-6a2081509080","Type":"ContainerStarted","Data":"d76c049df3eb0de0f412cd20199664ec6cbad532ed34bc561691c39d2d0eb211"} Oct 29 22:33:45.432361 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:45.432217 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-84866f86bd-2v6d7" event={"ID":"989956e8-1893-4bbe-a4bf-6a2081509080","Type":"ContainerStarted","Data":"1a573d2d443f71339edf3b0caa40e76794c0ede1c2c257700a55fd6a80fe572d"} Oct 29 22:33:45.433641 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:45.433618 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-78db97464f-6ljzh" event={"ID":"5296979a-7927-4ee2-84b9-a93780a2ad22","Type":"ContainerStarted","Data":"41452c10b3c4ce37981bdce20cea5af734997d17e7c85f9662a559369ad9879b"} Oct 29 22:33:45.433748 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:45.433654 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-78db97464f-6ljzh" event={"ID":"5296979a-7927-4ee2-84b9-a93780a2ad22","Type":"ContainerStarted","Data":"aa9cd79eac96497d75021725e7d6f4a8c7a012f3c8fd7e520590844af17e72b2"} Oct 29 22:33:45.434777 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:45.434755 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-6rr4f" event={"ID":"2accd775-2173-4bfe-a413-7d243bd0820f","Type":"ContainerStarted","Data":"1c0a8827c03e75848dc0f45110108a66ac03611fa709b35ea75b0797b897ada0"} Oct 29 22:33:45.560136 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:45.560093 5057 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-vj6md" Oct 29 22:33:45.560476 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:45.560459 5057 scope.go:122] "RemoveContainer" containerID="0d5a2e4675b4b9cb622cb6f909929ab1e98c8fce31ad1e68771db98245eb9f63" Oct 29 22:33:45.560659 ip-10-0-140-74 kubenswrapper[5057]: E1029 22:33:45.560639 5057 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-vj6md_openshift-console-operator(aa6a190e-3283-458a-9dfd-2de7cb193fc5)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-vj6md" podUID="aa6a190e-3283-458a-9dfd-2de7cb193fc5" Oct 29 22:33:45.626531 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:45.626509 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-khlhx" Oct 29 22:33:45.626721 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:45.626706 5057 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 29 22:33:45.756545 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:45.756519 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-78db97464f-6ljzh" Oct 29 22:33:45.962829 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:45.962784 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:33:46.158467 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:46.158390 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:33:46.362030 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:46.361992 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:33:46.440315 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:46.440247 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-6rr4f" event={"ID":"2accd775-2173-4bfe-a413-7d243bd0820f","Type":"ContainerStarted","Data":"0a86cab122b10d400c5ad4abdd6454811e3318f59ff4b45bed5125daaa9312bb"} Oct 29 22:33:46.440658 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:46.440362 5057 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 29 22:33:46.559022 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:46.558992 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:33:46.757202 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:46.757168 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:33:46.868186 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:46.868159 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:33:46.946697 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:46.946663 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-7jv5g" Oct 29 22:33:48.128174 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:48.128132 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/079ed4e4-bdc4-405c-90a1-a22b84fcf948-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-d2dnn\" (UID: \"079ed4e4-bdc4-405c-90a1-a22b84fcf948\") " pod="openshift-insights/insights-runtime-extractor-d2dnn" Oct 29 22:33:48.128174 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:48.128182 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/641f9ec1-c8e2-4774-9d4f-6a7a2da97b6f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-8c5hr\" (UID: \"641f9ec1-c8e2-4774-9d4f-6a7a2da97b6f\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-8c5hr" Oct 29 22:33:48.128631 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:48.128247 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/71d80eec-b5e7-41d0-b58d-8a8d9a6c769c-service-ca-bundle\") pod \"router-default-5d8d7cc579-8cwjp\" (UID: \"71d80eec-b5e7-41d0-b58d-8a8d9a6c769c\") " pod="openshift-ingress/router-default-5d8d7cc579-8cwjp" Oct 29 22:33:48.128981 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:48.128962 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/71d80eec-b5e7-41d0-b58d-8a8d9a6c769c-service-ca-bundle\") pod \"router-default-5d8d7cc579-8cwjp\" (UID: \"71d80eec-b5e7-41d0-b58d-8a8d9a6c769c\") " pod="openshift-ingress/router-default-5d8d7cc579-8cwjp" Oct 29 22:33:48.129776 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:48.129747 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-5d8d7cc579-8cwjp" Oct 29 22:33:48.130528 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:48.130510 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/079ed4e4-bdc4-405c-90a1-a22b84fcf948-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-d2dnn\" (UID: \"079ed4e4-bdc4-405c-90a1-a22b84fcf948\") " pod="openshift-insights/insights-runtime-extractor-d2dnn" Oct 29 22:33:48.130696 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:48.130679 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/641f9ec1-c8e2-4774-9d4f-6a7a2da97b6f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-8c5hr\" (UID: \"641f9ec1-c8e2-4774-9d4f-6a7a2da97b6f\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-8c5hr" Oct 29 22:33:48.144622 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:48.144602 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-d2dnn" Oct 29 22:33:48.430985 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:48.430920 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-8c5hr" Oct 29 22:33:48.442726 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:48.442703 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-5d8d7cc579-8cwjp"] Oct 29 22:33:48.446288 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:48.446253 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71d80eec_b5e7_41d0_b58d_8a8d9a6c769c.slice/crio-e4eb5d7ac08b57bf8216a9ce2ba6554861d78e243538ad8aecb308ed490807ad WatchSource:0}: Error finding container e4eb5d7ac08b57bf8216a9ce2ba6554861d78e243538ad8aecb308ed490807ad: Status 404 returned error can't find the container with id e4eb5d7ac08b57bf8216a9ce2ba6554861d78e243538ad8aecb308ed490807ad Oct 29 22:33:48.452771 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:48.452748 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-d2dnn"] Oct 29 22:33:48.462067 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:48.460832 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod079ed4e4_bdc4_405c_90a1_a22b84fcf948.slice/crio-646afa380748f2339fda4ae94432c55726e56f88295df74e6e69518c8728cb3b WatchSource:0}: Error finding container 646afa380748f2339fda4ae94432c55726e56f88295df74e6e69518c8728cb3b: Status 404 returned error can't find the container with id 646afa380748f2339fda4ae94432c55726e56f88295df74e6e69518c8728cb3b Oct 29 22:33:48.729966 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:48.729919 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-8c5hr"] Oct 29 22:33:48.733587 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:48.733566 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod641f9ec1_c8e2_4774_9d4f_6a7a2da97b6f.slice/crio-dbaa5de328d1f0af00bc123f3cb499b2f4c945ce21c92928862238191515f187 WatchSource:0}: Error finding container dbaa5de328d1f0af00bc123f3cb499b2f4c945ce21c92928862238191515f187: Status 404 returned error can't find the container with id dbaa5de328d1f0af00bc123f3cb499b2f4c945ce21c92928862238191515f187 Oct 29 22:33:49.076963 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:49.076890 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-vj6md" Oct 29 22:33:49.077363 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:49.077326 5057 scope.go:122] "RemoveContainer" containerID="0d5a2e4675b4b9cb622cb6f909929ab1e98c8fce31ad1e68771db98245eb9f63" Oct 29 22:33:49.077582 ip-10-0-140-74 kubenswrapper[5057]: E1029 22:33:49.077560 5057 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-vj6md_openshift-console-operator(aa6a190e-3283-458a-9dfd-2de7cb193fc5)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-vj6md" podUID="aa6a190e-3283-458a-9dfd-2de7cb193fc5" Oct 29 22:33:49.454458 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:49.454418 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-d2dnn" event={"ID":"079ed4e4-bdc4-405c-90a1-a22b84fcf948","Type":"ContainerStarted","Data":"d901954a3c3d5c6947b3fd57eb33a6dfaefc4e8e2878b2a7659141d2e61b9f9f"} Oct 29 22:33:49.454879 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:49.454483 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-d2dnn" event={"ID":"079ed4e4-bdc4-405c-90a1-a22b84fcf948","Type":"ContainerStarted","Data":"59976dbee90e8025127644732ca0ebf4fa393f209518dcbbce33ec0f6342257b"} Oct 29 22:33:49.454879 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:49.454509 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-d2dnn" event={"ID":"079ed4e4-bdc4-405c-90a1-a22b84fcf948","Type":"ContainerStarted","Data":"646afa380748f2339fda4ae94432c55726e56f88295df74e6e69518c8728cb3b"} Oct 29 22:33:49.455824 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:49.455798 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-8c5hr" event={"ID":"641f9ec1-c8e2-4774-9d4f-6a7a2da97b6f","Type":"ContainerStarted","Data":"dbaa5de328d1f0af00bc123f3cb499b2f4c945ce21c92928862238191515f187"} Oct 29 22:33:49.457456 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:49.457432 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-5d8d7cc579-8cwjp" event={"ID":"71d80eec-b5e7-41d0-b58d-8a8d9a6c769c","Type":"ContainerStarted","Data":"c79a0f08d0a13825f4132f97787a3568c8edcc4512a6ef8fa4b5498651918e07"} Oct 29 22:33:49.457536 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:49.457471 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-5d8d7cc579-8cwjp" event={"ID":"71d80eec-b5e7-41d0-b58d-8a8d9a6c769c","Type":"ContainerStarted","Data":"e4eb5d7ac08b57bf8216a9ce2ba6554861d78e243538ad8aecb308ed490807ad"} Oct 29 22:33:50.130549 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:50.130511 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-5d8d7cc579-8cwjp" Oct 29 22:33:50.133461 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:50.133432 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-5d8d7cc579-8cwjp" Oct 29 22:33:50.463140 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:50.463099 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-8c5hr" event={"ID":"641f9ec1-c8e2-4774-9d4f-6a7a2da97b6f","Type":"ContainerStarted","Data":"dda371c60408fe0e4df7597971176429b50ebc980760fa4d58c5b34c80f3f2d1"} Oct 29 22:33:50.661280 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:50.661254 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-5d8d7cc579-8cwjp" Oct 29 22:33:50.662664 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:50.662638 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-5d8d7cc579-8cwjp" Oct 29 22:33:50.898554 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:50.898481 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:33:51.009733 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:51.009696 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:33:51.300038 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:51.299999 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:33:51.320412 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:51.320388 5057 scope.go:122] "RemoveContainer" containerID="0d5a2e4675b4b9cb622cb6f909929ab1e98c8fce31ad1e68771db98245eb9f63" Oct 29 22:33:51.407432 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:51.407394 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:33:51.468768 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:51.468747 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-vj6md_aa6a190e-3283-458a-9dfd-2de7cb193fc5/console-operator/2.log" Oct 29 22:33:51.469120 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:51.468836 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-vj6md" event={"ID":"aa6a190e-3283-458a-9dfd-2de7cb193fc5","Type":"ContainerStarted","Data":"48276cacef1853675afd06fdd07fb49ff9197e90a0d6ec495fa7d236801525fc"} Oct 29 22:33:51.469234 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:51.469214 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-vj6md" Oct 29 22:33:51.470034 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:51.470004 5057 patch_prober.go:29] interesting pod/console-operator-7cd56b8dbc-vj6md container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.132.0.13:8443/readyz\": dial tcp 10.132.0.13:8443: connect: connection refused" start-of-body= Oct 29 22:33:51.470137 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:51.470048 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-7cd56b8dbc-vj6md" podUID="aa6a190e-3283-458a-9dfd-2de7cb193fc5" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.13:8443/readyz\": dial tcp 10.132.0.13:8443: connect: connection refused" Oct 29 22:33:51.471118 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:51.471090 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-d2dnn" event={"ID":"079ed4e4-bdc4-405c-90a1-a22b84fcf948","Type":"ContainerStarted","Data":"15d793e16750167f9427339f6e70bb0b800ccceff71121b83e2cf87917421871"} Oct 29 22:33:52.018035 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:52.017969 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:33:52.132539 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:52.132505 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:33:52.239589 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:52.239548 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:33:52.348622 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:52.348545 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:33:52.457452 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:52.457415 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:33:52.479637 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:52.479609 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-vj6md" Oct 29 22:33:53.023059 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:53.023024 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-hsxj9"] Oct 29 22:33:53.025678 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:53.025640 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-hsxj9" Oct 29 22:33:53.041906 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:53.041871 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-78db97464f-6ljzh"] Oct 29 22:33:53.050479 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:53.050453 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-z6wqq\"" Oct 29 22:33:53.050655 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:53.050533 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Oct 29 22:33:53.051606 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:53.051576 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-hsxj9"] Oct 29 22:33:53.141493 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:53.141458 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6dd789c8b7-9s8rp"] Oct 29 22:33:53.143672 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:53.143653 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-9s8rp" Oct 29 22:33:53.147829 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:53.147801 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-d49c68d56-vpbg5"] Oct 29 22:33:53.149983 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:53.149960 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-d49c68d56-vpbg5" Oct 29 22:33:53.161914 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:53.161892 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Oct 29 22:33:53.162038 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:53.162025 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Oct 29 22:33:53.165900 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:53.165879 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-9s8rp"] Oct 29 22:33:53.173154 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:53.173134 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/b83c5586-450b-46e6-9759-c3d94b50da13-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-hsxj9\" (UID: \"b83c5586-450b-46e6-9759-c3d94b50da13\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-hsxj9" Oct 29 22:33:53.178771 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:53.178741 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-d49c68d56-vpbg5"] Oct 29 22:33:53.216235 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:53.216210 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:33:53.274092 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:53.274035 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/b83c5586-450b-46e6-9759-c3d94b50da13-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-hsxj9\" (UID: \"b83c5586-450b-46e6-9759-c3d94b50da13\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-hsxj9" Oct 29 22:33:53.274092 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:53.274072 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2n4s\" (UniqueName: \"kubernetes.io/projected/e2fb7b48-ff14-42e7-86cc-b77746dcccb0-kube-api-access-l2n4s\") pod \"image-registry-d49c68d56-vpbg5\" (UID: \"e2fb7b48-ff14-42e7-86cc-b77746dcccb0\") " pod="openshift-image-registry/image-registry-d49c68d56-vpbg5" Oct 29 22:33:53.274241 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:53.274105 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfvdv\" (UniqueName: \"kubernetes.io/projected/612c3068-fbdf-4cd3-bf23-4099b112c655-kube-api-access-jfvdv\") pod \"downloads-6dd789c8b7-9s8rp\" (UID: \"612c3068-fbdf-4cd3-bf23-4099b112c655\") " pod="openshift-console/downloads-6dd789c8b7-9s8rp" Oct 29 22:33:53.274241 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:53.274171 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e2fb7b48-ff14-42e7-86cc-b77746dcccb0-image-registry-private-configuration\") pod \"image-registry-d49c68d56-vpbg5\" (UID: \"e2fb7b48-ff14-42e7-86cc-b77746dcccb0\") " pod="openshift-image-registry/image-registry-d49c68d56-vpbg5" Oct 29 22:33:53.274241 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:53.274208 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e2fb7b48-ff14-42e7-86cc-b77746dcccb0-installation-pull-secrets\") pod \"image-registry-d49c68d56-vpbg5\" (UID: \"e2fb7b48-ff14-42e7-86cc-b77746dcccb0\") " pod="openshift-image-registry/image-registry-d49c68d56-vpbg5" Oct 29 22:33:53.274395 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:53.274245 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e2fb7b48-ff14-42e7-86cc-b77746dcccb0-ca-trust-extracted\") pod \"image-registry-d49c68d56-vpbg5\" (UID: \"e2fb7b48-ff14-42e7-86cc-b77746dcccb0\") " pod="openshift-image-registry/image-registry-d49c68d56-vpbg5" Oct 29 22:33:53.274395 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:53.274352 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e2fb7b48-ff14-42e7-86cc-b77746dcccb0-trusted-ca\") pod \"image-registry-d49c68d56-vpbg5\" (UID: \"e2fb7b48-ff14-42e7-86cc-b77746dcccb0\") " pod="openshift-image-registry/image-registry-d49c68d56-vpbg5" Oct 29 22:33:53.274491 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:53.274402 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e2fb7b48-ff14-42e7-86cc-b77746dcccb0-registry-tls\") pod \"image-registry-d49c68d56-vpbg5\" (UID: \"e2fb7b48-ff14-42e7-86cc-b77746dcccb0\") " pod="openshift-image-registry/image-registry-d49c68d56-vpbg5" Oct 29 22:33:53.274491 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:53.274429 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/612c3068-fbdf-4cd3-bf23-4099b112c655-tmp\") pod \"downloads-6dd789c8b7-9s8rp\" (UID: \"612c3068-fbdf-4cd3-bf23-4099b112c655\") " pod="openshift-console/downloads-6dd789c8b7-9s8rp" Oct 29 22:33:53.274491 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:53.274483 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e2fb7b48-ff14-42e7-86cc-b77746dcccb0-bound-sa-token\") pod \"image-registry-d49c68d56-vpbg5\" (UID: \"e2fb7b48-ff14-42e7-86cc-b77746dcccb0\") " pod="openshift-image-registry/image-registry-d49c68d56-vpbg5" Oct 29 22:33:53.274588 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:53.274523 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e2fb7b48-ff14-42e7-86cc-b77746dcccb0-registry-certificates\") pod \"image-registry-d49c68d56-vpbg5\" (UID: \"e2fb7b48-ff14-42e7-86cc-b77746dcccb0\") " pod="openshift-image-registry/image-registry-d49c68d56-vpbg5" Oct 29 22:33:53.276663 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:53.276646 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/b83c5586-450b-46e6-9759-c3d94b50da13-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-hsxj9\" (UID: \"b83c5586-450b-46e6-9759-c3d94b50da13\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-hsxj9" Oct 29 22:33:53.338255 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:53.338232 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-hsxj9" Oct 29 22:33:53.376533 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:53.375757 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e2fb7b48-ff14-42e7-86cc-b77746dcccb0-ca-trust-extracted\") pod \"image-registry-d49c68d56-vpbg5\" (UID: \"e2fb7b48-ff14-42e7-86cc-b77746dcccb0\") " pod="openshift-image-registry/image-registry-d49c68d56-vpbg5" Oct 29 22:33:53.376533 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:53.375873 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e2fb7b48-ff14-42e7-86cc-b77746dcccb0-trusted-ca\") pod \"image-registry-d49c68d56-vpbg5\" (UID: \"e2fb7b48-ff14-42e7-86cc-b77746dcccb0\") " pod="openshift-image-registry/image-registry-d49c68d56-vpbg5" Oct 29 22:33:53.376533 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:53.375940 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e2fb7b48-ff14-42e7-86cc-b77746dcccb0-registry-tls\") pod \"image-registry-d49c68d56-vpbg5\" (UID: \"e2fb7b48-ff14-42e7-86cc-b77746dcccb0\") " pod="openshift-image-registry/image-registry-d49c68d56-vpbg5" Oct 29 22:33:53.376533 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:53.375973 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/612c3068-fbdf-4cd3-bf23-4099b112c655-tmp\") pod \"downloads-6dd789c8b7-9s8rp\" (UID: \"612c3068-fbdf-4cd3-bf23-4099b112c655\") " pod="openshift-console/downloads-6dd789c8b7-9s8rp" Oct 29 22:33:53.376533 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:53.376003 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e2fb7b48-ff14-42e7-86cc-b77746dcccb0-bound-sa-token\") pod \"image-registry-d49c68d56-vpbg5\" (UID: \"e2fb7b48-ff14-42e7-86cc-b77746dcccb0\") " pod="openshift-image-registry/image-registry-d49c68d56-vpbg5" Oct 29 22:33:53.376533 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:53.376031 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e2fb7b48-ff14-42e7-86cc-b77746dcccb0-registry-certificates\") pod \"image-registry-d49c68d56-vpbg5\" (UID: \"e2fb7b48-ff14-42e7-86cc-b77746dcccb0\") " pod="openshift-image-registry/image-registry-d49c68d56-vpbg5" Oct 29 22:33:53.376533 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:53.376089 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2n4s\" (UniqueName: \"kubernetes.io/projected/e2fb7b48-ff14-42e7-86cc-b77746dcccb0-kube-api-access-l2n4s\") pod \"image-registry-d49c68d56-vpbg5\" (UID: \"e2fb7b48-ff14-42e7-86cc-b77746dcccb0\") " pod="openshift-image-registry/image-registry-d49c68d56-vpbg5" Oct 29 22:33:53.376533 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:53.376122 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfvdv\" (UniqueName: \"kubernetes.io/projected/612c3068-fbdf-4cd3-bf23-4099b112c655-kube-api-access-jfvdv\") pod \"downloads-6dd789c8b7-9s8rp\" (UID: \"612c3068-fbdf-4cd3-bf23-4099b112c655\") " pod="openshift-console/downloads-6dd789c8b7-9s8rp" Oct 29 22:33:53.376533 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:53.376147 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e2fb7b48-ff14-42e7-86cc-b77746dcccb0-image-registry-private-configuration\") pod \"image-registry-d49c68d56-vpbg5\" (UID: \"e2fb7b48-ff14-42e7-86cc-b77746dcccb0\") " pod="openshift-image-registry/image-registry-d49c68d56-vpbg5" Oct 29 22:33:53.376533 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:53.376190 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e2fb7b48-ff14-42e7-86cc-b77746dcccb0-installation-pull-secrets\") pod \"image-registry-d49c68d56-vpbg5\" (UID: \"e2fb7b48-ff14-42e7-86cc-b77746dcccb0\") " pod="openshift-image-registry/image-registry-d49c68d56-vpbg5" Oct 29 22:33:53.377099 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:53.376952 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e2fb7b48-ff14-42e7-86cc-b77746dcccb0-trusted-ca\") pod \"image-registry-d49c68d56-vpbg5\" (UID: \"e2fb7b48-ff14-42e7-86cc-b77746dcccb0\") " pod="openshift-image-registry/image-registry-d49c68d56-vpbg5" Oct 29 22:33:53.377099 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:53.376992 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e2fb7b48-ff14-42e7-86cc-b77746dcccb0-ca-trust-extracted\") pod \"image-registry-d49c68d56-vpbg5\" (UID: \"e2fb7b48-ff14-42e7-86cc-b77746dcccb0\") " pod="openshift-image-registry/image-registry-d49c68d56-vpbg5" Oct 29 22:33:53.377323 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:53.377295 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e2fb7b48-ff14-42e7-86cc-b77746dcccb0-registry-certificates\") pod \"image-registry-d49c68d56-vpbg5\" (UID: \"e2fb7b48-ff14-42e7-86cc-b77746dcccb0\") " pod="openshift-image-registry/image-registry-d49c68d56-vpbg5" Oct 29 22:33:53.377715 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:53.377695 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/612c3068-fbdf-4cd3-bf23-4099b112c655-tmp\") pod \"downloads-6dd789c8b7-9s8rp\" (UID: \"612c3068-fbdf-4cd3-bf23-4099b112c655\") " pod="openshift-console/downloads-6dd789c8b7-9s8rp" Oct 29 22:33:53.379268 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:53.379245 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e2fb7b48-ff14-42e7-86cc-b77746dcccb0-image-registry-private-configuration\") pod \"image-registry-d49c68d56-vpbg5\" (UID: \"e2fb7b48-ff14-42e7-86cc-b77746dcccb0\") " pod="openshift-image-registry/image-registry-d49c68d56-vpbg5" Oct 29 22:33:53.379650 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:53.379629 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e2fb7b48-ff14-42e7-86cc-b77746dcccb0-installation-pull-secrets\") pod \"image-registry-d49c68d56-vpbg5\" (UID: \"e2fb7b48-ff14-42e7-86cc-b77746dcccb0\") " pod="openshift-image-registry/image-registry-d49c68d56-vpbg5" Oct 29 22:33:53.379778 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:53.379761 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e2fb7b48-ff14-42e7-86cc-b77746dcccb0-registry-tls\") pod \"image-registry-d49c68d56-vpbg5\" (UID: \"e2fb7b48-ff14-42e7-86cc-b77746dcccb0\") " pod="openshift-image-registry/image-registry-d49c68d56-vpbg5" Oct 29 22:33:53.402021 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:53.401994 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e2fb7b48-ff14-42e7-86cc-b77746dcccb0-bound-sa-token\") pod \"image-registry-d49c68d56-vpbg5\" (UID: \"e2fb7b48-ff14-42e7-86cc-b77746dcccb0\") " pod="openshift-image-registry/image-registry-d49c68d56-vpbg5" Oct 29 22:33:53.402643 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:53.402617 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2n4s\" (UniqueName: \"kubernetes.io/projected/e2fb7b48-ff14-42e7-86cc-b77746dcccb0-kube-api-access-l2n4s\") pod \"image-registry-d49c68d56-vpbg5\" (UID: \"e2fb7b48-ff14-42e7-86cc-b77746dcccb0\") " pod="openshift-image-registry/image-registry-d49c68d56-vpbg5" Oct 29 22:33:53.403512 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:53.403450 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfvdv\" (UniqueName: \"kubernetes.io/projected/612c3068-fbdf-4cd3-bf23-4099b112c655-kube-api-access-jfvdv\") pod \"downloads-6dd789c8b7-9s8rp\" (UID: \"612c3068-fbdf-4cd3-bf23-4099b112c655\") " pod="openshift-console/downloads-6dd789c8b7-9s8rp" Oct 29 22:33:53.416126 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:53.415430 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:33:53.425470 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:53.425451 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-jjz47" Oct 29 22:33:53.458068 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:53.458047 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-9s8rp" Oct 29 22:33:53.461796 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:53.461778 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-d49c68d56-vpbg5" Oct 29 22:33:53.524833 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:53.524620 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:33:53.595995 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:53.595968 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-khlhx" Oct 29 22:33:53.632745 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:53.632711 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-hsxj9"] Oct 29 22:33:53.636042 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:53.636017 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb83c5586_450b_46e6_9759_c3d94b50da13.slice/crio-7e21a538a58abe2deaaa312e73a61e7e418907693539642dfd0f7ea9cfe678a0 WatchSource:0}: Error finding container 7e21a538a58abe2deaaa312e73a61e7e418907693539642dfd0f7ea9cfe678a0: Status 404 returned error can't find the container with id 7e21a538a58abe2deaaa312e73a61e7e418907693539642dfd0f7ea9cfe678a0 Oct 29 22:33:53.759119 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:53.759085 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-9s8rp"] Oct 29 22:33:53.762503 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:53.762473 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod612c3068_fbdf_4cd3_bf23_4099b112c655.slice/crio-17dfb4e4a431ba2b3641ab18164fa1e80e16bc67e4fd028393817a26041ba70a WatchSource:0}: Error finding container 17dfb4e4a431ba2b3641ab18164fa1e80e16bc67e4fd028393817a26041ba70a: Status 404 returned error can't find the container with id 17dfb4e4a431ba2b3641ab18164fa1e80e16bc67e4fd028393817a26041ba70a Oct 29 22:33:53.774114 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:53.774091 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-d49c68d56-vpbg5"] Oct 29 22:33:53.777936 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:53.777912 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2fb7b48_ff14_42e7_86cc_b77746dcccb0.slice/crio-3aadd7990cb8489529997a1de16126792041476e636f34b5e462d68a1a5ae1eb WatchSource:0}: Error finding container 3aadd7990cb8489529997a1de16126792041476e636f34b5e462d68a1a5ae1eb: Status 404 returned error can't find the container with id 3aadd7990cb8489529997a1de16126792041476e636f34b5e462d68a1a5ae1eb Oct 29 22:33:53.833186 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:53.833156 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:33:53.943047 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:53.943015 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:33:54.198199 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:54.198083 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-84866f86bd-2v6d7" Oct 29 22:33:54.487536 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:54.487507 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-d49c68d56-vpbg5" event={"ID":"e2fb7b48-ff14-42e7-86cc-b77746dcccb0","Type":"ContainerStarted","Data":"645a5f4eda581c342fe5213130ac59733e09063f2ca48267585e8f118eddd37e"} Oct 29 22:33:54.487655 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:54.487557 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-d49c68d56-vpbg5" event={"ID":"e2fb7b48-ff14-42e7-86cc-b77746dcccb0","Type":"ContainerStarted","Data":"3aadd7990cb8489529997a1de16126792041476e636f34b5e462d68a1a5ae1eb"} Oct 29 22:33:54.487655 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:54.487625 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-d49c68d56-vpbg5" Oct 29 22:33:54.488893 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:54.488865 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-9s8rp" event={"ID":"612c3068-fbdf-4cd3-bf23-4099b112c655","Type":"ContainerStarted","Data":"17dfb4e4a431ba2b3641ab18164fa1e80e16bc67e4fd028393817a26041ba70a"} Oct 29 22:33:54.490215 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:54.490185 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-hsxj9" event={"ID":"b83c5586-450b-46e6-9759-c3d94b50da13","Type":"ContainerStarted","Data":"7e21a538a58abe2deaaa312e73a61e7e418907693539642dfd0f7ea9cfe678a0"} Oct 29 22:33:54.529393 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:54.529321 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-registry-d49c68d56-vpbg5" podStartSLOduration=1.529306927 podStartE2EDuration="1.529306927s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:33:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:33:54.528519814 +0000 UTC m=+15.311733259" watchObservedRunningTime="2026-10-29 22:33:54.529306927 +0000 UTC m=+15.312520353" Oct 29 22:33:55.496626 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:55.496582 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-hsxj9" event={"ID":"b83c5586-450b-46e6-9759-c3d94b50da13","Type":"ContainerStarted","Data":"0b0db6fe495f3b37b460bd5ec69b7283ffb5cb9253beb3b13b141e16126b0d1a"} Oct 29 22:33:55.496973 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:55.496953 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-hsxj9" Oct 29 22:33:55.502510 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:55.502485 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-hsxj9" Oct 29 22:33:55.554542 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:55.554487 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-hsxj9" podStartSLOduration=2.731844727 podStartE2EDuration="3.554470101s" totalImagesPullingTime="822.625374ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:33:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:33:55.553667364 +0000 UTC m=+16.336880791" watchObservedRunningTime="2026-10-29 22:33:55.554470101 +0000 UTC m=+16.337683530" Oct 29 22:33:56.422831 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:56.422773 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:33:56.527766 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:56.527728 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-jz2pw"] Oct 29 22:33:56.530677 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:56.530650 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-jz2pw" Oct 29 22:33:56.549838 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:56.549799 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:33:56.558654 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:56.558630 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Oct 29 22:33:56.559022 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:56.558787 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Oct 29 22:33:56.559022 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:56.558847 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-bnn57\"" Oct 29 22:33:56.559022 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:56.558915 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 29 22:33:56.560579 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:56.560556 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-jz2pw"] Oct 29 22:33:56.604134 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:56.604097 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2229j\" (UniqueName: \"kubernetes.io/projected/4f1d0a5c-7cc7-4f1f-88d1-f1b1e6945963-kube-api-access-2229j\") pod \"prometheus-operator-9cd89b74c-jz2pw\" (UID: \"4f1d0a5c-7cc7-4f1f-88d1-f1b1e6945963\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-jz2pw" Oct 29 22:33:56.604282 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:56.604218 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4f1d0a5c-7cc7-4f1f-88d1-f1b1e6945963-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-jz2pw\" (UID: \"4f1d0a5c-7cc7-4f1f-88d1-f1b1e6945963\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-jz2pw" Oct 29 22:33:56.604379 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:56.604260 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/4f1d0a5c-7cc7-4f1f-88d1-f1b1e6945963-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-jz2pw\" (UID: \"4f1d0a5c-7cc7-4f1f-88d1-f1b1e6945963\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-jz2pw" Oct 29 22:33:56.604379 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:56.604314 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4f1d0a5c-7cc7-4f1f-88d1-f1b1e6945963-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-jz2pw\" (UID: \"4f1d0a5c-7cc7-4f1f-88d1-f1b1e6945963\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-jz2pw" Oct 29 22:33:56.705640 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:56.705612 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/4f1d0a5c-7cc7-4f1f-88d1-f1b1e6945963-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-jz2pw\" (UID: \"4f1d0a5c-7cc7-4f1f-88d1-f1b1e6945963\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-jz2pw" Oct 29 22:33:56.705825 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:56.705663 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4f1d0a5c-7cc7-4f1f-88d1-f1b1e6945963-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-jz2pw\" (UID: \"4f1d0a5c-7cc7-4f1f-88d1-f1b1e6945963\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-jz2pw" Oct 29 22:33:56.705825 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:56.705722 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2229j\" (UniqueName: \"kubernetes.io/projected/4f1d0a5c-7cc7-4f1f-88d1-f1b1e6945963-kube-api-access-2229j\") pod \"prometheus-operator-9cd89b74c-jz2pw\" (UID: \"4f1d0a5c-7cc7-4f1f-88d1-f1b1e6945963\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-jz2pw" Oct 29 22:33:56.705825 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:56.705798 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4f1d0a5c-7cc7-4f1f-88d1-f1b1e6945963-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-jz2pw\" (UID: \"4f1d0a5c-7cc7-4f1f-88d1-f1b1e6945963\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-jz2pw" Oct 29 22:33:56.705990 ip-10-0-140-74 kubenswrapper[5057]: E1029 22:33:56.705825 5057 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found Oct 29 22:33:56.705990 ip-10-0-140-74 kubenswrapper[5057]: E1029 22:33:56.705968 5057 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4f1d0a5c-7cc7-4f1f-88d1-f1b1e6945963-prometheus-operator-tls podName:4f1d0a5c-7cc7-4f1f-88d1-f1b1e6945963 nodeName:}" failed. No retries permitted until 2026-10-29 22:33:57.205877541 +0000 UTC m=+17.989090967 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/4f1d0a5c-7cc7-4f1f-88d1-f1b1e6945963-prometheus-operator-tls") pod "prometheus-operator-9cd89b74c-jz2pw" (UID: "4f1d0a5c-7cc7-4f1f-88d1-f1b1e6945963") : secret "prometheus-operator-tls" not found Oct 29 22:33:56.706824 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:56.706804 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4f1d0a5c-7cc7-4f1f-88d1-f1b1e6945963-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-jz2pw\" (UID: \"4f1d0a5c-7cc7-4f1f-88d1-f1b1e6945963\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-jz2pw" Oct 29 22:33:56.708701 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:56.708676 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4f1d0a5c-7cc7-4f1f-88d1-f1b1e6945963-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-jz2pw\" (UID: \"4f1d0a5c-7cc7-4f1f-88d1-f1b1e6945963\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-jz2pw" Oct 29 22:33:56.727572 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:56.727532 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2229j\" (UniqueName: \"kubernetes.io/projected/4f1d0a5c-7cc7-4f1f-88d1-f1b1e6945963-kube-api-access-2229j\") pod \"prometheus-operator-9cd89b74c-jz2pw\" (UID: \"4f1d0a5c-7cc7-4f1f-88d1-f1b1e6945963\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-jz2pw" Oct 29 22:33:56.855755 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:56.855716 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:33:57.210653 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:57.210614 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/4f1d0a5c-7cc7-4f1f-88d1-f1b1e6945963-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-jz2pw\" (UID: \"4f1d0a5c-7cc7-4f1f-88d1-f1b1e6945963\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-jz2pw" Oct 29 22:33:57.213447 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:57.213420 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/4f1d0a5c-7cc7-4f1f-88d1-f1b1e6945963-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-jz2pw\" (UID: \"4f1d0a5c-7cc7-4f1f-88d1-f1b1e6945963\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-jz2pw" Oct 29 22:33:57.444387 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:57.444346 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-jz2pw" Oct 29 22:33:57.749132 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:57.749090 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-jz2pw"] Oct 29 22:33:57.753135 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:57.753103 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f1d0a5c_7cc7_4f1f_88d1_f1b1e6945963.slice/crio-bd10b7045fe13ce20670ab779a6a03dc0f848023f895e55fd8cc23e9bdc95e6d WatchSource:0}: Error finding container bd10b7045fe13ce20670ab779a6a03dc0f848023f895e55fd8cc23e9bdc95e6d: Status 404 returned error can't find the container with id bd10b7045fe13ce20670ab779a6a03dc0f848023f895e55fd8cc23e9bdc95e6d Oct 29 22:33:57.837374 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:57.837344 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-68ff86fd-nb5ft"] Oct 29 22:33:57.840724 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:57.840700 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-68ff86fd-nb5ft" Oct 29 22:33:57.853497 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:57.853464 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Oct 29 22:33:57.863429 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:57.863405 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Oct 29 22:33:57.863532 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:57.863408 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Oct 29 22:33:57.863923 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:57.863905 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Oct 29 22:33:57.864024 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:57.863956 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-dclgw\"" Oct 29 22:33:57.864024 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:57.863997 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Oct 29 22:33:57.865768 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:57.865747 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-68ff86fd-nb5ft"] Oct 29 22:33:57.917409 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:57.917315 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/166d00be-a17a-41ba-9e75-7b7a4217b788-tmp\") pod \"console-68ff86fd-nb5ft\" (UID: \"166d00be-a17a-41ba-9e75-7b7a4217b788\") " pod="openshift-console/console-68ff86fd-nb5ft" Oct 29 22:33:57.917409 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:57.917373 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7lfv\" (UniqueName: \"kubernetes.io/projected/166d00be-a17a-41ba-9e75-7b7a4217b788-kube-api-access-q7lfv\") pod \"console-68ff86fd-nb5ft\" (UID: \"166d00be-a17a-41ba-9e75-7b7a4217b788\") " pod="openshift-console/console-68ff86fd-nb5ft" Oct 29 22:33:57.917580 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:57.917508 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/166d00be-a17a-41ba-9e75-7b7a4217b788-console-config\") pod \"console-68ff86fd-nb5ft\" (UID: \"166d00be-a17a-41ba-9e75-7b7a4217b788\") " pod="openshift-console/console-68ff86fd-nb5ft" Oct 29 22:33:57.917580 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:57.917547 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/166d00be-a17a-41ba-9e75-7b7a4217b788-service-ca\") pod \"console-68ff86fd-nb5ft\" (UID: \"166d00be-a17a-41ba-9e75-7b7a4217b788\") " pod="openshift-console/console-68ff86fd-nb5ft" Oct 29 22:33:57.917580 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:57.917573 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/166d00be-a17a-41ba-9e75-7b7a4217b788-oauth-serving-cert\") pod \"console-68ff86fd-nb5ft\" (UID: \"166d00be-a17a-41ba-9e75-7b7a4217b788\") " pod="openshift-console/console-68ff86fd-nb5ft" Oct 29 22:33:57.917711 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:57.917619 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/166d00be-a17a-41ba-9e75-7b7a4217b788-console-serving-cert\") pod \"console-68ff86fd-nb5ft\" (UID: \"166d00be-a17a-41ba-9e75-7b7a4217b788\") " pod="openshift-console/console-68ff86fd-nb5ft" Oct 29 22:33:57.917711 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:57.917633 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/166d00be-a17a-41ba-9e75-7b7a4217b788-console-oauth-config\") pod \"console-68ff86fd-nb5ft\" (UID: \"166d00be-a17a-41ba-9e75-7b7a4217b788\") " pod="openshift-console/console-68ff86fd-nb5ft" Oct 29 22:33:58.018474 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:58.018391 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/166d00be-a17a-41ba-9e75-7b7a4217b788-console-serving-cert\") pod \"console-68ff86fd-nb5ft\" (UID: \"166d00be-a17a-41ba-9e75-7b7a4217b788\") " pod="openshift-console/console-68ff86fd-nb5ft" Oct 29 22:33:58.018474 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:58.018432 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/166d00be-a17a-41ba-9e75-7b7a4217b788-console-oauth-config\") pod \"console-68ff86fd-nb5ft\" (UID: \"166d00be-a17a-41ba-9e75-7b7a4217b788\") " pod="openshift-console/console-68ff86fd-nb5ft" Oct 29 22:33:58.018668 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:58.018484 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/166d00be-a17a-41ba-9e75-7b7a4217b788-tmp\") pod \"console-68ff86fd-nb5ft\" (UID: \"166d00be-a17a-41ba-9e75-7b7a4217b788\") " pod="openshift-console/console-68ff86fd-nb5ft" Oct 29 22:33:58.018668 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:58.018500 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7lfv\" (UniqueName: \"kubernetes.io/projected/166d00be-a17a-41ba-9e75-7b7a4217b788-kube-api-access-q7lfv\") pod \"console-68ff86fd-nb5ft\" (UID: \"166d00be-a17a-41ba-9e75-7b7a4217b788\") " pod="openshift-console/console-68ff86fd-nb5ft" Oct 29 22:33:58.018668 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:58.018529 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/166d00be-a17a-41ba-9e75-7b7a4217b788-console-config\") pod \"console-68ff86fd-nb5ft\" (UID: \"166d00be-a17a-41ba-9e75-7b7a4217b788\") " pod="openshift-console/console-68ff86fd-nb5ft" Oct 29 22:33:58.018668 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:58.018623 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/166d00be-a17a-41ba-9e75-7b7a4217b788-service-ca\") pod \"console-68ff86fd-nb5ft\" (UID: \"166d00be-a17a-41ba-9e75-7b7a4217b788\") " pod="openshift-console/console-68ff86fd-nb5ft" Oct 29 22:33:58.018865 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:58.018670 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/166d00be-a17a-41ba-9e75-7b7a4217b788-oauth-serving-cert\") pod \"console-68ff86fd-nb5ft\" (UID: \"166d00be-a17a-41ba-9e75-7b7a4217b788\") " pod="openshift-console/console-68ff86fd-nb5ft" Oct 29 22:33:58.018967 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:58.018941 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/166d00be-a17a-41ba-9e75-7b7a4217b788-tmp\") pod \"console-68ff86fd-nb5ft\" (UID: \"166d00be-a17a-41ba-9e75-7b7a4217b788\") " pod="openshift-console/console-68ff86fd-nb5ft" Oct 29 22:33:58.019420 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:58.019397 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/166d00be-a17a-41ba-9e75-7b7a4217b788-console-config\") pod \"console-68ff86fd-nb5ft\" (UID: \"166d00be-a17a-41ba-9e75-7b7a4217b788\") " pod="openshift-console/console-68ff86fd-nb5ft" Oct 29 22:33:58.019490 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:58.019446 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/166d00be-a17a-41ba-9e75-7b7a4217b788-oauth-serving-cert\") pod \"console-68ff86fd-nb5ft\" (UID: \"166d00be-a17a-41ba-9e75-7b7a4217b788\") " pod="openshift-console/console-68ff86fd-nb5ft" Oct 29 22:33:58.019982 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:58.019961 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/166d00be-a17a-41ba-9e75-7b7a4217b788-service-ca\") pod \"console-68ff86fd-nb5ft\" (UID: \"166d00be-a17a-41ba-9e75-7b7a4217b788\") " pod="openshift-console/console-68ff86fd-nb5ft" Oct 29 22:33:58.021140 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:58.021115 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/166d00be-a17a-41ba-9e75-7b7a4217b788-console-oauth-config\") pod \"console-68ff86fd-nb5ft\" (UID: \"166d00be-a17a-41ba-9e75-7b7a4217b788\") " pod="openshift-console/console-68ff86fd-nb5ft" Oct 29 22:33:58.021317 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:58.021295 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/166d00be-a17a-41ba-9e75-7b7a4217b788-console-serving-cert\") pod \"console-68ff86fd-nb5ft\" (UID: \"166d00be-a17a-41ba-9e75-7b7a4217b788\") " pod="openshift-console/console-68ff86fd-nb5ft" Oct 29 22:33:58.040241 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:58.040214 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7lfv\" (UniqueName: \"kubernetes.io/projected/166d00be-a17a-41ba-9e75-7b7a4217b788-kube-api-access-q7lfv\") pod \"console-68ff86fd-nb5ft\" (UID: \"166d00be-a17a-41ba-9e75-7b7a4217b788\") " pod="openshift-console/console-68ff86fd-nb5ft" Oct 29 22:33:58.153418 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:58.153381 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-68ff86fd-nb5ft" Oct 29 22:33:58.229376 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:58.228593 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:33:58.348679 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:58.348565 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:33:58.458049 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:58.457644 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-68ff86fd-nb5ft"] Oct 29 22:33:58.460787 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:33:58.460757 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod166d00be_a17a_41ba_9e75_7b7a4217b788.slice/crio-6d0fc88b264bca0b71bf97f6d9bc334e78a294f862bc08f55769c40eeb924071 WatchSource:0}: Error finding container 6d0fc88b264bca0b71bf97f6d9bc334e78a294f862bc08f55769c40eeb924071: Status 404 returned error can't find the container with id 6d0fc88b264bca0b71bf97f6d9bc334e78a294f862bc08f55769c40eeb924071 Oct 29 22:33:58.521733 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:58.521701 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-68ff86fd-nb5ft" event={"ID":"166d00be-a17a-41ba-9e75-7b7a4217b788","Type":"ContainerStarted","Data":"6d0fc88b264bca0b71bf97f6d9bc334e78a294f862bc08f55769c40eeb924071"} Oct 29 22:33:58.523335 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:58.523305 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-jz2pw" event={"ID":"4f1d0a5c-7cc7-4f1f-88d1-f1b1e6945963","Type":"ContainerStarted","Data":"bd10b7045fe13ce20670ab779a6a03dc0f848023f895e55fd8cc23e9bdc95e6d"} Oct 29 22:33:59.512394 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:59.512349 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:33:59.532682 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:59.532644 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-jz2pw" event={"ID":"4f1d0a5c-7cc7-4f1f-88d1-f1b1e6945963","Type":"ContainerStarted","Data":"6f388c4d2abad0e2f86a0adeabf4e2c20303a1c3c79be7bb6a7b0e9dae52fefb"} Oct 29 22:33:59.532823 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:59.532709 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-jz2pw" event={"ID":"4f1d0a5c-7cc7-4f1f-88d1-f1b1e6945963","Type":"ContainerStarted","Data":"bf1da92d7403695b535c29b0f4bc71b0e18fa2a3f5dd7ce516859101f4b61023"} Oct 29 22:33:59.572446 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:59.572400 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-9cd89b74c-jz2pw" podStartSLOduration=2.241205657 podStartE2EDuration="3.572387042s" totalImagesPullingTime="1.331181385s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:33:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:33:59.57003193 +0000 UTC m=+20.353245357" watchObservedRunningTime="2026-10-29 22:33:59.572387042 +0000 UTC m=+20.355600487" Oct 29 22:33:59.634913 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:59.634878 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:33:59.979364 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:33:59.979305 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:34:01.547626 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:01.547580 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-68ff86fd-nb5ft" event={"ID":"166d00be-a17a-41ba-9e75-7b7a4217b788","Type":"ContainerStarted","Data":"eddaad68ee7f78b328416cbc237c3a6ba7e4aff3b76387ac5bbe69dddcaef492"} Oct 29 22:34:01.587755 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:01.587686 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-68ff86fd-nb5ft" podStartSLOduration=1.652477051 podStartE2EDuration="4.587666193s" totalImagesPullingTime="2.935189142s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:33:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:34:01.586282773 +0000 UTC m=+22.369496221" watchObservedRunningTime="2026-10-29 22:34:01.587666193 +0000 UTC m=+22.370879620" Oct 29 22:34:01.894860 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:01.894824 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:34:02.088822 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:02.088789 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:34:02.130267 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:02.130234 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-qjcwh"] Oct 29 22:34:02.182887 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:02.179571 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-qjcwh"] Oct 29 22:34:02.182887 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:02.179635 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-pcsj6"] Oct 29 22:34:02.188844 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:02.188818 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-qjcwh" Oct 29 22:34:02.203757 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:02.200919 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-pfvrr"] Oct 29 22:34:02.203757 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:02.201326 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-pcsj6" Oct 29 22:34:02.203757 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:02.202497 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Oct 29 22:34:02.203757 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:02.202589 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-lq2sd\"" Oct 29 22:34:02.203757 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:02.202636 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Oct 29 22:34:02.205642 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:02.205614 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:34:02.214493 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:02.214459 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 29 22:34:02.224183 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:02.224163 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-pfvrr"] Oct 29 22:34:02.224289 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:02.224262 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-pfvrr" Oct 29 22:34:02.261696 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:02.261649 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/3cdc2c68-0a75-426d-ab02-ab7f6af525be-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-qjcwh\" (UID: \"3cdc2c68-0a75-426d-ab02-ab7f6af525be\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-qjcwh" Oct 29 22:34:02.261696 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:02.261678 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3cdc2c68-0a75-426d-ab02-ab7f6af525be-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-qjcwh\" (UID: \"3cdc2c68-0a75-426d-ab02-ab7f6af525be\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-qjcwh" Oct 29 22:34:02.262010 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:02.261706 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a5b6a0e8-ad7a-4a47-a333-48e0d3b9eced-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-pcsj6\" (UID: \"a5b6a0e8-ad7a-4a47-a333-48e0d3b9eced\") " pod="openshift-monitoring/node-exporter-pcsj6" Oct 29 22:34:02.262010 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:02.261776 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/689c465f-09b8-42cb-a382-5a59526bb3b5-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-pfvrr\" (UID: \"689c465f-09b8-42cb-a382-5a59526bb3b5\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-pfvrr" Oct 29 22:34:02.262010 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:02.261908 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3cdc2c68-0a75-426d-ab02-ab7f6af525be-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-qjcwh\" (UID: \"3cdc2c68-0a75-426d-ab02-ab7f6af525be\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-qjcwh" Oct 29 22:34:02.262010 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:02.261952 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/689c465f-09b8-42cb-a382-5a59526bb3b5-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-pfvrr\" (UID: \"689c465f-09b8-42cb-a382-5a59526bb3b5\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-pfvrr" Oct 29 22:34:02.262010 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:02.261989 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a5b6a0e8-ad7a-4a47-a333-48e0d3b9eced-node-exporter-textfile\") pod \"node-exporter-pcsj6\" (UID: \"a5b6a0e8-ad7a-4a47-a333-48e0d3b9eced\") " pod="openshift-monitoring/node-exporter-pcsj6" Oct 29 22:34:02.262010 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:02.262013 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/689c465f-09b8-42cb-a382-5a59526bb3b5-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-pfvrr\" (UID: \"689c465f-09b8-42cb-a382-5a59526bb3b5\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-pfvrr" Oct 29 22:34:02.262261 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:02.262084 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/689c465f-09b8-42cb-a382-5a59526bb3b5-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-pfvrr\" (UID: \"689c465f-09b8-42cb-a382-5a59526bb3b5\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-pfvrr" Oct 29 22:34:02.262261 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:02.262144 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a5b6a0e8-ad7a-4a47-a333-48e0d3b9eced-sys\") pod \"node-exporter-pcsj6\" (UID: \"a5b6a0e8-ad7a-4a47-a333-48e0d3b9eced\") " pod="openshift-monitoring/node-exporter-pcsj6" Oct 29 22:34:02.262261 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:02.262184 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a5b6a0e8-ad7a-4a47-a333-48e0d3b9eced-node-exporter-accelerators-collector-config\") pod \"node-exporter-pcsj6\" (UID: \"a5b6a0e8-ad7a-4a47-a333-48e0d3b9eced\") " pod="openshift-monitoring/node-exporter-pcsj6" Oct 29 22:34:02.262261 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:02.262209 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whh2s\" (UniqueName: \"kubernetes.io/projected/a5b6a0e8-ad7a-4a47-a333-48e0d3b9eced-kube-api-access-whh2s\") pod \"node-exporter-pcsj6\" (UID: \"a5b6a0e8-ad7a-4a47-a333-48e0d3b9eced\") " pod="openshift-monitoring/node-exporter-pcsj6" Oct 29 22:34:02.262261 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:02.262254 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gd8hw\" (UniqueName: \"kubernetes.io/projected/689c465f-09b8-42cb-a382-5a59526bb3b5-kube-api-access-gd8hw\") pod \"kube-state-metrics-5dc8d7d89b-pfvrr\" (UID: \"689c465f-09b8-42cb-a382-5a59526bb3b5\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-pfvrr" Oct 29 22:34:02.262509 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:02.262347 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a5b6a0e8-ad7a-4a47-a333-48e0d3b9eced-node-exporter-wtmp\") pod \"node-exporter-pcsj6\" (UID: \"a5b6a0e8-ad7a-4a47-a333-48e0d3b9eced\") " pod="openshift-monitoring/node-exporter-pcsj6" Oct 29 22:34:02.262509 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:02.262376 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a5b6a0e8-ad7a-4a47-a333-48e0d3b9eced-root\") pod \"node-exporter-pcsj6\" (UID: \"a5b6a0e8-ad7a-4a47-a333-48e0d3b9eced\") " pod="openshift-monitoring/node-exporter-pcsj6" Oct 29 22:34:02.262509 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:02.262423 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gpf2j\" (UniqueName: \"kubernetes.io/projected/3cdc2c68-0a75-426d-ab02-ab7f6af525be-kube-api-access-gpf2j\") pod \"openshift-state-metrics-7bd7489957-qjcwh\" (UID: \"3cdc2c68-0a75-426d-ab02-ab7f6af525be\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-qjcwh" Oct 29 22:34:02.262509 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:02.262449 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/689c465f-09b8-42cb-a382-5a59526bb3b5-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-pfvrr\" (UID: \"689c465f-09b8-42cb-a382-5a59526bb3b5\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-pfvrr" Oct 29 22:34:02.262509 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:02.262477 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a5b6a0e8-ad7a-4a47-a333-48e0d3b9eced-node-exporter-tls\") pod \"node-exporter-pcsj6\" (UID: \"a5b6a0e8-ad7a-4a47-a333-48e0d3b9eced\") " pod="openshift-monitoring/node-exporter-pcsj6" Oct 29 22:34:02.262509 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:02.262499 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a5b6a0e8-ad7a-4a47-a333-48e0d3b9eced-metrics-client-ca\") pod \"node-exporter-pcsj6\" (UID: \"a5b6a0e8-ad7a-4a47-a333-48e0d3b9eced\") " pod="openshift-monitoring/node-exporter-pcsj6" Oct 29 22:34:02.302727 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:02.302705 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Oct 29 22:34:02.302867 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:02.302750 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 29 22:34:02.302867 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:02.302852 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Oct 29 22:34:02.302985 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:02.302874 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 29 22:34:02.302985 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:02.302905 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-v4zx9\"" Oct 29 22:34:02.302985 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:02.302956 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-pd9xt\"" Oct 29 22:34:02.314713 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:02.314687 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Oct 29 22:34:02.362938 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:02.362904 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a5b6a0e8-ad7a-4a47-a333-48e0d3b9eced-node-exporter-wtmp\") pod \"node-exporter-pcsj6\" (UID: \"a5b6a0e8-ad7a-4a47-a333-48e0d3b9eced\") " pod="openshift-monitoring/node-exporter-pcsj6" Oct 29 22:34:02.363048 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:02.362951 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a5b6a0e8-ad7a-4a47-a333-48e0d3b9eced-root\") pod \"node-exporter-pcsj6\" (UID: \"a5b6a0e8-ad7a-4a47-a333-48e0d3b9eced\") " pod="openshift-monitoring/node-exporter-pcsj6" Oct 29 22:34:02.363048 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:02.362994 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpf2j\" (UniqueName: \"kubernetes.io/projected/3cdc2c68-0a75-426d-ab02-ab7f6af525be-kube-api-access-gpf2j\") pod \"openshift-state-metrics-7bd7489957-qjcwh\" (UID: \"3cdc2c68-0a75-426d-ab02-ab7f6af525be\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-qjcwh" Oct 29 22:34:02.363048 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:02.363023 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/689c465f-09b8-42cb-a382-5a59526bb3b5-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-pfvrr\" (UID: \"689c465f-09b8-42cb-a382-5a59526bb3b5\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-pfvrr" Oct 29 22:34:02.363164 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:02.363053 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a5b6a0e8-ad7a-4a47-a333-48e0d3b9eced-node-exporter-tls\") pod \"node-exporter-pcsj6\" (UID: \"a5b6a0e8-ad7a-4a47-a333-48e0d3b9eced\") " pod="openshift-monitoring/node-exporter-pcsj6" Oct 29 22:34:02.363164 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:02.363076 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a5b6a0e8-ad7a-4a47-a333-48e0d3b9eced-metrics-client-ca\") pod \"node-exporter-pcsj6\" (UID: \"a5b6a0e8-ad7a-4a47-a333-48e0d3b9eced\") " pod="openshift-monitoring/node-exporter-pcsj6" Oct 29 22:34:02.363257 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:02.363230 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/3cdc2c68-0a75-426d-ab02-ab7f6af525be-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-qjcwh\" (UID: \"3cdc2c68-0a75-426d-ab02-ab7f6af525be\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-qjcwh" Oct 29 22:34:02.363312 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:02.363258 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3cdc2c68-0a75-426d-ab02-ab7f6af525be-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-qjcwh\" (UID: \"3cdc2c68-0a75-426d-ab02-ab7f6af525be\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-qjcwh" Oct 29 22:34:02.363312 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:02.363284 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a5b6a0e8-ad7a-4a47-a333-48e0d3b9eced-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-pcsj6\" (UID: \"a5b6a0e8-ad7a-4a47-a333-48e0d3b9eced\") " pod="openshift-monitoring/node-exporter-pcsj6" Oct 29 22:34:02.363430 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:02.363312 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/689c465f-09b8-42cb-a382-5a59526bb3b5-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-pfvrr\" (UID: \"689c465f-09b8-42cb-a382-5a59526bb3b5\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-pfvrr" Oct 29 22:34:02.363430 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:02.363364 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3cdc2c68-0a75-426d-ab02-ab7f6af525be-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-qjcwh\" (UID: \"3cdc2c68-0a75-426d-ab02-ab7f6af525be\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-qjcwh" Oct 29 22:34:02.363532 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:02.363435 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a5b6a0e8-ad7a-4a47-a333-48e0d3b9eced-root\") pod \"node-exporter-pcsj6\" (UID: \"a5b6a0e8-ad7a-4a47-a333-48e0d3b9eced\") " pod="openshift-monitoring/node-exporter-pcsj6" Oct 29 22:34:02.363532 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:02.363485 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/689c465f-09b8-42cb-a382-5a59526bb3b5-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-pfvrr\" (UID: \"689c465f-09b8-42cb-a382-5a59526bb3b5\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-pfvrr" Oct 29 22:34:02.363532 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:02.363523 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a5b6a0e8-ad7a-4a47-a333-48e0d3b9eced-node-exporter-textfile\") pod \"node-exporter-pcsj6\" (UID: \"a5b6a0e8-ad7a-4a47-a333-48e0d3b9eced\") " pod="openshift-monitoring/node-exporter-pcsj6" Oct 29 22:34:02.363649 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:02.363547 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/689c465f-09b8-42cb-a382-5a59526bb3b5-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-pfvrr\" (UID: \"689c465f-09b8-42cb-a382-5a59526bb3b5\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-pfvrr" Oct 29 22:34:02.363649 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:02.363586 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/689c465f-09b8-42cb-a382-5a59526bb3b5-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-pfvrr\" (UID: \"689c465f-09b8-42cb-a382-5a59526bb3b5\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-pfvrr" Oct 29 22:34:02.363649 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:02.363633 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a5b6a0e8-ad7a-4a47-a333-48e0d3b9eced-sys\") pod \"node-exporter-pcsj6\" (UID: \"a5b6a0e8-ad7a-4a47-a333-48e0d3b9eced\") " pod="openshift-monitoring/node-exporter-pcsj6" Oct 29 22:34:02.363755 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:02.363682 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a5b6a0e8-ad7a-4a47-a333-48e0d3b9eced-node-exporter-accelerators-collector-config\") pod \"node-exporter-pcsj6\" (UID: \"a5b6a0e8-ad7a-4a47-a333-48e0d3b9eced\") " pod="openshift-monitoring/node-exporter-pcsj6" Oct 29 22:34:02.363755 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:02.363723 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whh2s\" (UniqueName: \"kubernetes.io/projected/a5b6a0e8-ad7a-4a47-a333-48e0d3b9eced-kube-api-access-whh2s\") pod \"node-exporter-pcsj6\" (UID: \"a5b6a0e8-ad7a-4a47-a333-48e0d3b9eced\") " pod="openshift-monitoring/node-exporter-pcsj6" Oct 29 22:34:02.363825 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:02.363752 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gd8hw\" (UniqueName: \"kubernetes.io/projected/689c465f-09b8-42cb-a382-5a59526bb3b5-kube-api-access-gd8hw\") pod \"kube-state-metrics-5dc8d7d89b-pfvrr\" (UID: \"689c465f-09b8-42cb-a382-5a59526bb3b5\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-pfvrr" Oct 29 22:34:02.364026 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:02.364000 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3cdc2c68-0a75-426d-ab02-ab7f6af525be-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-qjcwh\" (UID: \"3cdc2c68-0a75-426d-ab02-ab7f6af525be\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-qjcwh" Oct 29 22:34:02.364845 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:02.364825 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a5b6a0e8-ad7a-4a47-a333-48e0d3b9eced-sys\") pod \"node-exporter-pcsj6\" (UID: \"a5b6a0e8-ad7a-4a47-a333-48e0d3b9eced\") " pod="openshift-monitoring/node-exporter-pcsj6" Oct 29 22:34:02.364956 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:02.364900 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a5b6a0e8-ad7a-4a47-a333-48e0d3b9eced-metrics-client-ca\") pod \"node-exporter-pcsj6\" (UID: \"a5b6a0e8-ad7a-4a47-a333-48e0d3b9eced\") " pod="openshift-monitoring/node-exporter-pcsj6" Oct 29 22:34:02.365182 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:02.365158 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a5b6a0e8-ad7a-4a47-a333-48e0d3b9eced-node-exporter-wtmp\") pod \"node-exporter-pcsj6\" (UID: \"a5b6a0e8-ad7a-4a47-a333-48e0d3b9eced\") " pod="openshift-monitoring/node-exporter-pcsj6" Oct 29 22:34:02.367136 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:02.365375 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/689c465f-09b8-42cb-a382-5a59526bb3b5-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-pfvrr\" (UID: \"689c465f-09b8-42cb-a382-5a59526bb3b5\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-pfvrr" Oct 29 22:34:02.367246 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:02.365595 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a5b6a0e8-ad7a-4a47-a333-48e0d3b9eced-node-exporter-accelerators-collector-config\") pod \"node-exporter-pcsj6\" (UID: \"a5b6a0e8-ad7a-4a47-a333-48e0d3b9eced\") " pod="openshift-monitoring/node-exporter-pcsj6" Oct 29 22:34:02.367246 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:02.365610 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a5b6a0e8-ad7a-4a47-a333-48e0d3b9eced-node-exporter-textfile\") pod \"node-exporter-pcsj6\" (UID: \"a5b6a0e8-ad7a-4a47-a333-48e0d3b9eced\") " pod="openshift-monitoring/node-exporter-pcsj6" Oct 29 22:34:02.367246 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:02.366469 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/689c465f-09b8-42cb-a382-5a59526bb3b5-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-pfvrr\" (UID: \"689c465f-09b8-42cb-a382-5a59526bb3b5\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-pfvrr" Oct 29 22:34:02.367246 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:02.366589 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3cdc2c68-0a75-426d-ab02-ab7f6af525be-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-qjcwh\" (UID: \"3cdc2c68-0a75-426d-ab02-ab7f6af525be\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-qjcwh" Oct 29 22:34:02.367246 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:02.367043 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/689c465f-09b8-42cb-a382-5a59526bb3b5-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-pfvrr\" (UID: \"689c465f-09b8-42cb-a382-5a59526bb3b5\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-pfvrr" Oct 29 22:34:02.367246 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:02.367166 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a5b6a0e8-ad7a-4a47-a333-48e0d3b9eced-node-exporter-tls\") pod \"node-exporter-pcsj6\" (UID: \"a5b6a0e8-ad7a-4a47-a333-48e0d3b9eced\") " pod="openshift-monitoring/node-exporter-pcsj6" Oct 29 22:34:02.367732 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:02.367703 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/689c465f-09b8-42cb-a382-5a59526bb3b5-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-pfvrr\" (UID: \"689c465f-09b8-42cb-a382-5a59526bb3b5\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-pfvrr" Oct 29 22:34:02.367985 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:02.367965 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a5b6a0e8-ad7a-4a47-a333-48e0d3b9eced-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-pcsj6\" (UID: \"a5b6a0e8-ad7a-4a47-a333-48e0d3b9eced\") " pod="openshift-monitoring/node-exporter-pcsj6" Oct 29 22:34:02.368371 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:02.368324 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/689c465f-09b8-42cb-a382-5a59526bb3b5-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-pfvrr\" (UID: \"689c465f-09b8-42cb-a382-5a59526bb3b5\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-pfvrr" Oct 29 22:34:02.369865 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:02.369843 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/3cdc2c68-0a75-426d-ab02-ab7f6af525be-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-qjcwh\" (UID: \"3cdc2c68-0a75-426d-ab02-ab7f6af525be\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-qjcwh" Oct 29 22:34:02.386166 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:02.386139 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-whh2s\" (UniqueName: \"kubernetes.io/projected/a5b6a0e8-ad7a-4a47-a333-48e0d3b9eced-kube-api-access-whh2s\") pod \"node-exporter-pcsj6\" (UID: \"a5b6a0e8-ad7a-4a47-a333-48e0d3b9eced\") " pod="openshift-monitoring/node-exporter-pcsj6" Oct 29 22:34:02.386259 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:02.386161 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gpf2j\" (UniqueName: \"kubernetes.io/projected/3cdc2c68-0a75-426d-ab02-ab7f6af525be-kube-api-access-gpf2j\") pod \"openshift-state-metrics-7bd7489957-qjcwh\" (UID: \"3cdc2c68-0a75-426d-ab02-ab7f6af525be\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-qjcwh" Oct 29 22:34:02.386449 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:02.386421 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gd8hw\" (UniqueName: \"kubernetes.io/projected/689c465f-09b8-42cb-a382-5a59526bb3b5-kube-api-access-gd8hw\") pod \"kube-state-metrics-5dc8d7d89b-pfvrr\" (UID: \"689c465f-09b8-42cb-a382-5a59526bb3b5\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-pfvrr" Oct 29 22:34:02.422115 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:02.422084 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:34:02.533468 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:02.533427 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:34:02.556545 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:02.556521 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-qjcwh" Oct 29 22:34:02.643317 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:02.643124 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-pcsj6" Oct 29 22:34:02.650519 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:02.650346 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:34:02.670355 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:02.668932 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-pfvrr" Oct 29 22:34:02.687147 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:34:02.687109 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5b6a0e8_ad7a_4a47_a333_48e0d3b9eced.slice/crio-dfc01f894d07e3a603e6b61b5e487400583777659288cfc30a6c2a6604def1bc WatchSource:0}: Error finding container dfc01f894d07e3a603e6b61b5e487400583777659288cfc30a6c2a6604def1bc: Status 404 returned error can't find the container with id dfc01f894d07e3a603e6b61b5e487400583777659288cfc30a6c2a6604def1bc Oct 29 22:34:02.881148 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:02.881068 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:34:02.899032 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:02.898999 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-qjcwh"] Oct 29 22:34:02.902733 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:34:02.902711 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cdc2c68_0a75_426d_ab02_ab7f6af525be.slice/crio-1fe065d2112f87adf53b8ab8ba3f21f251f04c75a3a67360ab076bdc506fa9a6 WatchSource:0}: Error finding container 1fe065d2112f87adf53b8ab8ba3f21f251f04c75a3a67360ab076bdc506fa9a6: Status 404 returned error can't find the container with id 1fe065d2112f87adf53b8ab8ba3f21f251f04c75a3a67360ab076bdc506fa9a6 Oct 29 22:34:02.999223 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:02.999197 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-pfvrr"] Oct 29 22:34:03.003276 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:34:03.003246 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod689c465f_09b8_42cb_a382_5a59526bb3b5.slice/crio-fc02df75b76b2a23da4ff9d3b176d089943b7e9815042c840b25034ed313d9f1 WatchSource:0}: Error finding container fc02df75b76b2a23da4ff9d3b176d089943b7e9815042c840b25034ed313d9f1: Status 404 returned error can't find the container with id fc02df75b76b2a23da4ff9d3b176d089943b7e9815042c840b25034ed313d9f1 Oct 29 22:34:03.048987 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:03.048962 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-78db97464f-6ljzh" Oct 29 22:34:03.345264 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:03.345223 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:34:03.472804 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:03.472700 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:34:03.561163 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:03.561125 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-pcsj6" event={"ID":"a5b6a0e8-ad7a-4a47-a333-48e0d3b9eced","Type":"ContainerStarted","Data":"dfc01f894d07e3a603e6b61b5e487400583777659288cfc30a6c2a6604def1bc"} Oct 29 22:34:03.562922 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:03.562886 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-pfvrr" event={"ID":"689c465f-09b8-42cb-a382-5a59526bb3b5","Type":"ContainerStarted","Data":"fc02df75b76b2a23da4ff9d3b176d089943b7e9815042c840b25034ed313d9f1"} Oct 29 22:34:03.566697 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:03.566671 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-qjcwh" event={"ID":"3cdc2c68-0a75-426d-ab02-ab7f6af525be","Type":"ContainerStarted","Data":"be4709c7381567ba3f9c180fc2e1004f229e3463cca0d7e5102fe10fa909afef"} Oct 29 22:34:03.566805 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:03.566728 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-qjcwh" event={"ID":"3cdc2c68-0a75-426d-ab02-ab7f6af525be","Type":"ContainerStarted","Data":"5ef32f1cb52eb0d9d3a37b85a5c40826639902434f1c4136ceb81be1ac82843f"} Oct 29 22:34:03.566805 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:03.566760 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-qjcwh" event={"ID":"3cdc2c68-0a75-426d-ab02-ab7f6af525be","Type":"ContainerStarted","Data":"1fe065d2112f87adf53b8ab8ba3f21f251f04c75a3a67360ab076bdc506fa9a6"} Oct 29 22:34:03.609140 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:03.609109 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 29 22:34:03.610400 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:03.610298 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:34:03.639129 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:03.639100 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:34:03.641434 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:03.641402 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 29 22:34:03.653744 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:03.653098 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Oct 29 22:34:03.653744 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:03.653150 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Oct 29 22:34:03.653744 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:03.653258 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Oct 29 22:34:03.653744 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:03.653269 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-stxjf\"" Oct 29 22:34:03.653744 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:03.653321 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Oct 29 22:34:03.653744 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:03.653098 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Oct 29 22:34:03.653744 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:03.653321 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Oct 29 22:34:03.667146 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:03.667124 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Oct 29 22:34:03.675672 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:03.675579 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/86214365-1f4f-41a0-b8be-e2e11144241e-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"86214365-1f4f-41a0-b8be-e2e11144241e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:34:03.675672 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:03.675625 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/86214365-1f4f-41a0-b8be-e2e11144241e-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"86214365-1f4f-41a0-b8be-e2e11144241e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:34:03.675672 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:03.675652 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/86214365-1f4f-41a0-b8be-e2e11144241e-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"86214365-1f4f-41a0-b8be-e2e11144241e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:34:03.675857 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:03.675812 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/86214365-1f4f-41a0-b8be-e2e11144241e-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"86214365-1f4f-41a0-b8be-e2e11144241e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:34:03.675857 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:03.675848 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/86214365-1f4f-41a0-b8be-e2e11144241e-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"86214365-1f4f-41a0-b8be-e2e11144241e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:34:03.675956 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:03.675904 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/86214365-1f4f-41a0-b8be-e2e11144241e-config-volume\") pod \"alertmanager-main-0\" (UID: \"86214365-1f4f-41a0-b8be-e2e11144241e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:34:03.675956 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:03.675930 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/86214365-1f4f-41a0-b8be-e2e11144241e-tls-assets\") pod \"alertmanager-main-0\" (UID: \"86214365-1f4f-41a0-b8be-e2e11144241e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:34:03.676045 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:03.675960 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/86214365-1f4f-41a0-b8be-e2e11144241e-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"86214365-1f4f-41a0-b8be-e2e11144241e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:34:03.676045 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:03.675990 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5md5\" (UniqueName: \"kubernetes.io/projected/86214365-1f4f-41a0-b8be-e2e11144241e-kube-api-access-z5md5\") pod \"alertmanager-main-0\" (UID: \"86214365-1f4f-41a0-b8be-e2e11144241e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:34:03.676045 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:03.676022 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/86214365-1f4f-41a0-b8be-e2e11144241e-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"86214365-1f4f-41a0-b8be-e2e11144241e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:34:03.676150 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:03.676073 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/86214365-1f4f-41a0-b8be-e2e11144241e-web-config\") pod \"alertmanager-main-0\" (UID: \"86214365-1f4f-41a0-b8be-e2e11144241e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:34:03.676150 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:03.676123 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/86214365-1f4f-41a0-b8be-e2e11144241e-config-out\") pod \"alertmanager-main-0\" (UID: \"86214365-1f4f-41a0-b8be-e2e11144241e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:34:03.676235 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:03.676146 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/86214365-1f4f-41a0-b8be-e2e11144241e-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"86214365-1f4f-41a0-b8be-e2e11144241e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:34:03.752484 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:03.752419 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Oct 29 22:34:03.752484 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:03.752444 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Oct 29 22:34:03.778439 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:03.777092 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/86214365-1f4f-41a0-b8be-e2e11144241e-web-config\") pod \"alertmanager-main-0\" (UID: \"86214365-1f4f-41a0-b8be-e2e11144241e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:34:03.778439 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:03.777156 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/86214365-1f4f-41a0-b8be-e2e11144241e-config-out\") pod \"alertmanager-main-0\" (UID: \"86214365-1f4f-41a0-b8be-e2e11144241e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:34:03.778439 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:03.777181 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/86214365-1f4f-41a0-b8be-e2e11144241e-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"86214365-1f4f-41a0-b8be-e2e11144241e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:34:03.778439 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:03.777226 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/86214365-1f4f-41a0-b8be-e2e11144241e-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"86214365-1f4f-41a0-b8be-e2e11144241e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:34:03.778439 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:03.777253 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/86214365-1f4f-41a0-b8be-e2e11144241e-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"86214365-1f4f-41a0-b8be-e2e11144241e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:34:03.778439 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:03.777278 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/86214365-1f4f-41a0-b8be-e2e11144241e-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"86214365-1f4f-41a0-b8be-e2e11144241e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:34:03.778439 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:03.777353 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/86214365-1f4f-41a0-b8be-e2e11144241e-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"86214365-1f4f-41a0-b8be-e2e11144241e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:34:03.778439 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:03.777381 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/86214365-1f4f-41a0-b8be-e2e11144241e-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"86214365-1f4f-41a0-b8be-e2e11144241e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:34:03.778439 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:03.777434 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/86214365-1f4f-41a0-b8be-e2e11144241e-config-volume\") pod \"alertmanager-main-0\" (UID: \"86214365-1f4f-41a0-b8be-e2e11144241e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:34:03.778439 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:03.777458 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/86214365-1f4f-41a0-b8be-e2e11144241e-tls-assets\") pod \"alertmanager-main-0\" (UID: \"86214365-1f4f-41a0-b8be-e2e11144241e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:34:03.778439 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:03.777486 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/86214365-1f4f-41a0-b8be-e2e11144241e-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"86214365-1f4f-41a0-b8be-e2e11144241e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:34:03.778439 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:03.777513 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5md5\" (UniqueName: \"kubernetes.io/projected/86214365-1f4f-41a0-b8be-e2e11144241e-kube-api-access-z5md5\") pod \"alertmanager-main-0\" (UID: \"86214365-1f4f-41a0-b8be-e2e11144241e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:34:03.778439 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:03.777540 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/86214365-1f4f-41a0-b8be-e2e11144241e-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"86214365-1f4f-41a0-b8be-e2e11144241e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:34:03.779583 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:03.779557 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/86214365-1f4f-41a0-b8be-e2e11144241e-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"86214365-1f4f-41a0-b8be-e2e11144241e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:34:03.781101 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:03.781080 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/86214365-1f4f-41a0-b8be-e2e11144241e-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"86214365-1f4f-41a0-b8be-e2e11144241e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:34:03.782630 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:03.782602 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/86214365-1f4f-41a0-b8be-e2e11144241e-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"86214365-1f4f-41a0-b8be-e2e11144241e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:34:03.782732 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:03.782650 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/86214365-1f4f-41a0-b8be-e2e11144241e-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"86214365-1f4f-41a0-b8be-e2e11144241e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:34:03.782848 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:03.782825 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/86214365-1f4f-41a0-b8be-e2e11144241e-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"86214365-1f4f-41a0-b8be-e2e11144241e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:34:03.790181 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:03.790162 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/86214365-1f4f-41a0-b8be-e2e11144241e-config-out\") pod \"alertmanager-main-0\" (UID: \"86214365-1f4f-41a0-b8be-e2e11144241e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:34:03.791927 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:03.791903 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/86214365-1f4f-41a0-b8be-e2e11144241e-web-config\") pod \"alertmanager-main-0\" (UID: \"86214365-1f4f-41a0-b8be-e2e11144241e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:34:03.798265 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:03.798237 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/86214365-1f4f-41a0-b8be-e2e11144241e-config-volume\") pod \"alertmanager-main-0\" (UID: \"86214365-1f4f-41a0-b8be-e2e11144241e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:34:03.798265 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:03.798257 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/86214365-1f4f-41a0-b8be-e2e11144241e-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"86214365-1f4f-41a0-b8be-e2e11144241e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:34:03.798640 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:03.798621 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/86214365-1f4f-41a0-b8be-e2e11144241e-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"86214365-1f4f-41a0-b8be-e2e11144241e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:34:03.798709 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:03.798637 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/86214365-1f4f-41a0-b8be-e2e11144241e-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"86214365-1f4f-41a0-b8be-e2e11144241e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:34:03.799727 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:03.799599 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/86214365-1f4f-41a0-b8be-e2e11144241e-tls-assets\") pod \"alertmanager-main-0\" (UID: \"86214365-1f4f-41a0-b8be-e2e11144241e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:34:03.802381 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:03.802326 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5md5\" (UniqueName: \"kubernetes.io/projected/86214365-1f4f-41a0-b8be-e2e11144241e-kube-api-access-z5md5\") pod \"alertmanager-main-0\" (UID: \"86214365-1f4f-41a0-b8be-e2e11144241e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:34:03.807897 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:03.807877 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:34:04.155845 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:04.155671 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 29 22:34:04.159230 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:34:04.159120 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86214365_1f4f_41a0_b8be_e2e11144241e.slice/crio-d46b663b0acff115f38d66009fcba16a014e7b43c2eda5a591c11551baa8b2b8 WatchSource:0}: Error finding container d46b663b0acff115f38d66009fcba16a014e7b43c2eda5a591c11551baa8b2b8: Status 404 returned error can't find the container with id d46b663b0acff115f38d66009fcba16a014e7b43c2eda5a591c11551baa8b2b8 Oct 29 22:34:04.205434 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:04.204555 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-84866f86bd-2v6d7" Oct 29 22:34:04.432545 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:04.432508 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-74bbd64f85-jjdcv"] Oct 29 22:34:04.463609 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:04.463531 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-74bbd64f85-jjdcv"] Oct 29 22:34:04.463732 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:04.463653 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-74bbd64f85-jjdcv" Oct 29 22:34:04.477075 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:04.477056 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-55mrlepqn09fp\"" Oct 29 22:34:04.477174 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:04.477120 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Oct 29 22:34:04.477213 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:04.477189 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Oct 29 22:34:04.477250 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:04.477233 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Oct 29 22:34:04.477282 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:04.477272 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Oct 29 22:34:04.477369 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:04.477356 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Oct 29 22:34:04.477460 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:04.477447 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-dgfwx\"" Oct 29 22:34:04.485141 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:04.485111 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cfc225f9-52b5-4e36-916c-2467e3fe6e0b-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-74bbd64f85-jjdcv\" (UID: \"cfc225f9-52b5-4e36-916c-2467e3fe6e0b\") " pod="openshift-monitoring/thanos-querier-74bbd64f85-jjdcv" Oct 29 22:34:04.485249 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:04.485156 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zpp29\" (UniqueName: \"kubernetes.io/projected/cfc225f9-52b5-4e36-916c-2467e3fe6e0b-kube-api-access-zpp29\") pod \"thanos-querier-74bbd64f85-jjdcv\" (UID: \"cfc225f9-52b5-4e36-916c-2467e3fe6e0b\") " pod="openshift-monitoring/thanos-querier-74bbd64f85-jjdcv" Oct 29 22:34:04.485249 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:04.485217 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/cfc225f9-52b5-4e36-916c-2467e3fe6e0b-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-74bbd64f85-jjdcv\" (UID: \"cfc225f9-52b5-4e36-916c-2467e3fe6e0b\") " pod="openshift-monitoring/thanos-querier-74bbd64f85-jjdcv" Oct 29 22:34:04.485413 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:04.485272 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/cfc225f9-52b5-4e36-916c-2467e3fe6e0b-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-74bbd64f85-jjdcv\" (UID: \"cfc225f9-52b5-4e36-916c-2467e3fe6e0b\") " pod="openshift-monitoring/thanos-querier-74bbd64f85-jjdcv" Oct 29 22:34:04.485413 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:04.485383 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cfc225f9-52b5-4e36-916c-2467e3fe6e0b-tmp\") pod \"thanos-querier-74bbd64f85-jjdcv\" (UID: \"cfc225f9-52b5-4e36-916c-2467e3fe6e0b\") " pod="openshift-monitoring/thanos-querier-74bbd64f85-jjdcv" Oct 29 22:34:04.485512 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:04.485451 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/cfc225f9-52b5-4e36-916c-2467e3fe6e0b-secret-thanos-querier-tls\") pod \"thanos-querier-74bbd64f85-jjdcv\" (UID: \"cfc225f9-52b5-4e36-916c-2467e3fe6e0b\") " pod="openshift-monitoring/thanos-querier-74bbd64f85-jjdcv" Oct 29 22:34:04.485512 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:04.485486 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cfc225f9-52b5-4e36-916c-2467e3fe6e0b-metrics-client-ca\") pod \"thanos-querier-74bbd64f85-jjdcv\" (UID: \"cfc225f9-52b5-4e36-916c-2467e3fe6e0b\") " pod="openshift-monitoring/thanos-querier-74bbd64f85-jjdcv" Oct 29 22:34:04.485629 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:04.485591 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/cfc225f9-52b5-4e36-916c-2467e3fe6e0b-secret-grpc-tls\") pod \"thanos-querier-74bbd64f85-jjdcv\" (UID: \"cfc225f9-52b5-4e36-916c-2467e3fe6e0b\") " pod="openshift-monitoring/thanos-querier-74bbd64f85-jjdcv" Oct 29 22:34:04.485629 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:04.485616 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cfc225f9-52b5-4e36-916c-2467e3fe6e0b-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-74bbd64f85-jjdcv\" (UID: \"cfc225f9-52b5-4e36-916c-2467e3fe6e0b\") " pod="openshift-monitoring/thanos-querier-74bbd64f85-jjdcv" Oct 29 22:34:04.577199 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:04.577100 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"86214365-1f4f-41a0-b8be-e2e11144241e","Type":"ContainerStarted","Data":"d46b663b0acff115f38d66009fcba16a014e7b43c2eda5a591c11551baa8b2b8"} Oct 29 22:34:04.580230 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:04.580204 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-pcsj6" event={"ID":"a5b6a0e8-ad7a-4a47-a333-48e0d3b9eced","Type":"ContainerStarted","Data":"f6bf74ea3e38d96c13dc044dac6eeb94b02c2ef392e3e4205455aae6151b4ced"} Oct 29 22:34:04.587523 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:04.586981 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/cfc225f9-52b5-4e36-916c-2467e3fe6e0b-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-74bbd64f85-jjdcv\" (UID: \"cfc225f9-52b5-4e36-916c-2467e3fe6e0b\") " pod="openshift-monitoring/thanos-querier-74bbd64f85-jjdcv" Oct 29 22:34:04.587523 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:04.587059 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/cfc225f9-52b5-4e36-916c-2467e3fe6e0b-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-74bbd64f85-jjdcv\" (UID: \"cfc225f9-52b5-4e36-916c-2467e3fe6e0b\") " pod="openshift-monitoring/thanos-querier-74bbd64f85-jjdcv" Oct 29 22:34:04.587523 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:04.587118 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cfc225f9-52b5-4e36-916c-2467e3fe6e0b-tmp\") pod \"thanos-querier-74bbd64f85-jjdcv\" (UID: \"cfc225f9-52b5-4e36-916c-2467e3fe6e0b\") " pod="openshift-monitoring/thanos-querier-74bbd64f85-jjdcv" Oct 29 22:34:04.587523 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:04.587187 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/cfc225f9-52b5-4e36-916c-2467e3fe6e0b-secret-thanos-querier-tls\") pod \"thanos-querier-74bbd64f85-jjdcv\" (UID: \"cfc225f9-52b5-4e36-916c-2467e3fe6e0b\") " pod="openshift-monitoring/thanos-querier-74bbd64f85-jjdcv" Oct 29 22:34:04.587523 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:04.587389 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cfc225f9-52b5-4e36-916c-2467e3fe6e0b-metrics-client-ca\") pod \"thanos-querier-74bbd64f85-jjdcv\" (UID: \"cfc225f9-52b5-4e36-916c-2467e3fe6e0b\") " pod="openshift-monitoring/thanos-querier-74bbd64f85-jjdcv" Oct 29 22:34:04.587523 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:04.587440 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/cfc225f9-52b5-4e36-916c-2467e3fe6e0b-secret-grpc-tls\") pod \"thanos-querier-74bbd64f85-jjdcv\" (UID: \"cfc225f9-52b5-4e36-916c-2467e3fe6e0b\") " pod="openshift-monitoring/thanos-querier-74bbd64f85-jjdcv" Oct 29 22:34:04.587523 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:04.587477 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cfc225f9-52b5-4e36-916c-2467e3fe6e0b-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-74bbd64f85-jjdcv\" (UID: \"cfc225f9-52b5-4e36-916c-2467e3fe6e0b\") " pod="openshift-monitoring/thanos-querier-74bbd64f85-jjdcv" Oct 29 22:34:04.587523 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:04.587495 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cfc225f9-52b5-4e36-916c-2467e3fe6e0b-tmp\") pod \"thanos-querier-74bbd64f85-jjdcv\" (UID: \"cfc225f9-52b5-4e36-916c-2467e3fe6e0b\") " pod="openshift-monitoring/thanos-querier-74bbd64f85-jjdcv" Oct 29 22:34:04.587523 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:04.587510 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cfc225f9-52b5-4e36-916c-2467e3fe6e0b-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-74bbd64f85-jjdcv\" (UID: \"cfc225f9-52b5-4e36-916c-2467e3fe6e0b\") " pod="openshift-monitoring/thanos-querier-74bbd64f85-jjdcv" Oct 29 22:34:04.588046 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:04.587542 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpp29\" (UniqueName: \"kubernetes.io/projected/cfc225f9-52b5-4e36-916c-2467e3fe6e0b-kube-api-access-zpp29\") pod \"thanos-querier-74bbd64f85-jjdcv\" (UID: \"cfc225f9-52b5-4e36-916c-2467e3fe6e0b\") " pod="openshift-monitoring/thanos-querier-74bbd64f85-jjdcv" Oct 29 22:34:04.588848 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:04.588820 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cfc225f9-52b5-4e36-916c-2467e3fe6e0b-metrics-client-ca\") pod \"thanos-querier-74bbd64f85-jjdcv\" (UID: \"cfc225f9-52b5-4e36-916c-2467e3fe6e0b\") " pod="openshift-monitoring/thanos-querier-74bbd64f85-jjdcv" Oct 29 22:34:04.590001 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:04.589977 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/cfc225f9-52b5-4e36-916c-2467e3fe6e0b-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-74bbd64f85-jjdcv\" (UID: \"cfc225f9-52b5-4e36-916c-2467e3fe6e0b\") " pod="openshift-monitoring/thanos-querier-74bbd64f85-jjdcv" Oct 29 22:34:04.590513 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:04.590459 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/cfc225f9-52b5-4e36-916c-2467e3fe6e0b-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-74bbd64f85-jjdcv\" (UID: \"cfc225f9-52b5-4e36-916c-2467e3fe6e0b\") " pod="openshift-monitoring/thanos-querier-74bbd64f85-jjdcv" Oct 29 22:34:04.590513 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:04.590464 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/cfc225f9-52b5-4e36-916c-2467e3fe6e0b-secret-thanos-querier-tls\") pod \"thanos-querier-74bbd64f85-jjdcv\" (UID: \"cfc225f9-52b5-4e36-916c-2467e3fe6e0b\") " pod="openshift-monitoring/thanos-querier-74bbd64f85-jjdcv" Oct 29 22:34:04.590968 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:04.590942 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cfc225f9-52b5-4e36-916c-2467e3fe6e0b-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-74bbd64f85-jjdcv\" (UID: \"cfc225f9-52b5-4e36-916c-2467e3fe6e0b\") " pod="openshift-monitoring/thanos-querier-74bbd64f85-jjdcv" Oct 29 22:34:04.591528 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:04.591509 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/cfc225f9-52b5-4e36-916c-2467e3fe6e0b-secret-grpc-tls\") pod \"thanos-querier-74bbd64f85-jjdcv\" (UID: \"cfc225f9-52b5-4e36-916c-2467e3fe6e0b\") " pod="openshift-monitoring/thanos-querier-74bbd64f85-jjdcv" Oct 29 22:34:04.591667 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:04.591645 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cfc225f9-52b5-4e36-916c-2467e3fe6e0b-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-74bbd64f85-jjdcv\" (UID: \"cfc225f9-52b5-4e36-916c-2467e3fe6e0b\") " pod="openshift-monitoring/thanos-querier-74bbd64f85-jjdcv" Oct 29 22:34:04.609108 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:04.609089 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zpp29\" (UniqueName: \"kubernetes.io/projected/cfc225f9-52b5-4e36-916c-2467e3fe6e0b-kube-api-access-zpp29\") pod \"thanos-querier-74bbd64f85-jjdcv\" (UID: \"cfc225f9-52b5-4e36-916c-2467e3fe6e0b\") " pod="openshift-monitoring/thanos-querier-74bbd64f85-jjdcv" Oct 29 22:34:04.871762 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:04.871733 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-74bbd64f85-jjdcv" Oct 29 22:34:04.883749 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:04.883513 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:34:04.993850 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:04.993816 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:34:05.216157 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:05.216132 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-74bbd64f85-jjdcv"] Oct 29 22:34:05.221924 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:34:05.221892 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfc225f9_52b5_4e36_916c_2467e3fe6e0b.slice/crio-14239e1d5e7551699604dd376090b713e602c5fda3f3859e62141f0874ab633a WatchSource:0}: Error finding container 14239e1d5e7551699604dd376090b713e602c5fda3f3859e62141f0874ab633a: Status 404 returned error can't find the container with id 14239e1d5e7551699604dd376090b713e602c5fda3f3859e62141f0874ab633a Oct 29 22:34:05.422463 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:05.422433 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:34:05.549277 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:05.549195 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:34:05.590119 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:05.590086 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-74bbd64f85-jjdcv" event={"ID":"cfc225f9-52b5-4e36-916c-2467e3fe6e0b","Type":"ContainerStarted","Data":"14239e1d5e7551699604dd376090b713e602c5fda3f3859e62141f0874ab633a"} Oct 29 22:34:05.593673 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:05.593644 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-qjcwh" event={"ID":"3cdc2c68-0a75-426d-ab02-ab7f6af525be","Type":"ContainerStarted","Data":"d95bc9d63a68f93ba6f304dd4c897e9de0d8ccbb88f012d4b441f2eb89817080"} Oct 29 22:34:05.595472 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:05.595448 5057 generic.go:356] "Generic (PLEG): container finished" podID="a5b6a0e8-ad7a-4a47-a333-48e0d3b9eced" containerID="f6bf74ea3e38d96c13dc044dac6eeb94b02c2ef392e3e4205455aae6151b4ced" exitCode=0 Oct 29 22:34:05.595599 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:05.595532 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-pcsj6" event={"ID":"a5b6a0e8-ad7a-4a47-a333-48e0d3b9eced","Type":"ContainerDied","Data":"f6bf74ea3e38d96c13dc044dac6eeb94b02c2ef392e3e4205455aae6151b4ced"} Oct 29 22:34:05.598039 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:05.597947 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-pfvrr" event={"ID":"689c465f-09b8-42cb-a382-5a59526bb3b5","Type":"ContainerStarted","Data":"8fc4c9230db7e8ab064165b70227d3cb2a2a45233769be20b8fa5ee503884d82"} Oct 29 22:34:05.598039 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:05.597993 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-pfvrr" event={"ID":"689c465f-09b8-42cb-a382-5a59526bb3b5","Type":"ContainerStarted","Data":"1a5d8d9dd8c94bd5c7d838e0738fe4e960efb8f2da47b72821b9a23e768a621f"} Oct 29 22:34:05.643066 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:05.641791 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-qjcwh" podStartSLOduration=1.9525280120000001 podStartE2EDuration="3.641771177s" totalImagesPullingTime="1.689243165s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:34:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:34:05.633302628 +0000 UTC m=+26.416516082" watchObservedRunningTime="2026-10-29 22:34:05.641771177 +0000 UTC m=+26.424984604" Oct 29 22:34:05.664686 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:05.664632 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:34:06.172595 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:06.172524 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:34:06.286071 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:06.285939 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:34:06.607611 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:06.607534 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-pcsj6" event={"ID":"a5b6a0e8-ad7a-4a47-a333-48e0d3b9eced","Type":"ContainerStarted","Data":"c4b2e90209ac0ec23f3697a11a7d899eeb21f530d0a57b10f329e25584c0b83e"} Oct 29 22:34:06.607611 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:06.607587 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-pcsj6" event={"ID":"a5b6a0e8-ad7a-4a47-a333-48e0d3b9eced","Type":"ContainerStarted","Data":"00695f9e5f00994f88eeb73db755422aa5cb1353b72ea0a622efd18cd20807d6"} Oct 29 22:34:06.610508 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:06.610475 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-pfvrr" event={"ID":"689c465f-09b8-42cb-a382-5a59526bb3b5","Type":"ContainerStarted","Data":"644c2180b297fd3d8f593030a2314b7f8b38cd686d603fc74681d10831c40e06"} Oct 29 22:34:06.613018 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:06.612990 5057 generic.go:356] "Generic (PLEG): container finished" podID="86214365-1f4f-41a0-b8be-e2e11144241e" containerID="d7d287f36ab310608630b0f0b92d678cf2202e4b854dbd78e653bf0ad38b195a" exitCode=0 Oct 29 22:34:06.614048 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:06.614029 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"86214365-1f4f-41a0-b8be-e2e11144241e","Type":"ContainerDied","Data":"d7d287f36ab310608630b0f0b92d678cf2202e4b854dbd78e653bf0ad38b195a"} Oct 29 22:34:06.630765 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:06.630739 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-6f9dbf8897-m6njl"] Oct 29 22:34:06.649553 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:06.649487 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-pcsj6" podStartSLOduration=2.884274214 podStartE2EDuration="4.649469288s" totalImagesPullingTime="1.672786202s" totalInitContainerRuntime="92.408872ms" isStatefulPod=true podCreationTimestamp="2026-10-29 22:34:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:34:06.648275572 +0000 UTC m=+27.431488998" watchObservedRunningTime="2026-10-29 22:34:06.649469288 +0000 UTC m=+27.432682716" Oct 29 22:34:06.656670 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:06.656637 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-6f9dbf8897-m6njl" Oct 29 22:34:06.657457 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:06.657431 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-6f9dbf8897-m6njl"] Oct 29 22:34:06.670241 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:06.669466 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Oct 29 22:34:06.670241 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:06.669808 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Oct 29 22:34:06.670241 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:06.669871 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-np7p7\"" Oct 29 22:34:06.670241 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:06.669945 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-9ht8bi1fl73bt\"" Oct 29 22:34:06.670241 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:06.670049 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Oct 29 22:34:06.670241 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:06.670118 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Oct 29 22:34:06.689019 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:06.688977 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-pfvrr" podStartSLOduration=2.485711643 podStartE2EDuration="4.688963502s" totalImagesPullingTime="2.203251859s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:34:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:34:06.687687885 +0000 UTC m=+27.470901310" watchObservedRunningTime="2026-10-29 22:34:06.688963502 +0000 UTC m=+27.472176928" Oct 29 22:34:06.711756 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:06.711729 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/e02743b7-7cc7-44d9-a69b-26cd1a0c7f55-client-ca-bundle\") pod \"metrics-server-6f9dbf8897-m6njl\" (UID: \"e02743b7-7cc7-44d9-a69b-26cd1a0c7f55\") " pod="openshift-monitoring/metrics-server-6f9dbf8897-m6njl" Oct 29 22:34:06.711893 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:06.711778 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sj9jr\" (UniqueName: \"kubernetes.io/projected/e02743b7-7cc7-44d9-a69b-26cd1a0c7f55-kube-api-access-sj9jr\") pod \"metrics-server-6f9dbf8897-m6njl\" (UID: \"e02743b7-7cc7-44d9-a69b-26cd1a0c7f55\") " pod="openshift-monitoring/metrics-server-6f9dbf8897-m6njl" Oct 29 22:34:06.712257 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:06.712236 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/e02743b7-7cc7-44d9-a69b-26cd1a0c7f55-secret-metrics-server-tls\") pod \"metrics-server-6f9dbf8897-m6njl\" (UID: \"e02743b7-7cc7-44d9-a69b-26cd1a0c7f55\") " pod="openshift-monitoring/metrics-server-6f9dbf8897-m6njl" Oct 29 22:34:06.712407 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:06.712387 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/e02743b7-7cc7-44d9-a69b-26cd1a0c7f55-audit-log\") pod \"metrics-server-6f9dbf8897-m6njl\" (UID: \"e02743b7-7cc7-44d9-a69b-26cd1a0c7f55\") " pod="openshift-monitoring/metrics-server-6f9dbf8897-m6njl" Oct 29 22:34:06.712479 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:06.712422 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/e02743b7-7cc7-44d9-a69b-26cd1a0c7f55-metrics-server-audit-profiles\") pod \"metrics-server-6f9dbf8897-m6njl\" (UID: \"e02743b7-7cc7-44d9-a69b-26cd1a0c7f55\") " pod="openshift-monitoring/metrics-server-6f9dbf8897-m6njl" Oct 29 22:34:06.712689 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:06.712666 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e02743b7-7cc7-44d9-a69b-26cd1a0c7f55-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6f9dbf8897-m6njl\" (UID: \"e02743b7-7cc7-44d9-a69b-26cd1a0c7f55\") " pod="openshift-monitoring/metrics-server-6f9dbf8897-m6njl" Oct 29 22:34:06.712817 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:06.712803 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/e02743b7-7cc7-44d9-a69b-26cd1a0c7f55-secret-metrics-server-client-certs\") pod \"metrics-server-6f9dbf8897-m6njl\" (UID: \"e02743b7-7cc7-44d9-a69b-26cd1a0c7f55\") " pod="openshift-monitoring/metrics-server-6f9dbf8897-m6njl" Oct 29 22:34:06.814195 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:06.814163 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/e02743b7-7cc7-44d9-a69b-26cd1a0c7f55-secret-metrics-server-client-certs\") pod \"metrics-server-6f9dbf8897-m6njl\" (UID: \"e02743b7-7cc7-44d9-a69b-26cd1a0c7f55\") " pod="openshift-monitoring/metrics-server-6f9dbf8897-m6njl" Oct 29 22:34:06.814408 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:06.814229 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/e02743b7-7cc7-44d9-a69b-26cd1a0c7f55-client-ca-bundle\") pod \"metrics-server-6f9dbf8897-m6njl\" (UID: \"e02743b7-7cc7-44d9-a69b-26cd1a0c7f55\") " pod="openshift-monitoring/metrics-server-6f9dbf8897-m6njl" Oct 29 22:34:06.814408 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:06.814253 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sj9jr\" (UniqueName: \"kubernetes.io/projected/e02743b7-7cc7-44d9-a69b-26cd1a0c7f55-kube-api-access-sj9jr\") pod \"metrics-server-6f9dbf8897-m6njl\" (UID: \"e02743b7-7cc7-44d9-a69b-26cd1a0c7f55\") " pod="openshift-monitoring/metrics-server-6f9dbf8897-m6njl" Oct 29 22:34:06.814408 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:06.814305 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/e02743b7-7cc7-44d9-a69b-26cd1a0c7f55-secret-metrics-server-tls\") pod \"metrics-server-6f9dbf8897-m6njl\" (UID: \"e02743b7-7cc7-44d9-a69b-26cd1a0c7f55\") " pod="openshift-monitoring/metrics-server-6f9dbf8897-m6njl" Oct 29 22:34:06.814695 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:06.814668 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/e02743b7-7cc7-44d9-a69b-26cd1a0c7f55-audit-log\") pod \"metrics-server-6f9dbf8897-m6njl\" (UID: \"e02743b7-7cc7-44d9-a69b-26cd1a0c7f55\") " pod="openshift-monitoring/metrics-server-6f9dbf8897-m6njl" Oct 29 22:34:06.814781 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:06.814710 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/e02743b7-7cc7-44d9-a69b-26cd1a0c7f55-metrics-server-audit-profiles\") pod \"metrics-server-6f9dbf8897-m6njl\" (UID: \"e02743b7-7cc7-44d9-a69b-26cd1a0c7f55\") " pod="openshift-monitoring/metrics-server-6f9dbf8897-m6njl" Oct 29 22:34:06.814781 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:06.814759 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e02743b7-7cc7-44d9-a69b-26cd1a0c7f55-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6f9dbf8897-m6njl\" (UID: \"e02743b7-7cc7-44d9-a69b-26cd1a0c7f55\") " pod="openshift-monitoring/metrics-server-6f9dbf8897-m6njl" Oct 29 22:34:06.815107 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:06.815080 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/e02743b7-7cc7-44d9-a69b-26cd1a0c7f55-audit-log\") pod \"metrics-server-6f9dbf8897-m6njl\" (UID: \"e02743b7-7cc7-44d9-a69b-26cd1a0c7f55\") " pod="openshift-monitoring/metrics-server-6f9dbf8897-m6njl" Oct 29 22:34:06.815569 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:06.815541 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e02743b7-7cc7-44d9-a69b-26cd1a0c7f55-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6f9dbf8897-m6njl\" (UID: \"e02743b7-7cc7-44d9-a69b-26cd1a0c7f55\") " pod="openshift-monitoring/metrics-server-6f9dbf8897-m6njl" Oct 29 22:34:06.815838 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:06.815812 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/e02743b7-7cc7-44d9-a69b-26cd1a0c7f55-metrics-server-audit-profiles\") pod \"metrics-server-6f9dbf8897-m6njl\" (UID: \"e02743b7-7cc7-44d9-a69b-26cd1a0c7f55\") " pod="openshift-monitoring/metrics-server-6f9dbf8897-m6njl" Oct 29 22:34:06.817403 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:06.817382 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/e02743b7-7cc7-44d9-a69b-26cd1a0c7f55-secret-metrics-server-client-certs\") pod \"metrics-server-6f9dbf8897-m6njl\" (UID: \"e02743b7-7cc7-44d9-a69b-26cd1a0c7f55\") " pod="openshift-monitoring/metrics-server-6f9dbf8897-m6njl" Oct 29 22:34:06.817488 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:06.817432 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/e02743b7-7cc7-44d9-a69b-26cd1a0c7f55-secret-metrics-server-tls\") pod \"metrics-server-6f9dbf8897-m6njl\" (UID: \"e02743b7-7cc7-44d9-a69b-26cd1a0c7f55\") " pod="openshift-monitoring/metrics-server-6f9dbf8897-m6njl" Oct 29 22:34:06.819033 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:06.819002 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/e02743b7-7cc7-44d9-a69b-26cd1a0c7f55-client-ca-bundle\") pod \"metrics-server-6f9dbf8897-m6njl\" (UID: \"e02743b7-7cc7-44d9-a69b-26cd1a0c7f55\") " pod="openshift-monitoring/metrics-server-6f9dbf8897-m6njl" Oct 29 22:34:06.836536 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:06.836515 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sj9jr\" (UniqueName: \"kubernetes.io/projected/e02743b7-7cc7-44d9-a69b-26cd1a0c7f55-kube-api-access-sj9jr\") pod \"metrics-server-6f9dbf8897-m6njl\" (UID: \"e02743b7-7cc7-44d9-a69b-26cd1a0c7f55\") " pod="openshift-monitoring/metrics-server-6f9dbf8897-m6njl" Oct 29 22:34:06.951627 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:06.950691 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-7jv5g" Oct 29 22:34:06.963495 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:06.963446 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:34:06.983596 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:06.983563 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-6f9dbf8897-m6njl" Oct 29 22:34:06.991845 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:06.991817 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-mjnxx"] Oct 29 22:34:07.011987 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:07.011962 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-mjnxx" Oct 29 22:34:07.036487 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:07.036458 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-mjnxx"] Oct 29 22:34:07.040408 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:07.040131 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Oct 29 22:34:07.040408 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:07.040259 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-bjpxh\"" Oct 29 22:34:07.086543 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:07.086495 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:34:07.124670 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:07.124424 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/957c3493-0edc-4c9a-a85b-9ed7b23cb7ba-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-mjnxx\" (UID: \"957c3493-0edc-4c9a-a85b-9ed7b23cb7ba\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-mjnxx" Oct 29 22:34:07.124670 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:07.124557 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvbr2\" (UniqueName: \"kubernetes.io/projected/957c3493-0edc-4c9a-a85b-9ed7b23cb7ba-kube-api-access-wvbr2\") pod \"monitoring-plugin-77f4f76594-mjnxx\" (UID: \"957c3493-0edc-4c9a-a85b-9ed7b23cb7ba\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-mjnxx" Oct 29 22:34:07.216210 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:07.216120 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:34:07.226153 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:07.226119 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/957c3493-0edc-4c9a-a85b-9ed7b23cb7ba-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-mjnxx\" (UID: \"957c3493-0edc-4c9a-a85b-9ed7b23cb7ba\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-mjnxx" Oct 29 22:34:07.226317 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:07.226244 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvbr2\" (UniqueName: \"kubernetes.io/projected/957c3493-0edc-4c9a-a85b-9ed7b23cb7ba-kube-api-access-wvbr2\") pod \"monitoring-plugin-77f4f76594-mjnxx\" (UID: \"957c3493-0edc-4c9a-a85b-9ed7b23cb7ba\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-mjnxx" Oct 29 22:34:07.229944 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:07.229916 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/957c3493-0edc-4c9a-a85b-9ed7b23cb7ba-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-mjnxx\" (UID: \"957c3493-0edc-4c9a-a85b-9ed7b23cb7ba\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-mjnxx" Oct 29 22:34:07.248574 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:07.248492 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvbr2\" (UniqueName: \"kubernetes.io/projected/957c3493-0edc-4c9a-a85b-9ed7b23cb7ba-kube-api-access-wvbr2\") pod \"monitoring-plugin-77f4f76594-mjnxx\" (UID: \"957c3493-0edc-4c9a-a85b-9ed7b23cb7ba\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-mjnxx" Oct 29 22:34:07.330270 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:07.330162 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-6f9dbf8897-m6njl"] Oct 29 22:34:07.334831 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:34:07.334799 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode02743b7_7cc7_44d9_a69b_26cd1a0c7f55.slice/crio-c2687415e54744a507eee18c2a322d590290b4fc96ea55a9c617b6a85b55ae96 WatchSource:0}: Error finding container c2687415e54744a507eee18c2a322d590290b4fc96ea55a9c617b6a85b55ae96: Status 404 returned error can't find the container with id c2687415e54744a507eee18c2a322d590290b4fc96ea55a9c617b6a85b55ae96 Oct 29 22:34:07.373164 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:07.372929 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-mjnxx" Oct 29 22:34:07.621118 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:07.621084 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-6f9dbf8897-m6njl" event={"ID":"e02743b7-7cc7-44d9-a69b-26cd1a0c7f55","Type":"ContainerStarted","Data":"c2687415e54744a507eee18c2a322d590290b4fc96ea55a9c617b6a85b55ae96"} Oct 29 22:34:07.711774 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:07.711742 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-mjnxx"] Oct 29 22:34:07.845502 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:34:07.845409 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod957c3493_0edc_4c9a_a85b_9ed7b23cb7ba.slice/crio-b78303f96e02a8a5944e961040940f404fb2bb3ec909d62f957105d444d7c051 WatchSource:0}: Error finding container b78303f96e02a8a5944e961040940f404fb2bb3ec909d62f957105d444d7c051: Status 404 returned error can't find the container with id b78303f96e02a8a5944e961040940f404fb2bb3ec909d62f957105d444d7c051 Oct 29 22:34:08.154690 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:08.154554 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-68ff86fd-nb5ft" Oct 29 22:34:08.154795 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:08.154690 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-68ff86fd-nb5ft" Oct 29 22:34:08.161507 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:08.161287 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-68ff86fd-nb5ft" Oct 29 22:34:08.365848 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:08.365820 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-54f9x" Oct 29 22:34:08.633748 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:08.633709 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-mjnxx" event={"ID":"957c3493-0edc-4c9a-a85b-9ed7b23cb7ba","Type":"ContainerStarted","Data":"b78303f96e02a8a5944e961040940f404fb2bb3ec909d62f957105d444d7c051"} Oct 29 22:34:08.641934 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:08.638437 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-74bbd64f85-jjdcv" event={"ID":"cfc225f9-52b5-4e36-916c-2467e3fe6e0b","Type":"ContainerStarted","Data":"950da03dff1d0b7e9f7cc37337de52d936f4c60f373b17946c979b6f128725c4"} Oct 29 22:34:08.641934 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:08.638502 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-74bbd64f85-jjdcv" event={"ID":"cfc225f9-52b5-4e36-916c-2467e3fe6e0b","Type":"ContainerStarted","Data":"80be43c747eed7157ab8edae6a3d5672bc059bb427ff85c10ed7de379fe3505d"} Oct 29 22:34:08.641934 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:08.638540 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-74bbd64f85-jjdcv" event={"ID":"cfc225f9-52b5-4e36-916c-2467e3fe6e0b","Type":"ContainerStarted","Data":"85fbbc834d4eb411acfd8ea2ffc98e83b0683775d8dd1d08f435dda52fa60aa6"} Oct 29 22:34:08.644006 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:08.643286 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-68ff86fd-nb5ft" Oct 29 22:34:09.157179 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:09.157132 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 29 22:34:09.173054 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:09.171665 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:34:09.185413 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:09.185387 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Oct 29 22:34:09.185525 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:09.185425 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Oct 29 22:34:09.185525 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:09.185489 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Oct 29 22:34:09.185652 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:09.185540 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Oct 29 22:34:09.185652 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:09.185588 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Oct 29 22:34:09.192590 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:09.192562 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Oct 29 22:34:09.194936 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:09.194909 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:34:09.198410 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:09.198381 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Oct 29 22:34:09.198608 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:09.198507 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Oct 29 22:34:09.198755 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:09.198514 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Oct 29 22:34:09.198929 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:09.198576 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Oct 29 22:34:09.200773 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:09.200750 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 29 22:34:09.238589 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:09.237966 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Oct 29 22:34:09.247949 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:09.247928 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Oct 29 22:34:09.250741 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:09.250719 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/af0e8682-1347-4724-b0a4-870a0130fd00-web-config\") pod \"prometheus-k8s-0\" (UID: \"af0e8682-1347-4724-b0a4-870a0130fd00\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:34:09.250976 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:09.250958 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/af0e8682-1347-4724-b0a4-870a0130fd00-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"af0e8682-1347-4724-b0a4-870a0130fd00\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:34:09.251118 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:09.251099 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/af0e8682-1347-4724-b0a4-870a0130fd00-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"af0e8682-1347-4724-b0a4-870a0130fd00\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:34:09.251298 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:09.251280 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/af0e8682-1347-4724-b0a4-870a0130fd00-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"af0e8682-1347-4724-b0a4-870a0130fd00\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:34:09.251501 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:09.251485 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/af0e8682-1347-4724-b0a4-870a0130fd00-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"af0e8682-1347-4724-b0a4-870a0130fd00\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:34:09.251677 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:09.251660 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/af0e8682-1347-4724-b0a4-870a0130fd00-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"af0e8682-1347-4724-b0a4-870a0130fd00\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:34:09.251861 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:09.251846 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/af0e8682-1347-4724-b0a4-870a0130fd00-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"af0e8682-1347-4724-b0a4-870a0130fd00\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:34:09.252043 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:09.252026 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/af0e8682-1347-4724-b0a4-870a0130fd00-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"af0e8682-1347-4724-b0a4-870a0130fd00\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:34:09.252215 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:09.252200 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/af0e8682-1347-4724-b0a4-870a0130fd00-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"af0e8682-1347-4724-b0a4-870a0130fd00\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:34:09.252563 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:09.252543 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/af0e8682-1347-4724-b0a4-870a0130fd00-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"af0e8682-1347-4724-b0a4-870a0130fd00\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:34:09.252748 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:09.252732 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/af0e8682-1347-4724-b0a4-870a0130fd00-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"af0e8682-1347-4724-b0a4-870a0130fd00\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:34:09.252916 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:09.252898 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5wj4\" (UniqueName: \"kubernetes.io/projected/af0e8682-1347-4724-b0a4-870a0130fd00-kube-api-access-g5wj4\") pod \"prometheus-k8s-0\" (UID: \"af0e8682-1347-4724-b0a4-870a0130fd00\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:34:09.253099 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:09.253082 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/af0e8682-1347-4724-b0a4-870a0130fd00-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"af0e8682-1347-4724-b0a4-870a0130fd00\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:34:09.253285 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:09.253267 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/af0e8682-1347-4724-b0a4-870a0130fd00-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"af0e8682-1347-4724-b0a4-870a0130fd00\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:34:09.253480 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:09.253462 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/af0e8682-1347-4724-b0a4-870a0130fd00-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"af0e8682-1347-4724-b0a4-870a0130fd00\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:34:09.253685 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:09.253669 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/af0e8682-1347-4724-b0a4-870a0130fd00-config-out\") pod \"prometheus-k8s-0\" (UID: \"af0e8682-1347-4724-b0a4-870a0130fd00\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:34:09.253855 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:09.253836 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/af0e8682-1347-4724-b0a4-870a0130fd00-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"af0e8682-1347-4724-b0a4-870a0130fd00\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:34:09.254027 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:09.254010 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/af0e8682-1347-4724-b0a4-870a0130fd00-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"af0e8682-1347-4724-b0a4-870a0130fd00\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:34:09.254189 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:09.254174 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/af0e8682-1347-4724-b0a4-870a0130fd00-config\") pod \"prometheus-k8s-0\" (UID: \"af0e8682-1347-4724-b0a4-870a0130fd00\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:34:09.254371 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:09.254353 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/af0e8682-1347-4724-b0a4-870a0130fd00-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"af0e8682-1347-4724-b0a4-870a0130fd00\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:34:09.285837 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:09.285483 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Oct 29 22:34:09.285837 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:09.285646 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Oct 29 22:34:09.285837 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:09.285713 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-dc59t2jksncvi\"" Oct 29 22:34:09.285837 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:09.285796 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-cjpf4\"" Oct 29 22:34:09.357984 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:09.356833 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/af0e8682-1347-4724-b0a4-870a0130fd00-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"af0e8682-1347-4724-b0a4-870a0130fd00\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:34:09.357984 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:09.356887 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/af0e8682-1347-4724-b0a4-870a0130fd00-config\") pod \"prometheus-k8s-0\" (UID: \"af0e8682-1347-4724-b0a4-870a0130fd00\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:34:09.357984 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:09.356922 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/af0e8682-1347-4724-b0a4-870a0130fd00-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"af0e8682-1347-4724-b0a4-870a0130fd00\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:34:09.357984 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:09.356951 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/af0e8682-1347-4724-b0a4-870a0130fd00-web-config\") pod \"prometheus-k8s-0\" (UID: \"af0e8682-1347-4724-b0a4-870a0130fd00\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:34:09.357984 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:09.356985 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/af0e8682-1347-4724-b0a4-870a0130fd00-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"af0e8682-1347-4724-b0a4-870a0130fd00\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:34:09.357984 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:09.357013 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/af0e8682-1347-4724-b0a4-870a0130fd00-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"af0e8682-1347-4724-b0a4-870a0130fd00\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:34:09.357984 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:09.357044 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/af0e8682-1347-4724-b0a4-870a0130fd00-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"af0e8682-1347-4724-b0a4-870a0130fd00\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:34:09.357984 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:09.357076 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/af0e8682-1347-4724-b0a4-870a0130fd00-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"af0e8682-1347-4724-b0a4-870a0130fd00\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:34:09.357984 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:09.357102 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/af0e8682-1347-4724-b0a4-870a0130fd00-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"af0e8682-1347-4724-b0a4-870a0130fd00\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:34:09.357984 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:09.357160 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/af0e8682-1347-4724-b0a4-870a0130fd00-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"af0e8682-1347-4724-b0a4-870a0130fd00\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:34:09.357984 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:09.357204 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/af0e8682-1347-4724-b0a4-870a0130fd00-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"af0e8682-1347-4724-b0a4-870a0130fd00\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:34:09.357984 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:09.357245 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/af0e8682-1347-4724-b0a4-870a0130fd00-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"af0e8682-1347-4724-b0a4-870a0130fd00\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:34:09.357984 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:09.357272 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/af0e8682-1347-4724-b0a4-870a0130fd00-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"af0e8682-1347-4724-b0a4-870a0130fd00\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:34:09.357984 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:09.357297 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/af0e8682-1347-4724-b0a4-870a0130fd00-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"af0e8682-1347-4724-b0a4-870a0130fd00\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:34:09.357984 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:09.357321 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5wj4\" (UniqueName: \"kubernetes.io/projected/af0e8682-1347-4724-b0a4-870a0130fd00-kube-api-access-g5wj4\") pod \"prometheus-k8s-0\" (UID: \"af0e8682-1347-4724-b0a4-870a0130fd00\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:34:09.357984 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:09.357382 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/af0e8682-1347-4724-b0a4-870a0130fd00-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"af0e8682-1347-4724-b0a4-870a0130fd00\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:34:09.357984 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:09.357423 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/af0e8682-1347-4724-b0a4-870a0130fd00-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"af0e8682-1347-4724-b0a4-870a0130fd00\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:34:09.357984 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:09.357447 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/af0e8682-1347-4724-b0a4-870a0130fd00-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"af0e8682-1347-4724-b0a4-870a0130fd00\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:34:09.357984 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:09.357486 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/af0e8682-1347-4724-b0a4-870a0130fd00-config-out\") pod \"prometheus-k8s-0\" (UID: \"af0e8682-1347-4724-b0a4-870a0130fd00\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:34:09.357984 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:09.357515 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/af0e8682-1347-4724-b0a4-870a0130fd00-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"af0e8682-1347-4724-b0a4-870a0130fd00\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:34:09.362727 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:09.359093 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/af0e8682-1347-4724-b0a4-870a0130fd00-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"af0e8682-1347-4724-b0a4-870a0130fd00\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:34:09.362727 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:09.359972 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/af0e8682-1347-4724-b0a4-870a0130fd00-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"af0e8682-1347-4724-b0a4-870a0130fd00\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:34:09.362727 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:09.360248 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/af0e8682-1347-4724-b0a4-870a0130fd00-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"af0e8682-1347-4724-b0a4-870a0130fd00\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:34:09.362727 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:09.360472 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/af0e8682-1347-4724-b0a4-870a0130fd00-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"af0e8682-1347-4724-b0a4-870a0130fd00\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:34:09.362727 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:09.360892 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/af0e8682-1347-4724-b0a4-870a0130fd00-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"af0e8682-1347-4724-b0a4-870a0130fd00\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:34:09.362727 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:09.361058 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/af0e8682-1347-4724-b0a4-870a0130fd00-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"af0e8682-1347-4724-b0a4-870a0130fd00\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:34:09.362727 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:09.362506 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/af0e8682-1347-4724-b0a4-870a0130fd00-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"af0e8682-1347-4724-b0a4-870a0130fd00\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:34:09.365794 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:09.364029 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/af0e8682-1347-4724-b0a4-870a0130fd00-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"af0e8682-1347-4724-b0a4-870a0130fd00\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:34:09.365794 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:09.364946 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/af0e8682-1347-4724-b0a4-870a0130fd00-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"af0e8682-1347-4724-b0a4-870a0130fd00\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:34:09.365794 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:09.365411 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/af0e8682-1347-4724-b0a4-870a0130fd00-config-out\") pod \"prometheus-k8s-0\" (UID: \"af0e8682-1347-4724-b0a4-870a0130fd00\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:34:09.365794 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:09.365439 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/af0e8682-1347-4724-b0a4-870a0130fd00-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"af0e8682-1347-4724-b0a4-870a0130fd00\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:34:09.365794 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:09.365774 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/af0e8682-1347-4724-b0a4-870a0130fd00-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"af0e8682-1347-4724-b0a4-870a0130fd00\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:34:09.366118 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:09.366059 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/af0e8682-1347-4724-b0a4-870a0130fd00-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"af0e8682-1347-4724-b0a4-870a0130fd00\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:34:09.368186 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:09.367984 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/af0e8682-1347-4724-b0a4-870a0130fd00-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"af0e8682-1347-4724-b0a4-870a0130fd00\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:34:09.372313 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:09.369166 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/af0e8682-1347-4724-b0a4-870a0130fd00-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"af0e8682-1347-4724-b0a4-870a0130fd00\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:34:09.372313 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:09.370800 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/af0e8682-1347-4724-b0a4-870a0130fd00-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"af0e8682-1347-4724-b0a4-870a0130fd00\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:34:09.372313 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:09.370897 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/af0e8682-1347-4724-b0a4-870a0130fd00-web-config\") pod \"prometheus-k8s-0\" (UID: \"af0e8682-1347-4724-b0a4-870a0130fd00\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:34:09.372313 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:09.371803 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/af0e8682-1347-4724-b0a4-870a0130fd00-config\") pod \"prometheus-k8s-0\" (UID: \"af0e8682-1347-4724-b0a4-870a0130fd00\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:34:09.372760 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:09.372740 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/af0e8682-1347-4724-b0a4-870a0130fd00-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"af0e8682-1347-4724-b0a4-870a0130fd00\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:34:09.382189 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:09.380487 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5wj4\" (UniqueName: \"kubernetes.io/projected/af0e8682-1347-4724-b0a4-870a0130fd00-kube-api-access-g5wj4\") pod \"prometheus-k8s-0\" (UID: \"af0e8682-1347-4724-b0a4-870a0130fd00\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:34:09.389908 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:09.389825 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:34:09.471578 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:09.471545 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-f8f65dd7c-j9qgf"] Oct 29 22:34:09.491281 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:09.491244 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-f8f65dd7c-j9qgf" Oct 29 22:34:09.497621 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:09.497598 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-f8f65dd7c-j9qgf"] Oct 29 22:34:09.509245 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:09.506224 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Oct 29 22:34:09.551741 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:09.551520 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:34:09.558727 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:09.558699 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9fdd36b9-cabf-4104-9d4e-bfe0e6aa083d-trusted-ca-bundle\") pod \"console-f8f65dd7c-j9qgf\" (UID: \"9fdd36b9-cabf-4104-9d4e-bfe0e6aa083d\") " pod="openshift-console/console-f8f65dd7c-j9qgf" Oct 29 22:34:09.558840 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:09.558760 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9fdd36b9-cabf-4104-9d4e-bfe0e6aa083d-console-config\") pod \"console-f8f65dd7c-j9qgf\" (UID: \"9fdd36b9-cabf-4104-9d4e-bfe0e6aa083d\") " pod="openshift-console/console-f8f65dd7c-j9qgf" Oct 29 22:34:09.558840 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:09.558813 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9fdd36b9-cabf-4104-9d4e-bfe0e6aa083d-oauth-serving-cert\") pod \"console-f8f65dd7c-j9qgf\" (UID: \"9fdd36b9-cabf-4104-9d4e-bfe0e6aa083d\") " pod="openshift-console/console-f8f65dd7c-j9qgf" Oct 29 22:34:09.558962 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:09.558855 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9fdd36b9-cabf-4104-9d4e-bfe0e6aa083d-service-ca\") pod \"console-f8f65dd7c-j9qgf\" (UID: \"9fdd36b9-cabf-4104-9d4e-bfe0e6aa083d\") " pod="openshift-console/console-f8f65dd7c-j9qgf" Oct 29 22:34:09.558962 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:09.558899 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52qmj\" (UniqueName: \"kubernetes.io/projected/9fdd36b9-cabf-4104-9d4e-bfe0e6aa083d-kube-api-access-52qmj\") pod \"console-f8f65dd7c-j9qgf\" (UID: \"9fdd36b9-cabf-4104-9d4e-bfe0e6aa083d\") " pod="openshift-console/console-f8f65dd7c-j9qgf" Oct 29 22:34:09.559062 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:09.558963 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9fdd36b9-cabf-4104-9d4e-bfe0e6aa083d-tmp\") pod \"console-f8f65dd7c-j9qgf\" (UID: \"9fdd36b9-cabf-4104-9d4e-bfe0e6aa083d\") " pod="openshift-console/console-f8f65dd7c-j9qgf" Oct 29 22:34:09.559062 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:09.558995 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9fdd36b9-cabf-4104-9d4e-bfe0e6aa083d-console-serving-cert\") pod \"console-f8f65dd7c-j9qgf\" (UID: \"9fdd36b9-cabf-4104-9d4e-bfe0e6aa083d\") " pod="openshift-console/console-f8f65dd7c-j9qgf" Oct 29 22:34:09.559062 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:09.559021 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9fdd36b9-cabf-4104-9d4e-bfe0e6aa083d-console-oauth-config\") pod \"console-f8f65dd7c-j9qgf\" (UID: \"9fdd36b9-cabf-4104-9d4e-bfe0e6aa083d\") " pod="openshift-console/console-f8f65dd7c-j9qgf" Oct 29 22:34:09.654602 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:09.654562 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"86214365-1f4f-41a0-b8be-e2e11144241e","Type":"ContainerStarted","Data":"513b36d25a3af6474b29848cb8dca81da4c37d4dbae3fcc414e7665eb7aa181d"} Oct 29 22:34:09.655219 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:09.655192 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"86214365-1f4f-41a0-b8be-e2e11144241e","Type":"ContainerStarted","Data":"750111f4ae99d3b9585092893f4af8b45fec6791220c9a472bc861dba69a4888"} Oct 29 22:34:09.655395 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:09.655378 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"86214365-1f4f-41a0-b8be-e2e11144241e","Type":"ContainerStarted","Data":"4662b0a105ae1bd58a9fbef8995c2b945b6a0965208374d64f6c55368f5c9e8f"} Oct 29 22:34:09.655563 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:09.655546 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"86214365-1f4f-41a0-b8be-e2e11144241e","Type":"ContainerStarted","Data":"533c17e22fcf31771228a2991b31d812919dfdb55880fbd7188a5dfcdfb63e89"} Oct 29 22:34:09.655748 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:09.655714 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"86214365-1f4f-41a0-b8be-e2e11144241e","Type":"ContainerStarted","Data":"b1bb7d0f6b64084459c8859945660d9620d9ee72b4bb1d6e71bd3fc7b4632a65"} Oct 29 22:34:09.659861 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:09.659835 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9fdd36b9-cabf-4104-9d4e-bfe0e6aa083d-trusted-ca-bundle\") pod \"console-f8f65dd7c-j9qgf\" (UID: \"9fdd36b9-cabf-4104-9d4e-bfe0e6aa083d\") " pod="openshift-console/console-f8f65dd7c-j9qgf" Oct 29 22:34:09.659987 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:09.659887 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9fdd36b9-cabf-4104-9d4e-bfe0e6aa083d-console-config\") pod \"console-f8f65dd7c-j9qgf\" (UID: \"9fdd36b9-cabf-4104-9d4e-bfe0e6aa083d\") " pod="openshift-console/console-f8f65dd7c-j9qgf" Oct 29 22:34:09.659987 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:09.659921 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9fdd36b9-cabf-4104-9d4e-bfe0e6aa083d-oauth-serving-cert\") pod \"console-f8f65dd7c-j9qgf\" (UID: \"9fdd36b9-cabf-4104-9d4e-bfe0e6aa083d\") " pod="openshift-console/console-f8f65dd7c-j9qgf" Oct 29 22:34:09.659987 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:09.659957 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9fdd36b9-cabf-4104-9d4e-bfe0e6aa083d-service-ca\") pod \"console-f8f65dd7c-j9qgf\" (UID: \"9fdd36b9-cabf-4104-9d4e-bfe0e6aa083d\") " pod="openshift-console/console-f8f65dd7c-j9qgf" Oct 29 22:34:09.659987 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:09.659983 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52qmj\" (UniqueName: \"kubernetes.io/projected/9fdd36b9-cabf-4104-9d4e-bfe0e6aa083d-kube-api-access-52qmj\") pod \"console-f8f65dd7c-j9qgf\" (UID: \"9fdd36b9-cabf-4104-9d4e-bfe0e6aa083d\") " pod="openshift-console/console-f8f65dd7c-j9qgf" Oct 29 22:34:09.660212 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:09.660028 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9fdd36b9-cabf-4104-9d4e-bfe0e6aa083d-tmp\") pod \"console-f8f65dd7c-j9qgf\" (UID: \"9fdd36b9-cabf-4104-9d4e-bfe0e6aa083d\") " pod="openshift-console/console-f8f65dd7c-j9qgf" Oct 29 22:34:09.660212 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:09.660050 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9fdd36b9-cabf-4104-9d4e-bfe0e6aa083d-console-serving-cert\") pod \"console-f8f65dd7c-j9qgf\" (UID: \"9fdd36b9-cabf-4104-9d4e-bfe0e6aa083d\") " pod="openshift-console/console-f8f65dd7c-j9qgf" Oct 29 22:34:09.660212 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:09.660064 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9fdd36b9-cabf-4104-9d4e-bfe0e6aa083d-console-oauth-config\") pod \"console-f8f65dd7c-j9qgf\" (UID: \"9fdd36b9-cabf-4104-9d4e-bfe0e6aa083d\") " pod="openshift-console/console-f8f65dd7c-j9qgf" Oct 29 22:34:09.661081 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:09.661057 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9fdd36b9-cabf-4104-9d4e-bfe0e6aa083d-tmp\") pod \"console-f8f65dd7c-j9qgf\" (UID: \"9fdd36b9-cabf-4104-9d4e-bfe0e6aa083d\") " pod="openshift-console/console-f8f65dd7c-j9qgf" Oct 29 22:34:09.662369 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:09.661717 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9fdd36b9-cabf-4104-9d4e-bfe0e6aa083d-service-ca\") pod \"console-f8f65dd7c-j9qgf\" (UID: \"9fdd36b9-cabf-4104-9d4e-bfe0e6aa083d\") " pod="openshift-console/console-f8f65dd7c-j9qgf" Oct 29 22:34:09.662369 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:09.661854 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9fdd36b9-cabf-4104-9d4e-bfe0e6aa083d-trusted-ca-bundle\") pod \"console-f8f65dd7c-j9qgf\" (UID: \"9fdd36b9-cabf-4104-9d4e-bfe0e6aa083d\") " pod="openshift-console/console-f8f65dd7c-j9qgf" Oct 29 22:34:09.662553 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:09.662284 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9fdd36b9-cabf-4104-9d4e-bfe0e6aa083d-oauth-serving-cert\") pod \"console-f8f65dd7c-j9qgf\" (UID: \"9fdd36b9-cabf-4104-9d4e-bfe0e6aa083d\") " pod="openshift-console/console-f8f65dd7c-j9qgf" Oct 29 22:34:09.662553 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:09.662291 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9fdd36b9-cabf-4104-9d4e-bfe0e6aa083d-console-config\") pod \"console-f8f65dd7c-j9qgf\" (UID: \"9fdd36b9-cabf-4104-9d4e-bfe0e6aa083d\") " pod="openshift-console/console-f8f65dd7c-j9qgf" Oct 29 22:34:09.663228 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:09.663207 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9fdd36b9-cabf-4104-9d4e-bfe0e6aa083d-console-oauth-config\") pod \"console-f8f65dd7c-j9qgf\" (UID: \"9fdd36b9-cabf-4104-9d4e-bfe0e6aa083d\") " pod="openshift-console/console-f8f65dd7c-j9qgf" Oct 29 22:34:09.665923 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:09.665898 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9fdd36b9-cabf-4104-9d4e-bfe0e6aa083d-console-serving-cert\") pod \"console-f8f65dd7c-j9qgf\" (UID: \"9fdd36b9-cabf-4104-9d4e-bfe0e6aa083d\") " pod="openshift-console/console-f8f65dd7c-j9qgf" Oct 29 22:34:09.682939 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:09.682885 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-52qmj\" (UniqueName: \"kubernetes.io/projected/9fdd36b9-cabf-4104-9d4e-bfe0e6aa083d-kube-api-access-52qmj\") pod \"console-f8f65dd7c-j9qgf\" (UID: \"9fdd36b9-cabf-4104-9d4e-bfe0e6aa083d\") " pod="openshift-console/console-f8f65dd7c-j9qgf" Oct 29 22:34:09.827194 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:09.826980 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-f8f65dd7c-j9qgf" Oct 29 22:34:10.132214 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:10.132168 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 29 22:34:10.168951 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:10.168923 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-f8f65dd7c-j9qgf"] Oct 29 22:34:10.246526 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:34:10.246495 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fdd36b9_cabf_4104_9d4e_bfe0e6aa083d.slice/crio-1e324881c5c7d684ea1a11595b7e9ee3dc697b71c2ed68a6715a83e2283c757f WatchSource:0}: Error finding container 1e324881c5c7d684ea1a11595b7e9ee3dc697b71c2ed68a6715a83e2283c757f: Status 404 returned error can't find the container with id 1e324881c5c7d684ea1a11595b7e9ee3dc697b71c2ed68a6715a83e2283c757f Oct 29 22:34:10.667318 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:10.667237 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-mjnxx" event={"ID":"957c3493-0edc-4c9a-a85b-9ed7b23cb7ba","Type":"ContainerStarted","Data":"1d5516d552226e5633877cb5544c5dbf1c9f3af02bb13c24832464f57015f8d9"} Oct 29 22:34:10.667917 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:10.667773 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-mjnxx" Oct 29 22:34:10.673247 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:10.673222 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-74bbd64f85-jjdcv" event={"ID":"cfc225f9-52b5-4e36-916c-2467e3fe6e0b","Type":"ContainerStarted","Data":"a942469c78eef3c9ac9474df5c115aa1ae956c8fa3ae80a2846f2086ecfa7d96"} Oct 29 22:34:10.673379 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:10.673274 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-74bbd64f85-jjdcv" event={"ID":"cfc225f9-52b5-4e36-916c-2467e3fe6e0b","Type":"ContainerStarted","Data":"f3ceafc904370e5d9568b0c3bf58d979f29c5e09e24e7fb774a57db7212bb5e6"} Oct 29 22:34:10.673379 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:10.673318 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-74bbd64f85-jjdcv" Oct 29 22:34:10.673492 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:10.673390 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-74bbd64f85-jjdcv" event={"ID":"cfc225f9-52b5-4e36-916c-2467e3fe6e0b","Type":"ContainerStarted","Data":"a5b433965d013f9270489d992e684b44eec2f529dcf09281880634d3aac2a978"} Oct 29 22:34:10.673715 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:10.673693 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-mjnxx" Oct 29 22:34:10.675347 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:10.675296 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-6f9dbf8897-m6njl" event={"ID":"e02743b7-7cc7-44d9-a69b-26cd1a0c7f55","Type":"ContainerStarted","Data":"bb2561827eb126a55310cfcd524a933c4c6076e3bb39c4fa25774affc14d28a6"} Oct 29 22:34:10.677297 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:10.677273 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-f8f65dd7c-j9qgf" event={"ID":"9fdd36b9-cabf-4104-9d4e-bfe0e6aa083d","Type":"ContainerStarted","Data":"9161f995b8554f4488030c35b87226fa8fbb72dec5b99ac5b2e8dbf6195c545e"} Oct 29 22:34:10.677442 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:10.677317 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-f8f65dd7c-j9qgf" event={"ID":"9fdd36b9-cabf-4104-9d4e-bfe0e6aa083d","Type":"ContainerStarted","Data":"1e324881c5c7d684ea1a11595b7e9ee3dc697b71c2ed68a6715a83e2283c757f"} Oct 29 22:34:10.679234 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:10.679199 5057 generic.go:356] "Generic (PLEG): container finished" podID="af0e8682-1347-4724-b0a4-870a0130fd00" containerID="64027ca342298dbcd74d94bc62ee02cdfdc65123eb490f95d4a53d35ec81f67b" exitCode=0 Oct 29 22:34:10.679363 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:10.679288 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"af0e8682-1347-4724-b0a4-870a0130fd00","Type":"ContainerDied","Data":"64027ca342298dbcd74d94bc62ee02cdfdc65123eb490f95d4a53d35ec81f67b"} Oct 29 22:34:10.679441 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:10.679372 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"af0e8682-1347-4724-b0a4-870a0130fd00","Type":"ContainerStarted","Data":"def28858d816763418378a5405cc991bac32ce982d06832a69a1f71274c323ff"} Oct 29 22:34:10.683905 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:10.683620 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"86214365-1f4f-41a0-b8be-e2e11144241e","Type":"ContainerStarted","Data":"f4bb10070eb88aedd6ba257addc8ffe8589b7b09b3e81a6a05690bc09559f042"} Oct 29 22:34:10.704838 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:10.704789 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-77f4f76594-mjnxx" podStartSLOduration=2.300686299 podStartE2EDuration="4.704776179s" totalImagesPullingTime="2.40408988s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:34:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:34:10.703196693 +0000 UTC m=+31.486410124" watchObservedRunningTime="2026-10-29 22:34:10.704776179 +0000 UTC m=+31.487989635" Oct 29 22:34:10.749545 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:10.748322 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-74bbd64f85-jjdcv" podStartSLOduration=2.572019322 podStartE2EDuration="6.74829782s" totalImagesPullingTime="4.176278498s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:34:04 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:34:10.74650007 +0000 UTC m=+31.529713497" watchObservedRunningTime="2026-10-29 22:34:10.74829782 +0000 UTC m=+31.531511249" Oct 29 22:34:10.787568 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:10.787517 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-6f9dbf8897-m6njl" podStartSLOduration=2.330308761 podStartE2EDuration="4.78750375s" totalImagesPullingTime="2.457194989s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:34:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:34:10.786092326 +0000 UTC m=+31.569305809" watchObservedRunningTime="2026-10-29 22:34:10.78750375 +0000 UTC m=+31.570717176" Oct 29 22:34:10.913612 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:10.913559 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-f8f65dd7c-j9qgf" podStartSLOduration=1.913541392 podStartE2EDuration="1.913541392s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:34:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:34:10.911359213 +0000 UTC m=+31.694572665" watchObservedRunningTime="2026-10-29 22:34:10.913541392 +0000 UTC m=+31.696754818" Oct 29 22:34:10.997132 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:10.995136 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.3487872579999998 podStartE2EDuration="7.995118292s" totalImagesPullingTime="4.626198487s" totalInitContainerRuntime="20.132547ms" isStatefulPod=false podCreationTimestamp="2026-10-29 22:34:03 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:34:10.993806214 +0000 UTC m=+31.777019642" watchObservedRunningTime="2026-10-29 22:34:10.995118292 +0000 UTC m=+31.778331715" Oct 29 22:34:14.512306 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:14.512179 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:34:14.649187 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:14.649111 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:34:14.728157 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:14.725465 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"af0e8682-1347-4724-b0a4-870a0130fd00","Type":"ContainerStarted","Data":"877ec5d73ce266cb62cf4b6dfa9b9f2fb65b8aca6280dc8625734b5f6a362484"} Oct 29 22:34:14.728157 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:14.725589 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"af0e8682-1347-4724-b0a4-870a0130fd00","Type":"ContainerStarted","Data":"95113b796d327d72ba495de8bfa833fa28add556419768c49a579e4f5507e9fc"} Oct 29 22:34:14.728157 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:14.725633 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"af0e8682-1347-4724-b0a4-870a0130fd00","Type":"ContainerStarted","Data":"abb69760e74c80520fc09f6567ad98b02045f5b8b32abd44e52101dadaf64302"} Oct 29 22:34:14.730600 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:14.729439 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-9s8rp" event={"ID":"612c3068-fbdf-4cd3-bf23-4099b112c655","Type":"ContainerStarted","Data":"260f900d3f1bdf633ea9b707e5b52848a0cd92e844c790517f933e4469ef9068"} Oct 29 22:34:14.730600 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:14.730562 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6dd789c8b7-9s8rp" Oct 29 22:34:14.741523 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:14.741462 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6dd789c8b7-9s8rp" Oct 29 22:34:14.773919 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:14.773815 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:34:14.813302 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:14.809070 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/downloads-6dd789c8b7-9s8rp" podStartSLOduration=1.829768445 podStartE2EDuration="21.809050493s" totalImagesPullingTime="19.979282048s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:33:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:34:14.769988038 +0000 UTC m=+35.553201474" watchObservedRunningTime="2026-10-29 22:34:14.809050493 +0000 UTC m=+35.592263919" Oct 29 22:34:15.502173 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:15.502139 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-d49c68d56-vpbg5" Oct 29 22:34:15.577671 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:15.577638 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-84866f86bd-2v6d7"] Oct 29 22:34:15.702026 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:15.701991 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:34:15.740980 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:15.740941 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"af0e8682-1347-4724-b0a4-870a0130fd00","Type":"ContainerStarted","Data":"534a35847e5a1b683b1bcf4e588b625e3199cc9ab9d8f671d3bca0c479a18d7c"} Oct 29 22:34:15.741162 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:15.741025 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"af0e8682-1347-4724-b0a4-870a0130fd00","Type":"ContainerStarted","Data":"4dc31581b1d0ee53683e89f0f03e48a74495c944fa7696dc878c09e7a29274a8"} Oct 29 22:34:15.741162 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:15.741066 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"af0e8682-1347-4724-b0a4-870a0130fd00","Type":"ContainerStarted","Data":"38ebad248d161a5aea1cf5d3edb12d111de9a9851193124cb5419c726596116b"} Oct 29 22:34:15.802838 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:15.802710 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.200375733 podStartE2EDuration="6.802689119s" totalImagesPullingTime="3.574040959s" totalInitContainerRuntime="28.272427ms" isStatefulPod=false podCreationTimestamp="2026-10-29 22:34:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:34:15.8006022 +0000 UTC m=+36.583815627" watchObservedRunningTime="2026-10-29 22:34:15.802689119 +0000 UTC m=+36.585902544" Oct 29 22:34:15.844908 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:15.844872 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:34:15.953391 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:15.953317 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:34:16.699023 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:16.698990 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-74bbd64f85-jjdcv" Oct 29 22:34:16.838417 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:16.838383 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:34:16.946890 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:16.946783 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:34:18.063945 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:18.063882 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-78db97464f-6ljzh" podUID="5296979a-7927-4ee2-84b9-a93780a2ad22" containerName="registry" containerID="cri-o://41452c10b3c4ce37981bdce20cea5af734997d17e7c85f9662a559369ad9879b" gracePeriod=30 Oct 29 22:34:18.408228 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:18.407843 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-78db97464f-6ljzh" Oct 29 22:34:18.564846 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:18.564809 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5296979a-7927-4ee2-84b9-a93780a2ad22-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5296979a-7927-4ee2-84b9-a93780a2ad22-installation-pull-secrets\") pod \"5296979a-7927-4ee2-84b9-a93780a2ad22\" (UID: \"5296979a-7927-4ee2-84b9-a93780a2ad22\") " Oct 29 22:34:18.565080 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:18.564864 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5296979a-7927-4ee2-84b9-a93780a2ad22-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5296979a-7927-4ee2-84b9-a93780a2ad22-bound-sa-token\") pod \"5296979a-7927-4ee2-84b9-a93780a2ad22\" (UID: \"5296979a-7927-4ee2-84b9-a93780a2ad22\") " Oct 29 22:34:18.565080 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:18.564928 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5296979a-7927-4ee2-84b9-a93780a2ad22-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5296979a-7927-4ee2-84b9-a93780a2ad22-image-registry-private-configuration\") pod \"5296979a-7927-4ee2-84b9-a93780a2ad22\" (UID: \"5296979a-7927-4ee2-84b9-a93780a2ad22\") " Oct 29 22:34:18.565080 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:18.564963 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5296979a-7927-4ee2-84b9-a93780a2ad22-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5296979a-7927-4ee2-84b9-a93780a2ad22-ca-trust-extracted\") pod \"5296979a-7927-4ee2-84b9-a93780a2ad22\" (UID: \"5296979a-7927-4ee2-84b9-a93780a2ad22\") " Oct 29 22:34:18.565080 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:18.565005 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5296979a-7927-4ee2-84b9-a93780a2ad22-kube-api-access-fp2lh\" (UniqueName: \"kubernetes.io/projected/5296979a-7927-4ee2-84b9-a93780a2ad22-kube-api-access-fp2lh\") pod \"5296979a-7927-4ee2-84b9-a93780a2ad22\" (UID: \"5296979a-7927-4ee2-84b9-a93780a2ad22\") " Oct 29 22:34:18.565080 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:18.565047 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5296979a-7927-4ee2-84b9-a93780a2ad22-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5296979a-7927-4ee2-84b9-a93780a2ad22-trusted-ca\") pod \"5296979a-7927-4ee2-84b9-a93780a2ad22\" (UID: \"5296979a-7927-4ee2-84b9-a93780a2ad22\") " Oct 29 22:34:18.565389 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:18.565100 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5296979a-7927-4ee2-84b9-a93780a2ad22-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5296979a-7927-4ee2-84b9-a93780a2ad22-registry-certificates\") pod \"5296979a-7927-4ee2-84b9-a93780a2ad22\" (UID: \"5296979a-7927-4ee2-84b9-a93780a2ad22\") " Oct 29 22:34:18.565389 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:18.565210 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5296979a-7927-4ee2-84b9-a93780a2ad22-registry-tls\" (UniqueName: \"kubernetes.io/projected/5296979a-7927-4ee2-84b9-a93780a2ad22-registry-tls\") pod \"5296979a-7927-4ee2-84b9-a93780a2ad22\" (UID: \"5296979a-7927-4ee2-84b9-a93780a2ad22\") " Oct 29 22:34:18.565974 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:18.565939 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5296979a-7927-4ee2-84b9-a93780a2ad22-registry-certificates" pod "5296979a-7927-4ee2-84b9-a93780a2ad22" (UID: "5296979a-7927-4ee2-84b9-a93780a2ad22"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:34:18.566315 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:18.565997 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5296979a-7927-4ee2-84b9-a93780a2ad22-trusted-ca" pod "5296979a-7927-4ee2-84b9-a93780a2ad22" (UID: "5296979a-7927-4ee2-84b9-a93780a2ad22"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:34:18.568375 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:18.568349 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5296979a-7927-4ee2-84b9-a93780a2ad22-registry-tls" pod "5296979a-7927-4ee2-84b9-a93780a2ad22" (UID: "5296979a-7927-4ee2-84b9-a93780a2ad22"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:34:18.569658 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:18.569631 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5296979a-7927-4ee2-84b9-a93780a2ad22-image-registry-private-configuration" pod "5296979a-7927-4ee2-84b9-a93780a2ad22" (UID: "5296979a-7927-4ee2-84b9-a93780a2ad22"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:34:18.570804 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:18.570779 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5296979a-7927-4ee2-84b9-a93780a2ad22-kube-api-access-fp2lh" pod "5296979a-7927-4ee2-84b9-a93780a2ad22" (UID: "5296979a-7927-4ee2-84b9-a93780a2ad22"). InnerVolumeSpecName "kube-api-access-fp2lh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:34:18.570908 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:18.570880 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5296979a-7927-4ee2-84b9-a93780a2ad22-bound-sa-token" pod "5296979a-7927-4ee2-84b9-a93780a2ad22" (UID: "5296979a-7927-4ee2-84b9-a93780a2ad22"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:34:18.570971 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:18.570953 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5296979a-7927-4ee2-84b9-a93780a2ad22-installation-pull-secrets" pod "5296979a-7927-4ee2-84b9-a93780a2ad22" (UID: "5296979a-7927-4ee2-84b9-a93780a2ad22"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:34:18.575359 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:18.575310 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5296979a-7927-4ee2-84b9-a93780a2ad22-ca-trust-extracted" pod "5296979a-7927-4ee2-84b9-a93780a2ad22" (UID: "5296979a-7927-4ee2-84b9-a93780a2ad22"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:34:18.666449 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:18.666377 5057 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5296979a-7927-4ee2-84b9-a93780a2ad22-installation-pull-secrets\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 22:34:18.666449 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:18.666408 5057 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5296979a-7927-4ee2-84b9-a93780a2ad22-bound-sa-token\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 22:34:18.666449 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:18.666422 5057 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5296979a-7927-4ee2-84b9-a93780a2ad22-image-registry-private-configuration\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 22:34:18.666449 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:18.666435 5057 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5296979a-7927-4ee2-84b9-a93780a2ad22-ca-trust-extracted\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 22:34:18.666696 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:18.666464 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fp2lh\" (UniqueName: \"kubernetes.io/projected/5296979a-7927-4ee2-84b9-a93780a2ad22-kube-api-access-fp2lh\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 22:34:18.666696 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:18.666480 5057 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5296979a-7927-4ee2-84b9-a93780a2ad22-trusted-ca\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 22:34:18.666696 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:18.666492 5057 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5296979a-7927-4ee2-84b9-a93780a2ad22-registry-certificates\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 22:34:18.666696 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:18.666504 5057 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5296979a-7927-4ee2-84b9-a93780a2ad22-registry-tls\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 22:34:18.760303 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:18.760271 5057 generic.go:356] "Generic (PLEG): container finished" podID="5296979a-7927-4ee2-84b9-a93780a2ad22" containerID="41452c10b3c4ce37981bdce20cea5af734997d17e7c85f9662a559369ad9879b" exitCode=0 Oct 29 22:34:18.760467 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:18.760326 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-78db97464f-6ljzh" Oct 29 22:34:18.760467 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:18.760370 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-78db97464f-6ljzh" event={"ID":"5296979a-7927-4ee2-84b9-a93780a2ad22","Type":"ContainerDied","Data":"41452c10b3c4ce37981bdce20cea5af734997d17e7c85f9662a559369ad9879b"} Oct 29 22:34:18.760467 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:18.760432 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-78db97464f-6ljzh" event={"ID":"5296979a-7927-4ee2-84b9-a93780a2ad22","Type":"ContainerDied","Data":"aa9cd79eac96497d75021725e7d6f4a8c7a012f3c8fd7e520590844af17e72b2"} Oct 29 22:34:18.760467 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:18.760457 5057 scope.go:122] "RemoveContainer" containerID="41452c10b3c4ce37981bdce20cea5af734997d17e7c85f9662a559369ad9879b" Oct 29 22:34:18.772398 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:18.772367 5057 scope.go:122] "RemoveContainer" containerID="41452c10b3c4ce37981bdce20cea5af734997d17e7c85f9662a559369ad9879b" Oct 29 22:34:18.772728 ip-10-0-140-74 kubenswrapper[5057]: E1029 22:34:18.772684 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41452c10b3c4ce37981bdce20cea5af734997d17e7c85f9662a559369ad9879b\": container with ID starting with 41452c10b3c4ce37981bdce20cea5af734997d17e7c85f9662a559369ad9879b not found: ID does not exist" containerID="41452c10b3c4ce37981bdce20cea5af734997d17e7c85f9662a559369ad9879b" Oct 29 22:34:18.772813 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:18.772740 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41452c10b3c4ce37981bdce20cea5af734997d17e7c85f9662a559369ad9879b"} err="failed to get container status \"41452c10b3c4ce37981bdce20cea5af734997d17e7c85f9662a559369ad9879b\": rpc error: code = NotFound desc = could not find container \"41452c10b3c4ce37981bdce20cea5af734997d17e7c85f9662a559369ad9879b\": container with ID starting with 41452c10b3c4ce37981bdce20cea5af734997d17e7c85f9662a559369ad9879b not found: ID does not exist" Oct 29 22:34:18.807178 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:18.807155 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-78db97464f-6ljzh"] Oct 29 22:34:18.810604 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:18.810584 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-78db97464f-6ljzh"] Oct 29 22:34:19.032462 ip-10-0-140-74 kubenswrapper[5057]: E1029 22:34:19.032422 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d2dnn" containerName="extractor" Oct 29 22:34:19.328096 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:19.328021 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5296979a-7927-4ee2-84b9-a93780a2ad22" path="/var/lib/kubelet/pods/5296979a-7927-4ee2-84b9-a93780a2ad22/volumes" Oct 29 22:34:19.551877 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:19.551845 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:34:19.827371 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:19.827319 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-f8f65dd7c-j9qgf" Oct 29 22:34:19.827550 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:19.827410 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-f8f65dd7c-j9qgf" Oct 29 22:34:19.832950 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:19.832926 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-f8f65dd7c-j9qgf" Oct 29 22:34:20.175457 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:20.175379 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:34:20.291091 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:20.291053 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:34:20.778849 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:20.778765 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-f8f65dd7c-j9qgf" Oct 29 22:34:20.848180 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:20.848148 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-68ff86fd-nb5ft"] Oct 29 22:34:22.446289 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:22.446248 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:34:23.843605 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:23.843575 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:34:26.178687 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:26.178649 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:34:26.286507 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:26.286470 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:34:26.394090 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:26.394052 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:34:26.984422 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:26.984391 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-6f9dbf8897-m6njl" Oct 29 22:34:26.984709 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:26.984451 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-6f9dbf8897-m6njl" Oct 29 22:34:28.747154 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:28.747118 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:34:28.889916 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:28.889876 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:34:29.508844 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:29.508813 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:34:29.617890 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:29.617853 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:34:39.408927 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:39.408881 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:34:39.519646 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:39.519614 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:34:40.606451 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:40.606390 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-84866f86bd-2v6d7" podUID="989956e8-1893-4bbe-a4bf-6a2081509080" containerName="registry" containerID="cri-o://d76c049df3eb0de0f412cd20199664ec6cbad532ed34bc561691c39d2d0eb211" gracePeriod=30 Oct 29 22:34:40.870770 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:40.870743 5057 generic.go:356] "Generic (PLEG): container finished" podID="989956e8-1893-4bbe-a4bf-6a2081509080" containerID="d76c049df3eb0de0f412cd20199664ec6cbad532ed34bc561691c39d2d0eb211" exitCode=0 Oct 29 22:34:40.870875 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:40.870827 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-84866f86bd-2v6d7" event={"ID":"989956e8-1893-4bbe-a4bf-6a2081509080","Type":"ContainerDied","Data":"d76c049df3eb0de0f412cd20199664ec6cbad532ed34bc561691c39d2d0eb211"} Oct 29 22:34:40.908955 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:40.908929 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-84866f86bd-2v6d7" Oct 29 22:34:40.970758 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:40.970724 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/989956e8-1893-4bbe-a4bf-6a2081509080-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/989956e8-1893-4bbe-a4bf-6a2081509080-bound-sa-token\") pod \"989956e8-1893-4bbe-a4bf-6a2081509080\" (UID: \"989956e8-1893-4bbe-a4bf-6a2081509080\") " Oct 29 22:34:40.970922 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:40.970766 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/989956e8-1893-4bbe-a4bf-6a2081509080-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/989956e8-1893-4bbe-a4bf-6a2081509080-installation-pull-secrets\") pod \"989956e8-1893-4bbe-a4bf-6a2081509080\" (UID: \"989956e8-1893-4bbe-a4bf-6a2081509080\") " Oct 29 22:34:40.970922 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:40.970825 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/989956e8-1893-4bbe-a4bf-6a2081509080-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/989956e8-1893-4bbe-a4bf-6a2081509080-registry-certificates\") pod \"989956e8-1893-4bbe-a4bf-6a2081509080\" (UID: \"989956e8-1893-4bbe-a4bf-6a2081509080\") " Oct 29 22:34:40.970922 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:40.970850 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/989956e8-1893-4bbe-a4bf-6a2081509080-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/989956e8-1893-4bbe-a4bf-6a2081509080-trusted-ca\") pod \"989956e8-1893-4bbe-a4bf-6a2081509080\" (UID: \"989956e8-1893-4bbe-a4bf-6a2081509080\") " Oct 29 22:34:40.971194 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:40.971173 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/989956e8-1893-4bbe-a4bf-6a2081509080-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/989956e8-1893-4bbe-a4bf-6a2081509080-image-registry-private-configuration\") pod \"989956e8-1893-4bbe-a4bf-6a2081509080\" (UID: \"989956e8-1893-4bbe-a4bf-6a2081509080\") " Oct 29 22:34:40.971262 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:40.971221 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/989956e8-1893-4bbe-a4bf-6a2081509080-registry-tls\" (UniqueName: \"kubernetes.io/projected/989956e8-1893-4bbe-a4bf-6a2081509080-registry-tls\") pod \"989956e8-1893-4bbe-a4bf-6a2081509080\" (UID: \"989956e8-1893-4bbe-a4bf-6a2081509080\") " Oct 29 22:34:40.971316 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:40.971274 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/989956e8-1893-4bbe-a4bf-6a2081509080-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/989956e8-1893-4bbe-a4bf-6a2081509080-ca-trust-extracted\") pod \"989956e8-1893-4bbe-a4bf-6a2081509080\" (UID: \"989956e8-1893-4bbe-a4bf-6a2081509080\") " Oct 29 22:34:40.971404 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:40.971318 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/989956e8-1893-4bbe-a4bf-6a2081509080-kube-api-access-gfk2t\" (UniqueName: \"kubernetes.io/projected/989956e8-1893-4bbe-a4bf-6a2081509080-kube-api-access-gfk2t\") pod \"989956e8-1893-4bbe-a4bf-6a2081509080\" (UID: \"989956e8-1893-4bbe-a4bf-6a2081509080\") " Oct 29 22:34:40.971838 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:40.971312 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/989956e8-1893-4bbe-a4bf-6a2081509080-registry-certificates" pod "989956e8-1893-4bbe-a4bf-6a2081509080" (UID: "989956e8-1893-4bbe-a4bf-6a2081509080"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:34:40.971838 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:40.971355 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/989956e8-1893-4bbe-a4bf-6a2081509080-trusted-ca" pod "989956e8-1893-4bbe-a4bf-6a2081509080" (UID: "989956e8-1893-4bbe-a4bf-6a2081509080"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:34:40.971838 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:40.971789 5057 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/989956e8-1893-4bbe-a4bf-6a2081509080-registry-certificates\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 22:34:40.971838 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:40.971816 5057 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/989956e8-1893-4bbe-a4bf-6a2081509080-trusted-ca\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 22:34:40.974359 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:40.973845 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/989956e8-1893-4bbe-a4bf-6a2081509080-image-registry-private-configuration" pod "989956e8-1893-4bbe-a4bf-6a2081509080" (UID: "989956e8-1893-4bbe-a4bf-6a2081509080"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:34:40.974359 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:40.973858 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/989956e8-1893-4bbe-a4bf-6a2081509080-bound-sa-token" pod "989956e8-1893-4bbe-a4bf-6a2081509080" (UID: "989956e8-1893-4bbe-a4bf-6a2081509080"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:34:40.974359 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:40.973994 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/989956e8-1893-4bbe-a4bf-6a2081509080-kube-api-access-gfk2t" pod "989956e8-1893-4bbe-a4bf-6a2081509080" (UID: "989956e8-1893-4bbe-a4bf-6a2081509080"). InnerVolumeSpecName "kube-api-access-gfk2t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:34:40.974359 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:40.973996 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/989956e8-1893-4bbe-a4bf-6a2081509080-installation-pull-secrets" pod "989956e8-1893-4bbe-a4bf-6a2081509080" (UID: "989956e8-1893-4bbe-a4bf-6a2081509080"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:34:40.974359 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:40.974092 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/989956e8-1893-4bbe-a4bf-6a2081509080-registry-tls" pod "989956e8-1893-4bbe-a4bf-6a2081509080" (UID: "989956e8-1893-4bbe-a4bf-6a2081509080"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:34:40.978666 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:40.978639 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/989956e8-1893-4bbe-a4bf-6a2081509080-ca-trust-extracted" pod "989956e8-1893-4bbe-a4bf-6a2081509080" (UID: "989956e8-1893-4bbe-a4bf-6a2081509080"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:34:41.072528 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:41.072503 5057 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/989956e8-1893-4bbe-a4bf-6a2081509080-installation-pull-secrets\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 22:34:41.072528 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:41.072528 5057 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/989956e8-1893-4bbe-a4bf-6a2081509080-image-registry-private-configuration\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 22:34:41.072656 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:41.072540 5057 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/989956e8-1893-4bbe-a4bf-6a2081509080-registry-tls\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 22:34:41.072656 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:41.072549 5057 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/989956e8-1893-4bbe-a4bf-6a2081509080-ca-trust-extracted\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 22:34:41.072656 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:41.072558 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gfk2t\" (UniqueName: \"kubernetes.io/projected/989956e8-1893-4bbe-a4bf-6a2081509080-kube-api-access-gfk2t\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 22:34:41.072656 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:41.072566 5057 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/989956e8-1893-4bbe-a4bf-6a2081509080-bound-sa-token\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 22:34:41.877205 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:41.877181 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-84866f86bd-2v6d7" Oct 29 22:34:41.877699 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:41.877207 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-84866f86bd-2v6d7" event={"ID":"989956e8-1893-4bbe-a4bf-6a2081509080","Type":"ContainerDied","Data":"1a573d2d443f71339edf3b0caa40e76794c0ede1c2c257700a55fd6a80fe572d"} Oct 29 22:34:41.877699 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:41.877263 5057 scope.go:122] "RemoveContainer" containerID="d76c049df3eb0de0f412cd20199664ec6cbad532ed34bc561691c39d2d0eb211" Oct 29 22:34:41.934006 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:41.933979 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-84866f86bd-2v6d7"] Oct 29 22:34:41.944124 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:41.944098 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-84866f86bd-2v6d7"] Oct 29 22:34:43.325979 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:43.325942 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="989956e8-1893-4bbe-a4bf-6a2081509080" path="/var/lib/kubelet/pods/989956e8-1893-4bbe-a4bf-6a2081509080/volumes" Oct 29 22:34:44.513016 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:44.512984 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:34:44.627365 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:44.627319 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:34:45.875025 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:45.874990 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-68ff86fd-nb5ft" podUID="166d00be-a17a-41ba-9e75-7b7a4217b788" containerName="console" containerID="cri-o://eddaad68ee7f78b328416cbc237c3a6ba7e4aff3b76387ac5bbe69dddcaef492" gracePeriod=15 Oct 29 22:34:46.148153 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:46.148130 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-68ff86fd-nb5ft" Oct 29 22:34:46.219959 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:46.219931 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/166d00be-a17a-41ba-9e75-7b7a4217b788-console-config\" (UniqueName: \"kubernetes.io/configmap/166d00be-a17a-41ba-9e75-7b7a4217b788-console-config\") pod \"166d00be-a17a-41ba-9e75-7b7a4217b788\" (UID: \"166d00be-a17a-41ba-9e75-7b7a4217b788\") " Oct 29 22:34:46.220124 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:46.219972 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/166d00be-a17a-41ba-9e75-7b7a4217b788-tmp\" (UniqueName: \"kubernetes.io/empty-dir/166d00be-a17a-41ba-9e75-7b7a4217b788-tmp\") pod \"166d00be-a17a-41ba-9e75-7b7a4217b788\" (UID: \"166d00be-a17a-41ba-9e75-7b7a4217b788\") " Oct 29 22:34:46.220124 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:46.219992 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/166d00be-a17a-41ba-9e75-7b7a4217b788-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/166d00be-a17a-41ba-9e75-7b7a4217b788-console-serving-cert\") pod \"166d00be-a17a-41ba-9e75-7b7a4217b788\" (UID: \"166d00be-a17a-41ba-9e75-7b7a4217b788\") " Oct 29 22:34:46.220124 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:46.220089 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/166d00be-a17a-41ba-9e75-7b7a4217b788-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/166d00be-a17a-41ba-9e75-7b7a4217b788-console-oauth-config\") pod \"166d00be-a17a-41ba-9e75-7b7a4217b788\" (UID: \"166d00be-a17a-41ba-9e75-7b7a4217b788\") " Oct 29 22:34:46.220124 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:46.220118 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/166d00be-a17a-41ba-9e75-7b7a4217b788-kube-api-access-q7lfv\" (UniqueName: \"kubernetes.io/projected/166d00be-a17a-41ba-9e75-7b7a4217b788-kube-api-access-q7lfv\") pod \"166d00be-a17a-41ba-9e75-7b7a4217b788\" (UID: \"166d00be-a17a-41ba-9e75-7b7a4217b788\") " Oct 29 22:34:46.220367 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:46.220176 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/166d00be-a17a-41ba-9e75-7b7a4217b788-service-ca\" (UniqueName: \"kubernetes.io/configmap/166d00be-a17a-41ba-9e75-7b7a4217b788-service-ca\") pod \"166d00be-a17a-41ba-9e75-7b7a4217b788\" (UID: \"166d00be-a17a-41ba-9e75-7b7a4217b788\") " Oct 29 22:34:46.220367 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:46.220203 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/166d00be-a17a-41ba-9e75-7b7a4217b788-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/166d00be-a17a-41ba-9e75-7b7a4217b788-oauth-serving-cert\") pod \"166d00be-a17a-41ba-9e75-7b7a4217b788\" (UID: \"166d00be-a17a-41ba-9e75-7b7a4217b788\") " Oct 29 22:34:46.220480 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:46.220387 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/166d00be-a17a-41ba-9e75-7b7a4217b788-tmp" pod "166d00be-a17a-41ba-9e75-7b7a4217b788" (UID: "166d00be-a17a-41ba-9e75-7b7a4217b788"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:34:46.220480 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:46.220454 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/166d00be-a17a-41ba-9e75-7b7a4217b788-console-config" pod "166d00be-a17a-41ba-9e75-7b7a4217b788" (UID: "166d00be-a17a-41ba-9e75-7b7a4217b788"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:34:46.220704 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:46.220673 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/166d00be-a17a-41ba-9e75-7b7a4217b788-service-ca" pod "166d00be-a17a-41ba-9e75-7b7a4217b788" (UID: "166d00be-a17a-41ba-9e75-7b7a4217b788"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:34:46.220790 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:46.220689 5057 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/166d00be-a17a-41ba-9e75-7b7a4217b788-console-config\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 22:34:46.220790 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:46.220730 5057 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/166d00be-a17a-41ba-9e75-7b7a4217b788-tmp\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 22:34:46.220893 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:46.220796 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/166d00be-a17a-41ba-9e75-7b7a4217b788-oauth-serving-cert" pod "166d00be-a17a-41ba-9e75-7b7a4217b788" (UID: "166d00be-a17a-41ba-9e75-7b7a4217b788"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:34:46.222263 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:46.222237 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/166d00be-a17a-41ba-9e75-7b7a4217b788-console-oauth-config" pod "166d00be-a17a-41ba-9e75-7b7a4217b788" (UID: "166d00be-a17a-41ba-9e75-7b7a4217b788"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:34:46.222465 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:46.222284 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/166d00be-a17a-41ba-9e75-7b7a4217b788-kube-api-access-q7lfv" pod "166d00be-a17a-41ba-9e75-7b7a4217b788" (UID: "166d00be-a17a-41ba-9e75-7b7a4217b788"). InnerVolumeSpecName "kube-api-access-q7lfv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:34:46.222465 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:46.222309 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/166d00be-a17a-41ba-9e75-7b7a4217b788-console-serving-cert" pod "166d00be-a17a-41ba-9e75-7b7a4217b788" (UID: "166d00be-a17a-41ba-9e75-7b7a4217b788"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:34:46.321256 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:46.321229 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q7lfv\" (UniqueName: \"kubernetes.io/projected/166d00be-a17a-41ba-9e75-7b7a4217b788-kube-api-access-q7lfv\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 22:34:46.321256 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:46.321255 5057 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/166d00be-a17a-41ba-9e75-7b7a4217b788-service-ca\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 22:34:46.321415 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:46.321265 5057 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/166d00be-a17a-41ba-9e75-7b7a4217b788-oauth-serving-cert\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 22:34:46.321415 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:46.321282 5057 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/166d00be-a17a-41ba-9e75-7b7a4217b788-console-serving-cert\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 22:34:46.321415 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:46.321290 5057 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/166d00be-a17a-41ba-9e75-7b7a4217b788-console-oauth-config\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 22:34:46.907512 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:46.907476 5057 generic.go:356] "Generic (PLEG): container finished" podID="166d00be-a17a-41ba-9e75-7b7a4217b788" containerID="eddaad68ee7f78b328416cbc237c3a6ba7e4aff3b76387ac5bbe69dddcaef492" exitCode=0 Oct 29 22:34:46.907910 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:46.907524 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-68ff86fd-nb5ft" Oct 29 22:34:46.907910 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:46.907564 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-68ff86fd-nb5ft" event={"ID":"166d00be-a17a-41ba-9e75-7b7a4217b788","Type":"ContainerDied","Data":"eddaad68ee7f78b328416cbc237c3a6ba7e4aff3b76387ac5bbe69dddcaef492"} Oct 29 22:34:46.907910 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:46.907614 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-68ff86fd-nb5ft" event={"ID":"166d00be-a17a-41ba-9e75-7b7a4217b788","Type":"ContainerDied","Data":"6d0fc88b264bca0b71bf97f6d9bc334e78a294f862bc08f55769c40eeb924071"} Oct 29 22:34:46.907910 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:46.907634 5057 scope.go:122] "RemoveContainer" containerID="eddaad68ee7f78b328416cbc237c3a6ba7e4aff3b76387ac5bbe69dddcaef492" Oct 29 22:34:46.916616 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:46.916598 5057 scope.go:122] "RemoveContainer" containerID="eddaad68ee7f78b328416cbc237c3a6ba7e4aff3b76387ac5bbe69dddcaef492" Oct 29 22:34:46.916874 ip-10-0-140-74 kubenswrapper[5057]: E1029 22:34:46.916858 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eddaad68ee7f78b328416cbc237c3a6ba7e4aff3b76387ac5bbe69dddcaef492\": container with ID starting with eddaad68ee7f78b328416cbc237c3a6ba7e4aff3b76387ac5bbe69dddcaef492 not found: ID does not exist" containerID="eddaad68ee7f78b328416cbc237c3a6ba7e4aff3b76387ac5bbe69dddcaef492" Oct 29 22:34:46.916939 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:46.916881 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eddaad68ee7f78b328416cbc237c3a6ba7e4aff3b76387ac5bbe69dddcaef492"} err="failed to get container status \"eddaad68ee7f78b328416cbc237c3a6ba7e4aff3b76387ac5bbe69dddcaef492\": rpc error: code = NotFound desc = could not find container \"eddaad68ee7f78b328416cbc237c3a6ba7e4aff3b76387ac5bbe69dddcaef492\": container with ID starting with eddaad68ee7f78b328416cbc237c3a6ba7e4aff3b76387ac5bbe69dddcaef492 not found: ID does not exist" Oct 29 22:34:46.952353 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:46.952317 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-68ff86fd-nb5ft"] Oct 29 22:34:46.962624 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:46.962602 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-68ff86fd-nb5ft"] Oct 29 22:34:46.989372 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:46.989349 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-6f9dbf8897-m6njl" Oct 29 22:34:46.992904 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:46.992888 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-6f9dbf8897-m6njl" Oct 29 22:34:47.331868 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:47.331834 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="166d00be-a17a-41ba-9e75-7b7a4217b788" path="/var/lib/kubelet/pods/166d00be-a17a-41ba-9e75-7b7a4217b788/volumes" Oct 29 22:34:49.026872 ip-10-0-140-74 kubenswrapper[5057]: E1029 22:34:49.026838 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d2dnn" containerName="extractor" Oct 29 22:34:55.952998 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:55.952966 5057 generic.go:356] "Generic (PLEG): container finished" podID="f176efe0-febd-4652-94fc-91548ee3ab93" containerID="b7903026ceba7033f380cb6218731c4d3fc52045ff5b91e796c1a3658dd2219d" exitCode=0 Oct 29 22:34:55.953461 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:55.953048 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-lnrkb-btcvq" event={"ID":"f176efe0-febd-4652-94fc-91548ee3ab93","Type":"ContainerDied","Data":"b7903026ceba7033f380cb6218731c4d3fc52045ff5b91e796c1a3658dd2219d"} Oct 29 22:34:57.158285 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:57.158258 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-lnrkb-btcvq" Oct 29 22:34:57.220884 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:57.220828 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f176efe0-febd-4652-94fc-91548ee3ab93-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/f176efe0-febd-4652-94fc-91548ee3ab93-archives-path\") pod \"f176efe0-febd-4652-94fc-91548ee3ab93\" (UID: \"f176efe0-febd-4652-94fc-91548ee3ab93\") " Oct 29 22:34:57.221002 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:57.220895 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f176efe0-febd-4652-94fc-91548ee3ab93-kube-api-access-p9pvq\" (UniqueName: \"kubernetes.io/projected/f176efe0-febd-4652-94fc-91548ee3ab93-kube-api-access-p9pvq\") pod \"f176efe0-febd-4652-94fc-91548ee3ab93\" (UID: \"f176efe0-febd-4652-94fc-91548ee3ab93\") " Oct 29 22:34:57.221002 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:57.220982 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f176efe0-febd-4652-94fc-91548ee3ab93-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f176efe0-febd-4652-94fc-91548ee3ab93-service-ca-bundle\") pod \"f176efe0-febd-4652-94fc-91548ee3ab93\" (UID: \"f176efe0-febd-4652-94fc-91548ee3ab93\") " Oct 29 22:34:57.221096 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:57.221018 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f176efe0-febd-4652-94fc-91548ee3ab93-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f176efe0-febd-4652-94fc-91548ee3ab93-trusted-ca-bundle\") pod \"f176efe0-febd-4652-94fc-91548ee3ab93\" (UID: \"f176efe0-febd-4652-94fc-91548ee3ab93\") " Oct 29 22:34:57.221150 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:57.221123 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f176efe0-febd-4652-94fc-91548ee3ab93-archives-path" pod "f176efe0-febd-4652-94fc-91548ee3ab93" (UID: "f176efe0-febd-4652-94fc-91548ee3ab93"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:34:57.228167 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:57.221648 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f176efe0-febd-4652-94fc-91548ee3ab93-service-ca-bundle" pod "f176efe0-febd-4652-94fc-91548ee3ab93" (UID: "f176efe0-febd-4652-94fc-91548ee3ab93"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:34:57.228167 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:57.221864 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f176efe0-febd-4652-94fc-91548ee3ab93-trusted-ca-bundle" pod "f176efe0-febd-4652-94fc-91548ee3ab93" (UID: "f176efe0-febd-4652-94fc-91548ee3ab93"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:34:57.228167 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:57.221978 5057 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/f176efe0-febd-4652-94fc-91548ee3ab93-archives-path\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 22:34:57.228471 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:57.228443 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f176efe0-febd-4652-94fc-91548ee3ab93-kube-api-access-p9pvq" pod "f176efe0-febd-4652-94fc-91548ee3ab93" (UID: "f176efe0-febd-4652-94fc-91548ee3ab93"). InnerVolumeSpecName "kube-api-access-p9pvq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:34:57.323144 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:57.323120 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p9pvq\" (UniqueName: \"kubernetes.io/projected/f176efe0-febd-4652-94fc-91548ee3ab93-kube-api-access-p9pvq\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 22:34:57.323144 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:57.323144 5057 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f176efe0-febd-4652-94fc-91548ee3ab93-service-ca-bundle\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 22:34:57.323281 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:57.323154 5057 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f176efe0-febd-4652-94fc-91548ee3ab93-trusted-ca-bundle\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 22:34:57.964032 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:57.964000 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-lnrkb-btcvq" Oct 29 22:34:57.964032 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:57.964011 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-lnrkb-btcvq" event={"ID":"f176efe0-febd-4652-94fc-91548ee3ab93","Type":"ContainerDied","Data":"137914b9aa694c6fd2799975b8439b50e6c4d98b6b27025bd3874923c51ad0c6"} Oct 29 22:34:57.964257 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:57.964071 5057 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="137914b9aa694c6fd2799975b8439b50e6c4d98b6b27025bd3874923c51ad0c6" Oct 29 22:34:58.476813 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:58.476786 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:34:58.586909 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:34:58.586867 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:35:01.165147 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:01.165118 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:35:01.182018 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:01.181992 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:35:01.998676 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:01.998651 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:35:12.535166 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:12.534980 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:35:12.645552 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:12.645521 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:35:13.873688 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:13.873654 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 29 22:35:13.874097 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:13.873967 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="86214365-1f4f-41a0-b8be-e2e11144241e" containerName="alertmanager" containerID="cri-o://b1bb7d0f6b64084459c8859945660d9620d9ee72b4bb1d6e71bd3fc7b4632a65" gracePeriod=120 Oct 29 22:35:13.874097 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:13.874008 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="86214365-1f4f-41a0-b8be-e2e11144241e" containerName="kube-rbac-proxy-metric" containerID="cri-o://513b36d25a3af6474b29848cb8dca81da4c37d4dbae3fcc414e7665eb7aa181d" gracePeriod=120 Oct 29 22:35:13.874097 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:13.874085 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="86214365-1f4f-41a0-b8be-e2e11144241e" containerName="kube-rbac-proxy-web" containerID="cri-o://4662b0a105ae1bd58a9fbef8995c2b945b6a0965208374d64f6c55368f5c9e8f" gracePeriod=120 Oct 29 22:35:13.874267 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:13.874116 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="86214365-1f4f-41a0-b8be-e2e11144241e" containerName="config-reloader" containerID="cri-o://533c17e22fcf31771228a2991b31d812919dfdb55880fbd7188a5dfcdfb63e89" gracePeriod=120 Oct 29 22:35:13.874267 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:13.874089 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="86214365-1f4f-41a0-b8be-e2e11144241e" containerName="prom-label-proxy" containerID="cri-o://f4bb10070eb88aedd6ba257addc8ffe8589b7b09b3e81a6a05690bc09559f042" gracePeriod=120 Oct 29 22:35:13.874267 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:13.874179 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="86214365-1f4f-41a0-b8be-e2e11144241e" containerName="kube-rbac-proxy" containerID="cri-o://750111f4ae99d3b9585092893f4af8b45fec6791220c9a472bc861dba69a4888" gracePeriod=120 Oct 29 22:35:14.040056 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:14.040026 5057 generic.go:356] "Generic (PLEG): container finished" podID="86214365-1f4f-41a0-b8be-e2e11144241e" containerID="f4bb10070eb88aedd6ba257addc8ffe8589b7b09b3e81a6a05690bc09559f042" exitCode=0 Oct 29 22:35:14.040056 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:14.040050 5057 generic.go:356] "Generic (PLEG): container finished" podID="86214365-1f4f-41a0-b8be-e2e11144241e" containerID="750111f4ae99d3b9585092893f4af8b45fec6791220c9a472bc861dba69a4888" exitCode=0 Oct 29 22:35:14.040056 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:14.040056 5057 generic.go:356] "Generic (PLEG): container finished" podID="86214365-1f4f-41a0-b8be-e2e11144241e" containerID="533c17e22fcf31771228a2991b31d812919dfdb55880fbd7188a5dfcdfb63e89" exitCode=0 Oct 29 22:35:14.040221 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:14.040063 5057 generic.go:356] "Generic (PLEG): container finished" podID="86214365-1f4f-41a0-b8be-e2e11144241e" containerID="b1bb7d0f6b64084459c8859945660d9620d9ee72b4bb1d6e71bd3fc7b4632a65" exitCode=0 Oct 29 22:35:14.040221 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:14.040099 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"86214365-1f4f-41a0-b8be-e2e11144241e","Type":"ContainerDied","Data":"f4bb10070eb88aedd6ba257addc8ffe8589b7b09b3e81a6a05690bc09559f042"} Oct 29 22:35:14.040221 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:14.040160 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"86214365-1f4f-41a0-b8be-e2e11144241e","Type":"ContainerDied","Data":"750111f4ae99d3b9585092893f4af8b45fec6791220c9a472bc861dba69a4888"} Oct 29 22:35:14.040221 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:14.040187 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"86214365-1f4f-41a0-b8be-e2e11144241e","Type":"ContainerDied","Data":"533c17e22fcf31771228a2991b31d812919dfdb55880fbd7188a5dfcdfb63e89"} Oct 29 22:35:14.040221 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:14.040216 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"86214365-1f4f-41a0-b8be-e2e11144241e","Type":"ContainerDied","Data":"b1bb7d0f6b64084459c8859945660d9620d9ee72b4bb1d6e71bd3fc7b4632a65"} Oct 29 22:35:14.041620 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:14.041601 5057 generic.go:356] "Generic (PLEG): container finished" podID="65c96dfb-06e9-4bea-bb83-f63b4c62d860" containerID="9bd7d26c51b8fd48c4f3e9eb445aebd730ae353e8e68dc8efd4cd906f797088e" exitCode=0 Oct 29 22:35:14.041711 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:14.041648 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-b7bwd" event={"ID":"65c96dfb-06e9-4bea-bb83-f63b4c62d860","Type":"ContainerDied","Data":"9bd7d26c51b8fd48c4f3e9eb445aebd730ae353e8e68dc8efd4cd906f797088e"} Oct 29 22:35:14.041963 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:14.041951 5057 scope.go:122] "RemoveContainer" containerID="9bd7d26c51b8fd48c4f3e9eb445aebd730ae353e8e68dc8efd4cd906f797088e" Oct 29 22:35:14.678373 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:14.678324 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-ktp5j-2qdc5"] Oct 29 22:35:14.678886 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:14.678866 5057 cpu_manager.go:414] "containerMap: removing container" podUID="5296979a-7927-4ee2-84b9-a93780a2ad22" containerName="registry" Oct 29 22:35:14.678886 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:14.678886 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5296979a-7927-4ee2-84b9-a93780a2ad22" containerName="registry" Oct 29 22:35:14.679064 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:14.678917 5057 cpu_manager.go:414] "containerMap: removing container" podUID="f176efe0-febd-4652-94fc-91548ee3ab93" containerName="insights-gathering" Oct 29 22:35:14.679064 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:14.678925 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f176efe0-febd-4652-94fc-91548ee3ab93" containerName="insights-gathering" Oct 29 22:35:14.679064 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:14.678943 5057 cpu_manager.go:414] "containerMap: removing container" podUID="166d00be-a17a-41ba-9e75-7b7a4217b788" containerName="console" Oct 29 22:35:14.679064 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:14.678951 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="166d00be-a17a-41ba-9e75-7b7a4217b788" containerName="console" Oct 29 22:35:14.679064 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:14.678986 5057 cpu_manager.go:414] "containerMap: removing container" podUID="989956e8-1893-4bbe-a4bf-6a2081509080" containerName="registry" Oct 29 22:35:14.679064 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:14.678993 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="989956e8-1893-4bbe-a4bf-6a2081509080" containerName="registry" Oct 29 22:35:14.679374 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:14.679113 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="5296979a-7927-4ee2-84b9-a93780a2ad22" containerName="registry" Oct 29 22:35:14.679374 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:14.679128 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="f176efe0-febd-4652-94fc-91548ee3ab93" containerName="insights-gathering" Oct 29 22:35:14.679374 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:14.679139 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="166d00be-a17a-41ba-9e75-7b7a4217b788" containerName="console" Oct 29 22:35:14.679374 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:14.679152 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="989956e8-1893-4bbe-a4bf-6a2081509080" containerName="registry" Oct 29 22:35:14.681433 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:14.681413 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-ktp5j-2qdc5" Oct 29 22:35:14.685157 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:14.685139 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/84569698-cff8-449e-9acc-2c8408581d52-trusted-ca-bundle\") pod \"periodic-gathering-ktp5j-2qdc5\" (UID: \"84569698-cff8-449e-9acc-2c8408581d52\") " pod="openshift-insights/periodic-gathering-ktp5j-2qdc5" Oct 29 22:35:14.685250 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:14.685182 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/84569698-cff8-449e-9acc-2c8408581d52-archives-path\") pod \"periodic-gathering-ktp5j-2qdc5\" (UID: \"84569698-cff8-449e-9acc-2c8408581d52\") " pod="openshift-insights/periodic-gathering-ktp5j-2qdc5" Oct 29 22:35:14.685295 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:14.685278 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mpdqq\" (UniqueName: \"kubernetes.io/projected/84569698-cff8-449e-9acc-2c8408581d52-kube-api-access-mpdqq\") pod \"periodic-gathering-ktp5j-2qdc5\" (UID: \"84569698-cff8-449e-9acc-2c8408581d52\") " pod="openshift-insights/periodic-gathering-ktp5j-2qdc5" Oct 29 22:35:14.685438 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:14.685416 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/84569698-cff8-449e-9acc-2c8408581d52-service-ca-bundle\") pod \"periodic-gathering-ktp5j-2qdc5\" (UID: \"84569698-cff8-449e-9acc-2c8408581d52\") " pod="openshift-insights/periodic-gathering-ktp5j-2qdc5" Oct 29 22:35:14.700409 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:14.700384 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-ktp5j-2qdc5"] Oct 29 22:35:14.786205 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:14.786173 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/84569698-cff8-449e-9acc-2c8408581d52-service-ca-bundle\") pod \"periodic-gathering-ktp5j-2qdc5\" (UID: \"84569698-cff8-449e-9acc-2c8408581d52\") " pod="openshift-insights/periodic-gathering-ktp5j-2qdc5" Oct 29 22:35:14.786320 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:14.786218 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/84569698-cff8-449e-9acc-2c8408581d52-trusted-ca-bundle\") pod \"periodic-gathering-ktp5j-2qdc5\" (UID: \"84569698-cff8-449e-9acc-2c8408581d52\") " pod="openshift-insights/periodic-gathering-ktp5j-2qdc5" Oct 29 22:35:14.786320 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:14.786254 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/84569698-cff8-449e-9acc-2c8408581d52-archives-path\") pod \"periodic-gathering-ktp5j-2qdc5\" (UID: \"84569698-cff8-449e-9acc-2c8408581d52\") " pod="openshift-insights/periodic-gathering-ktp5j-2qdc5" Oct 29 22:35:14.786320 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:14.786283 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpdqq\" (UniqueName: \"kubernetes.io/projected/84569698-cff8-449e-9acc-2c8408581d52-kube-api-access-mpdqq\") pod \"periodic-gathering-ktp5j-2qdc5\" (UID: \"84569698-cff8-449e-9acc-2c8408581d52\") " pod="openshift-insights/periodic-gathering-ktp5j-2qdc5" Oct 29 22:35:14.786637 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:14.786617 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/84569698-cff8-449e-9acc-2c8408581d52-archives-path\") pod \"periodic-gathering-ktp5j-2qdc5\" (UID: \"84569698-cff8-449e-9acc-2c8408581d52\") " pod="openshift-insights/periodic-gathering-ktp5j-2qdc5" Oct 29 22:35:14.786808 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:14.786789 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/84569698-cff8-449e-9acc-2c8408581d52-service-ca-bundle\") pod \"periodic-gathering-ktp5j-2qdc5\" (UID: \"84569698-cff8-449e-9acc-2c8408581d52\") " pod="openshift-insights/periodic-gathering-ktp5j-2qdc5" Oct 29 22:35:14.787157 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:14.787134 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/84569698-cff8-449e-9acc-2c8408581d52-trusted-ca-bundle\") pod \"periodic-gathering-ktp5j-2qdc5\" (UID: \"84569698-cff8-449e-9acc-2c8408581d52\") " pod="openshift-insights/periodic-gathering-ktp5j-2qdc5" Oct 29 22:35:14.807828 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:14.807796 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mpdqq\" (UniqueName: \"kubernetes.io/projected/84569698-cff8-449e-9acc-2c8408581d52-kube-api-access-mpdqq\") pod \"periodic-gathering-ktp5j-2qdc5\" (UID: \"84569698-cff8-449e-9acc-2c8408581d52\") " pod="openshift-insights/periodic-gathering-ktp5j-2qdc5" Oct 29 22:35:14.992736 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:14.992711 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-ktp5j-2qdc5" Oct 29 22:35:15.053479 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:15.053448 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-b7bwd" event={"ID":"65c96dfb-06e9-4bea-bb83-f63b4c62d860","Type":"ContainerStarted","Data":"f9224649e6f37e5dec28c18ec28b42ad994eefa9b9b1b02e913bdb9517b66d42"} Oct 29 22:35:15.058803 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:15.058765 5057 generic.go:356] "Generic (PLEG): container finished" podID="86214365-1f4f-41a0-b8be-e2e11144241e" containerID="513b36d25a3af6474b29848cb8dca81da4c37d4dbae3fcc414e7665eb7aa181d" exitCode=0 Oct 29 22:35:15.058803 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:15.058804 5057 generic.go:356] "Generic (PLEG): container finished" podID="86214365-1f4f-41a0-b8be-e2e11144241e" containerID="4662b0a105ae1bd58a9fbef8995c2b945b6a0965208374d64f6c55368f5c9e8f" exitCode=0 Oct 29 22:35:15.058989 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:15.058848 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"86214365-1f4f-41a0-b8be-e2e11144241e","Type":"ContainerDied","Data":"513b36d25a3af6474b29848cb8dca81da4c37d4dbae3fcc414e7665eb7aa181d"} Oct 29 22:35:15.058989 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:15.058914 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"86214365-1f4f-41a0-b8be-e2e11144241e","Type":"ContainerDied","Data":"4662b0a105ae1bd58a9fbef8995c2b945b6a0965208374d64f6c55368f5c9e8f"} Oct 29 22:35:15.186718 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:15.186659 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:35:15.189240 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:15.189220 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/86214365-1f4f-41a0-b8be-e2e11144241e-web-config\" (UniqueName: \"kubernetes.io/secret/86214365-1f4f-41a0-b8be-e2e11144241e-web-config\") pod \"86214365-1f4f-41a0-b8be-e2e11144241e\" (UID: \"86214365-1f4f-41a0-b8be-e2e11144241e\") " Oct 29 22:35:15.189307 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:15.189266 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86214365-1f4f-41a0-b8be-e2e11144241e-kube-api-access-z5md5\" (UniqueName: \"kubernetes.io/projected/86214365-1f4f-41a0-b8be-e2e11144241e-kube-api-access-z5md5\") pod \"86214365-1f4f-41a0-b8be-e2e11144241e\" (UID: \"86214365-1f4f-41a0-b8be-e2e11144241e\") " Oct 29 22:35:15.189307 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:15.189289 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/86214365-1f4f-41a0-b8be-e2e11144241e-secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/86214365-1f4f-41a0-b8be-e2e11144241e-secret-alertmanager-kube-rbac-proxy\") pod \"86214365-1f4f-41a0-b8be-e2e11144241e\" (UID: \"86214365-1f4f-41a0-b8be-e2e11144241e\") " Oct 29 22:35:15.189439 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:15.189315 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/86214365-1f4f-41a0-b8be-e2e11144241e-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/86214365-1f4f-41a0-b8be-e2e11144241e-cluster-tls-config\") pod \"86214365-1f4f-41a0-b8be-e2e11144241e\" (UID: \"86214365-1f4f-41a0-b8be-e2e11144241e\") " Oct 29 22:35:15.189439 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:15.189369 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86214365-1f4f-41a0-b8be-e2e11144241e-alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/86214365-1f4f-41a0-b8be-e2e11144241e-alertmanager-main-db\") pod \"86214365-1f4f-41a0-b8be-e2e11144241e\" (UID: \"86214365-1f4f-41a0-b8be-e2e11144241e\") " Oct 29 22:35:15.189546 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:15.189490 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/86214365-1f4f-41a0-b8be-e2e11144241e-alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/86214365-1f4f-41a0-b8be-e2e11144241e-alertmanager-trusted-ca-bundle\") pod \"86214365-1f4f-41a0-b8be-e2e11144241e\" (UID: \"86214365-1f4f-41a0-b8be-e2e11144241e\") " Oct 29 22:35:15.189601 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:15.189558 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/86214365-1f4f-41a0-b8be-e2e11144241e-config-volume\" (UniqueName: \"kubernetes.io/secret/86214365-1f4f-41a0-b8be-e2e11144241e-config-volume\") pod \"86214365-1f4f-41a0-b8be-e2e11144241e\" (UID: \"86214365-1f4f-41a0-b8be-e2e11144241e\") " Oct 29 22:35:15.189663 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:15.189639 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86214365-1f4f-41a0-b8be-e2e11144241e-config-out\" (UniqueName: \"kubernetes.io/empty-dir/86214365-1f4f-41a0-b8be-e2e11144241e-config-out\") pod \"86214365-1f4f-41a0-b8be-e2e11144241e\" (UID: \"86214365-1f4f-41a0-b8be-e2e11144241e\") " Oct 29 22:35:15.189719 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:15.189692 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/86214365-1f4f-41a0-b8be-e2e11144241e-secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/86214365-1f4f-41a0-b8be-e2e11144241e-secret-alertmanager-kube-rbac-proxy-web\") pod \"86214365-1f4f-41a0-b8be-e2e11144241e\" (UID: \"86214365-1f4f-41a0-b8be-e2e11144241e\") " Oct 29 22:35:15.189782 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:15.189684 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86214365-1f4f-41a0-b8be-e2e11144241e-alertmanager-main-db" pod "86214365-1f4f-41a0-b8be-e2e11144241e" (UID: "86214365-1f4f-41a0-b8be-e2e11144241e"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:35:15.189782 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:15.189742 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86214365-1f4f-41a0-b8be-e2e11144241e-tls-assets\" (UniqueName: \"kubernetes.io/projected/86214365-1f4f-41a0-b8be-e2e11144241e-tls-assets\") pod \"86214365-1f4f-41a0-b8be-e2e11144241e\" (UID: \"86214365-1f4f-41a0-b8be-e2e11144241e\") " Oct 29 22:35:15.189883 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:15.189786 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/86214365-1f4f-41a0-b8be-e2e11144241e-secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/86214365-1f4f-41a0-b8be-e2e11144241e-secret-alertmanager-main-tls\") pod \"86214365-1f4f-41a0-b8be-e2e11144241e\" (UID: \"86214365-1f4f-41a0-b8be-e2e11144241e\") " Oct 29 22:35:15.189883 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:15.189824 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/86214365-1f4f-41a0-b8be-e2e11144241e-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/86214365-1f4f-41a0-b8be-e2e11144241e-metrics-client-ca\") pod \"86214365-1f4f-41a0-b8be-e2e11144241e\" (UID: \"86214365-1f4f-41a0-b8be-e2e11144241e\") " Oct 29 22:35:15.189883 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:15.189866 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/86214365-1f4f-41a0-b8be-e2e11144241e-secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/86214365-1f4f-41a0-b8be-e2e11144241e-secret-alertmanager-kube-rbac-proxy-metric\") pod \"86214365-1f4f-41a0-b8be-e2e11144241e\" (UID: \"86214365-1f4f-41a0-b8be-e2e11144241e\") " Oct 29 22:35:15.190037 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:15.189881 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86214365-1f4f-41a0-b8be-e2e11144241e-alertmanager-trusted-ca-bundle" pod "86214365-1f4f-41a0-b8be-e2e11144241e" (UID: "86214365-1f4f-41a0-b8be-e2e11144241e"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:35:15.190250 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:15.190227 5057 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/86214365-1f4f-41a0-b8be-e2e11144241e-alertmanager-main-db\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 22:35:15.190373 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:15.190256 5057 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/86214365-1f4f-41a0-b8be-e2e11144241e-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 22:35:15.190792 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:15.190763 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86214365-1f4f-41a0-b8be-e2e11144241e-metrics-client-ca" pod "86214365-1f4f-41a0-b8be-e2e11144241e" (UID: "86214365-1f4f-41a0-b8be-e2e11144241e"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:35:15.192220 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:15.192196 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86214365-1f4f-41a0-b8be-e2e11144241e-secret-alertmanager-kube-rbac-proxy" pod "86214365-1f4f-41a0-b8be-e2e11144241e" (UID: "86214365-1f4f-41a0-b8be-e2e11144241e"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:35:15.192544 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:15.192519 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86214365-1f4f-41a0-b8be-e2e11144241e-kube-api-access-z5md5" pod "86214365-1f4f-41a0-b8be-e2e11144241e" (UID: "86214365-1f4f-41a0-b8be-e2e11144241e"). InnerVolumeSpecName "kube-api-access-z5md5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:35:15.193103 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:15.192942 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86214365-1f4f-41a0-b8be-e2e11144241e-config-volume" pod "86214365-1f4f-41a0-b8be-e2e11144241e" (UID: "86214365-1f4f-41a0-b8be-e2e11144241e"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:35:15.193103 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:15.192965 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86214365-1f4f-41a0-b8be-e2e11144241e-secret-alertmanager-kube-rbac-proxy-metric" pod "86214365-1f4f-41a0-b8be-e2e11144241e" (UID: "86214365-1f4f-41a0-b8be-e2e11144241e"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:35:15.193103 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:15.193066 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86214365-1f4f-41a0-b8be-e2e11144241e-tls-assets" pod "86214365-1f4f-41a0-b8be-e2e11144241e" (UID: "86214365-1f4f-41a0-b8be-e2e11144241e"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:35:15.193307 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:15.193168 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86214365-1f4f-41a0-b8be-e2e11144241e-config-out" pod "86214365-1f4f-41a0-b8be-e2e11144241e" (UID: "86214365-1f4f-41a0-b8be-e2e11144241e"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:35:15.193481 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:15.193462 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86214365-1f4f-41a0-b8be-e2e11144241e-secret-alertmanager-main-tls" pod "86214365-1f4f-41a0-b8be-e2e11144241e" (UID: "86214365-1f4f-41a0-b8be-e2e11144241e"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:35:15.194083 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:15.194059 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86214365-1f4f-41a0-b8be-e2e11144241e-secret-alertmanager-kube-rbac-proxy-web" pod "86214365-1f4f-41a0-b8be-e2e11144241e" (UID: "86214365-1f4f-41a0-b8be-e2e11144241e"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:35:15.197479 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:15.197459 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86214365-1f4f-41a0-b8be-e2e11144241e-cluster-tls-config" pod "86214365-1f4f-41a0-b8be-e2e11144241e" (UID: "86214365-1f4f-41a0-b8be-e2e11144241e"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:35:15.205116 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:15.205094 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86214365-1f4f-41a0-b8be-e2e11144241e-web-config" pod "86214365-1f4f-41a0-b8be-e2e11144241e" (UID: "86214365-1f4f-41a0-b8be-e2e11144241e"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:35:15.289501 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:15.289363 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-ktp5j-2qdc5"] Oct 29 22:35:15.291934 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:15.291907 5057 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/86214365-1f4f-41a0-b8be-e2e11144241e-config-out\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 22:35:15.292057 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:15.291940 5057 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/86214365-1f4f-41a0-b8be-e2e11144241e-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 22:35:15.292057 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:15.291958 5057 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/86214365-1f4f-41a0-b8be-e2e11144241e-tls-assets\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 22:35:15.292057 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:15.291973 5057 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/86214365-1f4f-41a0-b8be-e2e11144241e-secret-alertmanager-main-tls\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 22:35:15.292057 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:15.291987 5057 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/86214365-1f4f-41a0-b8be-e2e11144241e-metrics-client-ca\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 22:35:15.292057 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:15.292004 5057 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/86214365-1f4f-41a0-b8be-e2e11144241e-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 22:35:15.292057 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:15.292018 5057 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/86214365-1f4f-41a0-b8be-e2e11144241e-web-config\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 22:35:15.292057 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:15.292033 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z5md5\" (UniqueName: \"kubernetes.io/projected/86214365-1f4f-41a0-b8be-e2e11144241e-kube-api-access-z5md5\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 22:35:15.292057 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:15.292048 5057 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/86214365-1f4f-41a0-b8be-e2e11144241e-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 22:35:15.292506 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:15.292063 5057 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/86214365-1f4f-41a0-b8be-e2e11144241e-cluster-tls-config\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 22:35:15.292506 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:15.292076 5057 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/86214365-1f4f-41a0-b8be-e2e11144241e-config-volume\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 22:35:15.293111 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:35:15.293085 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84569698_cff8_449e_9acc_2c8408581d52.slice/crio-8bf2b9d5a0188dc1cfc3f93bf266d881b6b2d307616bb6a6c859188aaf9622e1 WatchSource:0}: Error finding container 8bf2b9d5a0188dc1cfc3f93bf266d881b6b2d307616bb6a6c859188aaf9622e1: Status 404 returned error can't find the container with id 8bf2b9d5a0188dc1cfc3f93bf266d881b6b2d307616bb6a6c859188aaf9622e1 Oct 29 22:35:16.075428 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:16.075341 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"86214365-1f4f-41a0-b8be-e2e11144241e","Type":"ContainerDied","Data":"d46b663b0acff115f38d66009fcba16a014e7b43c2eda5a591c11551baa8b2b8"} Oct 29 22:35:16.075428 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:16.075386 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:35:16.075904 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:16.075455 5057 scope.go:122] "RemoveContainer" containerID="f4bb10070eb88aedd6ba257addc8ffe8589b7b09b3e81a6a05690bc09559f042" Oct 29 22:35:16.077010 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:16.076983 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-ktp5j-2qdc5" event={"ID":"84569698-cff8-449e-9acc-2c8408581d52","Type":"ContainerStarted","Data":"e0eaf237b3d7efe83083caa49c60993203bbbeeb68c952a53b13ecc5f457c211"} Oct 29 22:35:16.077125 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:16.077030 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-ktp5j-2qdc5" event={"ID":"84569698-cff8-449e-9acc-2c8408581d52","Type":"ContainerStarted","Data":"8bf2b9d5a0188dc1cfc3f93bf266d881b6b2d307616bb6a6c859188aaf9622e1"} Oct 29 22:35:16.083432 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:16.083411 5057 scope.go:122] "RemoveContainer" containerID="513b36d25a3af6474b29848cb8dca81da4c37d4dbae3fcc414e7665eb7aa181d" Oct 29 22:35:16.090843 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:16.090829 5057 scope.go:122] "RemoveContainer" containerID="750111f4ae99d3b9585092893f4af8b45fec6791220c9a472bc861dba69a4888" Oct 29 22:35:16.098027 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:16.098007 5057 scope.go:122] "RemoveContainer" containerID="4662b0a105ae1bd58a9fbef8995c2b945b6a0965208374d64f6c55368f5c9e8f" Oct 29 22:35:16.105035 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:16.105016 5057 scope.go:122] "RemoveContainer" containerID="533c17e22fcf31771228a2991b31d812919dfdb55880fbd7188a5dfcdfb63e89" Oct 29 22:35:16.112094 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:16.112077 5057 scope.go:122] "RemoveContainer" containerID="b1bb7d0f6b64084459c8859945660d9620d9ee72b4bb1d6e71bd3fc7b4632a65" Oct 29 22:35:16.119288 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:16.119272 5057 scope.go:122] "RemoveContainer" containerID="d7d287f36ab310608630b0f0b92d678cf2202e4b854dbd78e653bf0ad38b195a" Oct 29 22:35:16.134569 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:16.134541 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 29 22:35:16.144455 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:16.144435 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 29 22:35:16.168039 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:16.168018 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 29 22:35:16.168591 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:16.168574 5057 cpu_manager.go:414] "containerMap: removing container" podUID="86214365-1f4f-41a0-b8be-e2e11144241e" containerName="init-config-reloader" Oct 29 22:35:16.168591 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:16.168591 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86214365-1f4f-41a0-b8be-e2e11144241e" containerName="init-config-reloader" Oct 29 22:35:16.168701 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:16.168608 5057 cpu_manager.go:414] "containerMap: removing container" podUID="86214365-1f4f-41a0-b8be-e2e11144241e" containerName="kube-rbac-proxy-web" Oct 29 22:35:16.168701 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:16.168613 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86214365-1f4f-41a0-b8be-e2e11144241e" containerName="kube-rbac-proxy-web" Oct 29 22:35:16.168701 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:16.168622 5057 cpu_manager.go:414] "containerMap: removing container" podUID="86214365-1f4f-41a0-b8be-e2e11144241e" containerName="kube-rbac-proxy" Oct 29 22:35:16.168701 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:16.168628 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86214365-1f4f-41a0-b8be-e2e11144241e" containerName="kube-rbac-proxy" Oct 29 22:35:16.168701 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:16.168652 5057 cpu_manager.go:414] "containerMap: removing container" podUID="86214365-1f4f-41a0-b8be-e2e11144241e" containerName="alertmanager" Oct 29 22:35:16.168701 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:16.168658 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86214365-1f4f-41a0-b8be-e2e11144241e" containerName="alertmanager" Oct 29 22:35:16.168701 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:16.168666 5057 cpu_manager.go:414] "containerMap: removing container" podUID="86214365-1f4f-41a0-b8be-e2e11144241e" containerName="config-reloader" Oct 29 22:35:16.168701 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:16.168671 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86214365-1f4f-41a0-b8be-e2e11144241e" containerName="config-reloader" Oct 29 22:35:16.168701 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:16.168677 5057 cpu_manager.go:414] "containerMap: removing container" podUID="86214365-1f4f-41a0-b8be-e2e11144241e" containerName="kube-rbac-proxy-metric" Oct 29 22:35:16.168701 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:16.168682 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86214365-1f4f-41a0-b8be-e2e11144241e" containerName="kube-rbac-proxy-metric" Oct 29 22:35:16.168701 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:16.168694 5057 cpu_manager.go:414] "containerMap: removing container" podUID="86214365-1f4f-41a0-b8be-e2e11144241e" containerName="prom-label-proxy" Oct 29 22:35:16.168701 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:16.168699 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86214365-1f4f-41a0-b8be-e2e11144241e" containerName="prom-label-proxy" Oct 29 22:35:16.169037 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:16.168764 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="86214365-1f4f-41a0-b8be-e2e11144241e" containerName="kube-rbac-proxy-web" Oct 29 22:35:16.169037 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:16.168772 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="86214365-1f4f-41a0-b8be-e2e11144241e" containerName="kube-rbac-proxy" Oct 29 22:35:16.169037 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:16.168780 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="86214365-1f4f-41a0-b8be-e2e11144241e" containerName="alertmanager" Oct 29 22:35:16.169037 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:16.168787 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="86214365-1f4f-41a0-b8be-e2e11144241e" containerName="config-reloader" Oct 29 22:35:16.169037 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:16.168792 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="86214365-1f4f-41a0-b8be-e2e11144241e" containerName="kube-rbac-proxy-metric" Oct 29 22:35:16.169037 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:16.168799 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="86214365-1f4f-41a0-b8be-e2e11144241e" containerName="prom-label-proxy" Oct 29 22:35:16.172485 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:16.172464 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:35:16.173879 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:16.173838 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-ktp5j-2qdc5" podStartSLOduration=2.173827086 podStartE2EDuration="2.173827086s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:35:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:35:16.172208959 +0000 UTC m=+96.955422386" watchObservedRunningTime="2026-10-29 22:35:16.173827086 +0000 UTC m=+96.957040512" Oct 29 22:35:16.185815 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:16.185776 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Oct 29 22:35:16.185815 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:16.185798 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Oct 29 22:35:16.185815 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:16.185801 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Oct 29 22:35:16.185815 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:16.185842 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-stxjf\"" Oct 29 22:35:16.186255 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:16.185845 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Oct 29 22:35:16.186255 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:16.185845 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Oct 29 22:35:16.186255 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:16.185870 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Oct 29 22:35:16.186255 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:16.185877 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Oct 29 22:35:16.186255 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:16.185878 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Oct 29 22:35:16.196484 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:16.196466 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Oct 29 22:35:16.199374 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:16.199347 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f2fed222-16ad-4e9c-99c4-23f64db91c21-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"f2fed222-16ad-4e9c-99c4-23f64db91c21\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:35:16.199482 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:16.199378 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 29 22:35:16.199482 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:16.199401 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f2fed222-16ad-4e9c-99c4-23f64db91c21-config-out\") pod \"alertmanager-main-0\" (UID: \"f2fed222-16ad-4e9c-99c4-23f64db91c21\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:35:16.199482 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:16.199436 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f2fed222-16ad-4e9c-99c4-23f64db91c21-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"f2fed222-16ad-4e9c-99c4-23f64db91c21\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:35:16.199619 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:16.199536 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f2fed222-16ad-4e9c-99c4-23f64db91c21-web-config\") pod \"alertmanager-main-0\" (UID: \"f2fed222-16ad-4e9c-99c4-23f64db91c21\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:35:16.199619 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:16.199579 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f2fed222-16ad-4e9c-99c4-23f64db91c21-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"f2fed222-16ad-4e9c-99c4-23f64db91c21\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:35:16.199619 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:16.199604 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f2fed222-16ad-4e9c-99c4-23f64db91c21-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"f2fed222-16ad-4e9c-99c4-23f64db91c21\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:35:16.199730 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:16.199712 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f2fed222-16ad-4e9c-99c4-23f64db91c21-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"f2fed222-16ad-4e9c-99c4-23f64db91c21\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:35:16.199775 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:16.199758 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f2fed222-16ad-4e9c-99c4-23f64db91c21-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"f2fed222-16ad-4e9c-99c4-23f64db91c21\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:35:16.199837 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:16.199825 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-th292\" (UniqueName: \"kubernetes.io/projected/f2fed222-16ad-4e9c-99c4-23f64db91c21-kube-api-access-th292\") pod \"alertmanager-main-0\" (UID: \"f2fed222-16ad-4e9c-99c4-23f64db91c21\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:35:16.199886 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:16.199869 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f2fed222-16ad-4e9c-99c4-23f64db91c21-config-volume\") pod \"alertmanager-main-0\" (UID: \"f2fed222-16ad-4e9c-99c4-23f64db91c21\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:35:16.199937 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:16.199885 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f2fed222-16ad-4e9c-99c4-23f64db91c21-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"f2fed222-16ad-4e9c-99c4-23f64db91c21\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:35:16.200028 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:16.200010 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f2fed222-16ad-4e9c-99c4-23f64db91c21-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"f2fed222-16ad-4e9c-99c4-23f64db91c21\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:35:16.200086 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:16.200042 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f2fed222-16ad-4e9c-99c4-23f64db91c21-tls-assets\") pod \"alertmanager-main-0\" (UID: \"f2fed222-16ad-4e9c-99c4-23f64db91c21\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:35:16.301546 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:16.301506 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f2fed222-16ad-4e9c-99c4-23f64db91c21-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"f2fed222-16ad-4e9c-99c4-23f64db91c21\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:35:16.301707 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:16.301562 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f2fed222-16ad-4e9c-99c4-23f64db91c21-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"f2fed222-16ad-4e9c-99c4-23f64db91c21\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:35:16.301707 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:16.301588 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-th292\" (UniqueName: \"kubernetes.io/projected/f2fed222-16ad-4e9c-99c4-23f64db91c21-kube-api-access-th292\") pod \"alertmanager-main-0\" (UID: \"f2fed222-16ad-4e9c-99c4-23f64db91c21\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:35:16.301707 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:16.301635 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f2fed222-16ad-4e9c-99c4-23f64db91c21-config-volume\") pod \"alertmanager-main-0\" (UID: \"f2fed222-16ad-4e9c-99c4-23f64db91c21\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:35:16.301707 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:16.301685 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f2fed222-16ad-4e9c-99c4-23f64db91c21-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"f2fed222-16ad-4e9c-99c4-23f64db91c21\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:35:16.301906 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:16.301728 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f2fed222-16ad-4e9c-99c4-23f64db91c21-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"f2fed222-16ad-4e9c-99c4-23f64db91c21\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:35:16.301906 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:16.301756 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f2fed222-16ad-4e9c-99c4-23f64db91c21-tls-assets\") pod \"alertmanager-main-0\" (UID: \"f2fed222-16ad-4e9c-99c4-23f64db91c21\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:35:16.301906 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:16.301801 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f2fed222-16ad-4e9c-99c4-23f64db91c21-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"f2fed222-16ad-4e9c-99c4-23f64db91c21\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:35:16.301906 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:16.301844 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f2fed222-16ad-4e9c-99c4-23f64db91c21-config-out\") pod \"alertmanager-main-0\" (UID: \"f2fed222-16ad-4e9c-99c4-23f64db91c21\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:35:16.301906 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:16.301881 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f2fed222-16ad-4e9c-99c4-23f64db91c21-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"f2fed222-16ad-4e9c-99c4-23f64db91c21\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:35:16.302163 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:16.301917 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f2fed222-16ad-4e9c-99c4-23f64db91c21-web-config\") pod \"alertmanager-main-0\" (UID: \"f2fed222-16ad-4e9c-99c4-23f64db91c21\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:35:16.302163 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:16.301966 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f2fed222-16ad-4e9c-99c4-23f64db91c21-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"f2fed222-16ad-4e9c-99c4-23f64db91c21\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:35:16.302163 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:16.301993 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f2fed222-16ad-4e9c-99c4-23f64db91c21-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"f2fed222-16ad-4e9c-99c4-23f64db91c21\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:35:16.302707 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:16.302678 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f2fed222-16ad-4e9c-99c4-23f64db91c21-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"f2fed222-16ad-4e9c-99c4-23f64db91c21\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:35:16.302824 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:16.302730 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f2fed222-16ad-4e9c-99c4-23f64db91c21-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"f2fed222-16ad-4e9c-99c4-23f64db91c21\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:35:16.304673 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:16.304646 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f2fed222-16ad-4e9c-99c4-23f64db91c21-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"f2fed222-16ad-4e9c-99c4-23f64db91c21\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:35:16.304778 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:16.304646 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f2fed222-16ad-4e9c-99c4-23f64db91c21-config-volume\") pod \"alertmanager-main-0\" (UID: \"f2fed222-16ad-4e9c-99c4-23f64db91c21\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:35:16.304974 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:16.304946 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f2fed222-16ad-4e9c-99c4-23f64db91c21-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"f2fed222-16ad-4e9c-99c4-23f64db91c21\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:35:16.305046 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:16.305024 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f2fed222-16ad-4e9c-99c4-23f64db91c21-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"f2fed222-16ad-4e9c-99c4-23f64db91c21\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:35:16.305298 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:16.305277 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f2fed222-16ad-4e9c-99c4-23f64db91c21-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"f2fed222-16ad-4e9c-99c4-23f64db91c21\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:35:16.305428 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:16.305348 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f2fed222-16ad-4e9c-99c4-23f64db91c21-tls-assets\") pod \"alertmanager-main-0\" (UID: \"f2fed222-16ad-4e9c-99c4-23f64db91c21\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:35:16.305511 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:16.305441 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f2fed222-16ad-4e9c-99c4-23f64db91c21-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"f2fed222-16ad-4e9c-99c4-23f64db91c21\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:35:16.305733 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:16.305713 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f2fed222-16ad-4e9c-99c4-23f64db91c21-web-config\") pod \"alertmanager-main-0\" (UID: \"f2fed222-16ad-4e9c-99c4-23f64db91c21\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:35:16.305813 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:16.305792 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f2fed222-16ad-4e9c-99c4-23f64db91c21-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"f2fed222-16ad-4e9c-99c4-23f64db91c21\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:35:16.306621 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:16.306599 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f2fed222-16ad-4e9c-99c4-23f64db91c21-config-out\") pod \"alertmanager-main-0\" (UID: \"f2fed222-16ad-4e9c-99c4-23f64db91c21\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:35:16.322777 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:16.322755 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-th292\" (UniqueName: \"kubernetes.io/projected/f2fed222-16ad-4e9c-99c4-23f64db91c21-kube-api-access-th292\") pod \"alertmanager-main-0\" (UID: \"f2fed222-16ad-4e9c-99c4-23f64db91c21\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:35:16.490875 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:16.490847 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:35:16.820637 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:16.820566 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 29 22:35:16.835068 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:35:16.834846 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2fed222_16ad_4e9c_99c4_23f64db91c21.slice/crio-239191e5e6bcd01b3ee600135351c0619536b283d5b30541404a1f678775b1ed WatchSource:0}: Error finding container 239191e5e6bcd01b3ee600135351c0619536b283d5b30541404a1f678775b1ed: Status 404 returned error can't find the container with id 239191e5e6bcd01b3ee600135351c0619536b283d5b30541404a1f678775b1ed Oct 29 22:35:17.085092 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:17.085001 5057 generic.go:356] "Generic (PLEG): container finished" podID="f2fed222-16ad-4e9c-99c4-23f64db91c21" containerID="65c7cff62eb92f29fd9190edde70bcb8bcc81ccc30cf07085b65ecd1cc23a2eb" exitCode=0 Oct 29 22:35:17.085569 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:17.085086 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f2fed222-16ad-4e9c-99c4-23f64db91c21","Type":"ContainerDied","Data":"65c7cff62eb92f29fd9190edde70bcb8bcc81ccc30cf07085b65ecd1cc23a2eb"} Oct 29 22:35:17.085569 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:17.085170 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f2fed222-16ad-4e9c-99c4-23f64db91c21","Type":"ContainerStarted","Data":"239191e5e6bcd01b3ee600135351c0619536b283d5b30541404a1f678775b1ed"} Oct 29 22:35:17.327771 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:17.327741 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86214365-1f4f-41a0-b8be-e2e11144241e" path="/var/lib/kubelet/pods/86214365-1f4f-41a0-b8be-e2e11144241e/volumes" Oct 29 22:35:17.812762 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:17.812726 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-559f47964f-9wc6l"] Oct 29 22:35:17.815902 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:17.815878 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-559f47964f-9wc6l" Oct 29 22:35:17.828715 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:17.828693 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Oct 29 22:35:17.828999 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:17.828970 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Oct 29 22:35:17.829121 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:17.829050 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Oct 29 22:35:17.829121 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:17.829108 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Oct 29 22:35:17.829282 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:17.829160 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-td7ck\"" Oct 29 22:35:17.841633 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:17.841610 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Oct 29 22:35:17.851608 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:17.851587 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\"" Oct 29 22:35:17.855440 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:17.855413 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-559f47964f-9wc6l"] Oct 29 22:35:17.920349 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:17.920291 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvdbf\" (UniqueName: \"kubernetes.io/projected/47845ebc-dce7-4409-9fc9-ffe9ce036c77-kube-api-access-bvdbf\") pod \"telemeter-client-559f47964f-9wc6l\" (UID: \"47845ebc-dce7-4409-9fc9-ffe9ce036c77\") " pod="openshift-monitoring/telemeter-client-559f47964f-9wc6l" Oct 29 22:35:17.920496 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:17.920353 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/47845ebc-dce7-4409-9fc9-ffe9ce036c77-serving-certs-ca-bundle\") pod \"telemeter-client-559f47964f-9wc6l\" (UID: \"47845ebc-dce7-4409-9fc9-ffe9ce036c77\") " pod="openshift-monitoring/telemeter-client-559f47964f-9wc6l" Oct 29 22:35:17.920496 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:17.920419 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/47845ebc-dce7-4409-9fc9-ffe9ce036c77-secret-telemeter-client\") pod \"telemeter-client-559f47964f-9wc6l\" (UID: \"47845ebc-dce7-4409-9fc9-ffe9ce036c77\") " pod="openshift-monitoring/telemeter-client-559f47964f-9wc6l" Oct 29 22:35:17.920496 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:17.920451 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/47845ebc-dce7-4409-9fc9-ffe9ce036c77-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-559f47964f-9wc6l\" (UID: \"47845ebc-dce7-4409-9fc9-ffe9ce036c77\") " pod="openshift-monitoring/telemeter-client-559f47964f-9wc6l" Oct 29 22:35:17.920496 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:17.920493 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/47845ebc-dce7-4409-9fc9-ffe9ce036c77-metrics-client-ca\") pod \"telemeter-client-559f47964f-9wc6l\" (UID: \"47845ebc-dce7-4409-9fc9-ffe9ce036c77\") " pod="openshift-monitoring/telemeter-client-559f47964f-9wc6l" Oct 29 22:35:17.920627 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:17.920519 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/47845ebc-dce7-4409-9fc9-ffe9ce036c77-telemeter-client-tls\") pod \"telemeter-client-559f47964f-9wc6l\" (UID: \"47845ebc-dce7-4409-9fc9-ffe9ce036c77\") " pod="openshift-monitoring/telemeter-client-559f47964f-9wc6l" Oct 29 22:35:17.920627 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:17.920541 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/47845ebc-dce7-4409-9fc9-ffe9ce036c77-federate-client-tls\") pod \"telemeter-client-559f47964f-9wc6l\" (UID: \"47845ebc-dce7-4409-9fc9-ffe9ce036c77\") " pod="openshift-monitoring/telemeter-client-559f47964f-9wc6l" Oct 29 22:35:17.920694 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:17.920678 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/47845ebc-dce7-4409-9fc9-ffe9ce036c77-telemeter-trusted-ca-bundle\") pod \"telemeter-client-559f47964f-9wc6l\" (UID: \"47845ebc-dce7-4409-9fc9-ffe9ce036c77\") " pod="openshift-monitoring/telemeter-client-559f47964f-9wc6l" Oct 29 22:35:18.021340 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:18.021305 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvdbf\" (UniqueName: \"kubernetes.io/projected/47845ebc-dce7-4409-9fc9-ffe9ce036c77-kube-api-access-bvdbf\") pod \"telemeter-client-559f47964f-9wc6l\" (UID: \"47845ebc-dce7-4409-9fc9-ffe9ce036c77\") " pod="openshift-monitoring/telemeter-client-559f47964f-9wc6l" Oct 29 22:35:18.021500 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:18.021369 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/47845ebc-dce7-4409-9fc9-ffe9ce036c77-serving-certs-ca-bundle\") pod \"telemeter-client-559f47964f-9wc6l\" (UID: \"47845ebc-dce7-4409-9fc9-ffe9ce036c77\") " pod="openshift-monitoring/telemeter-client-559f47964f-9wc6l" Oct 29 22:35:18.021500 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:18.021391 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/47845ebc-dce7-4409-9fc9-ffe9ce036c77-secret-telemeter-client\") pod \"telemeter-client-559f47964f-9wc6l\" (UID: \"47845ebc-dce7-4409-9fc9-ffe9ce036c77\") " pod="openshift-monitoring/telemeter-client-559f47964f-9wc6l" Oct 29 22:35:18.021500 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:18.021421 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/47845ebc-dce7-4409-9fc9-ffe9ce036c77-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-559f47964f-9wc6l\" (UID: \"47845ebc-dce7-4409-9fc9-ffe9ce036c77\") " pod="openshift-monitoring/telemeter-client-559f47964f-9wc6l" Oct 29 22:35:18.021500 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:18.021465 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/47845ebc-dce7-4409-9fc9-ffe9ce036c77-metrics-client-ca\") pod \"telemeter-client-559f47964f-9wc6l\" (UID: \"47845ebc-dce7-4409-9fc9-ffe9ce036c77\") " pod="openshift-monitoring/telemeter-client-559f47964f-9wc6l" Oct 29 22:35:18.021696 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:18.021503 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/47845ebc-dce7-4409-9fc9-ffe9ce036c77-telemeter-client-tls\") pod \"telemeter-client-559f47964f-9wc6l\" (UID: \"47845ebc-dce7-4409-9fc9-ffe9ce036c77\") " pod="openshift-monitoring/telemeter-client-559f47964f-9wc6l" Oct 29 22:35:18.021696 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:18.021527 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/47845ebc-dce7-4409-9fc9-ffe9ce036c77-federate-client-tls\") pod \"telemeter-client-559f47964f-9wc6l\" (UID: \"47845ebc-dce7-4409-9fc9-ffe9ce036c77\") " pod="openshift-monitoring/telemeter-client-559f47964f-9wc6l" Oct 29 22:35:18.021696 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:18.021588 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/47845ebc-dce7-4409-9fc9-ffe9ce036c77-telemeter-trusted-ca-bundle\") pod \"telemeter-client-559f47964f-9wc6l\" (UID: \"47845ebc-dce7-4409-9fc9-ffe9ce036c77\") " pod="openshift-monitoring/telemeter-client-559f47964f-9wc6l" Oct 29 22:35:18.022179 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:18.022150 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/47845ebc-dce7-4409-9fc9-ffe9ce036c77-serving-certs-ca-bundle\") pod \"telemeter-client-559f47964f-9wc6l\" (UID: \"47845ebc-dce7-4409-9fc9-ffe9ce036c77\") " pod="openshift-monitoring/telemeter-client-559f47964f-9wc6l" Oct 29 22:35:18.022307 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:18.022271 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/47845ebc-dce7-4409-9fc9-ffe9ce036c77-metrics-client-ca\") pod \"telemeter-client-559f47964f-9wc6l\" (UID: \"47845ebc-dce7-4409-9fc9-ffe9ce036c77\") " pod="openshift-monitoring/telemeter-client-559f47964f-9wc6l" Oct 29 22:35:18.022639 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:18.022617 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/47845ebc-dce7-4409-9fc9-ffe9ce036c77-telemeter-trusted-ca-bundle\") pod \"telemeter-client-559f47964f-9wc6l\" (UID: \"47845ebc-dce7-4409-9fc9-ffe9ce036c77\") " pod="openshift-monitoring/telemeter-client-559f47964f-9wc6l" Oct 29 22:35:18.024004 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:18.023975 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/47845ebc-dce7-4409-9fc9-ffe9ce036c77-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-559f47964f-9wc6l\" (UID: \"47845ebc-dce7-4409-9fc9-ffe9ce036c77\") " pod="openshift-monitoring/telemeter-client-559f47964f-9wc6l" Oct 29 22:35:18.024110 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:18.024083 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/47845ebc-dce7-4409-9fc9-ffe9ce036c77-secret-telemeter-client\") pod \"telemeter-client-559f47964f-9wc6l\" (UID: \"47845ebc-dce7-4409-9fc9-ffe9ce036c77\") " pod="openshift-monitoring/telemeter-client-559f47964f-9wc6l" Oct 29 22:35:18.024391 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:18.024371 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/47845ebc-dce7-4409-9fc9-ffe9ce036c77-telemeter-client-tls\") pod \"telemeter-client-559f47964f-9wc6l\" (UID: \"47845ebc-dce7-4409-9fc9-ffe9ce036c77\") " pod="openshift-monitoring/telemeter-client-559f47964f-9wc6l" Oct 29 22:35:18.024452 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:18.024393 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/47845ebc-dce7-4409-9fc9-ffe9ce036c77-federate-client-tls\") pod \"telemeter-client-559f47964f-9wc6l\" (UID: \"47845ebc-dce7-4409-9fc9-ffe9ce036c77\") " pod="openshift-monitoring/telemeter-client-559f47964f-9wc6l" Oct 29 22:35:18.042496 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:18.042475 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvdbf\" (UniqueName: \"kubernetes.io/projected/47845ebc-dce7-4409-9fc9-ffe9ce036c77-kube-api-access-bvdbf\") pod \"telemeter-client-559f47964f-9wc6l\" (UID: \"47845ebc-dce7-4409-9fc9-ffe9ce036c77\") " pod="openshift-monitoring/telemeter-client-559f47964f-9wc6l" Oct 29 22:35:18.094187 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:18.094112 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f2fed222-16ad-4e9c-99c4-23f64db91c21","Type":"ContainerStarted","Data":"1b6436f449a030de719afbd1c9a78b1310117f1e9a5f26931691b10a4c749d56"} Oct 29 22:35:18.094607 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:18.094203 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f2fed222-16ad-4e9c-99c4-23f64db91c21","Type":"ContainerStarted","Data":"610efde425d06ded1f51ff75bdf832ac35d23276b3d55d0591fd2a5995d85e3a"} Oct 29 22:35:18.094607 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:18.094248 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f2fed222-16ad-4e9c-99c4-23f64db91c21","Type":"ContainerStarted","Data":"373cbf5efb26a4c0beec1eac300c329e5fcf76fc5cd727ce635b88c9f6725f4e"} Oct 29 22:35:18.094607 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:18.094295 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f2fed222-16ad-4e9c-99c4-23f64db91c21","Type":"ContainerStarted","Data":"1d61d7644b0db90ace7417551daa0df8b8ed34e67a587f6f5f0535c083fa79a8"} Oct 29 22:35:18.094607 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:18.094362 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f2fed222-16ad-4e9c-99c4-23f64db91c21","Type":"ContainerStarted","Data":"502d9d7ff48b35ed2d1cf862abaaa954681534b80f43d3e5ba74cf96de0868dd"} Oct 29 22:35:18.094607 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:18.094406 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f2fed222-16ad-4e9c-99c4-23f64db91c21","Type":"ContainerStarted","Data":"270b46743c146f6eeb000d924be664fc6802e701c40d09676658d59c8a0a5a22"} Oct 29 22:35:18.139018 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:18.138863 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-559f47964f-9wc6l" Oct 29 22:35:18.147130 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:18.147077 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.135221524 podStartE2EDuration="2.1470591s" totalImagesPullingTime="0s" totalInitContainerRuntime="11.837576ms" isStatefulPod=false podCreationTimestamp="2026-10-29 22:35:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:35:18.146655039 +0000 UTC m=+98.929868465" watchObservedRunningTime="2026-10-29 22:35:18.1470591 +0000 UTC m=+98.930272527" Oct 29 22:35:18.485114 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:18.483112 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-559f47964f-9wc6l"] Oct 29 22:35:18.490369 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:35:18.487494 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47845ebc_dce7_4409_9fc9_ffe9ce036c77.slice/crio-e9ed7e320d4b94aab41bd91d497adda0142cc50920cbf84e750b7cebf64f9b3d WatchSource:0}: Error finding container e9ed7e320d4b94aab41bd91d497adda0142cc50920cbf84e750b7cebf64f9b3d: Status 404 returned error can't find the container with id e9ed7e320d4b94aab41bd91d497adda0142cc50920cbf84e750b7cebf64f9b3d Oct 29 22:35:18.500161 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:18.500134 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 29 22:35:18.500607 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:18.500429 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="af0e8682-1347-4724-b0a4-870a0130fd00" containerName="prometheus" containerID="cri-o://abb69760e74c80520fc09f6567ad98b02045f5b8b32abd44e52101dadaf64302" gracePeriod=600 Oct 29 22:35:18.500607 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:18.500471 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="af0e8682-1347-4724-b0a4-870a0130fd00" containerName="kube-rbac-proxy-web" containerID="cri-o://38ebad248d161a5aea1cf5d3edb12d111de9a9851193124cb5419c726596116b" gracePeriod=600 Oct 29 22:35:18.500607 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:18.500481 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="af0e8682-1347-4724-b0a4-870a0130fd00" containerName="thanos-sidecar" containerID="cri-o://877ec5d73ce266cb62cf4b6dfa9b9f2fb65b8aca6280dc8625734b5f6a362484" gracePeriod=600 Oct 29 22:35:18.500607 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:18.500429 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="af0e8682-1347-4724-b0a4-870a0130fd00" containerName="kube-rbac-proxy" containerID="cri-o://4dc31581b1d0ee53683e89f0f03e48a74495c944fa7696dc878c09e7a29274a8" gracePeriod=600 Oct 29 22:35:18.500607 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:18.500528 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="af0e8682-1347-4724-b0a4-870a0130fd00" containerName="config-reloader" containerID="cri-o://95113b796d327d72ba495de8bfa833fa28add556419768c49a579e4f5507e9fc" gracePeriod=600 Oct 29 22:35:18.500607 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:18.500495 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="af0e8682-1347-4724-b0a4-870a0130fd00" containerName="kube-rbac-proxy-thanos" containerID="cri-o://534a35847e5a1b683b1bcf4e588b625e3199cc9ab9d8f671d3bca0c479a18d7c" gracePeriod=600 Oct 29 22:35:18.669821 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:18.669790 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:35:19.036766 ip-10-0-140-74 kubenswrapper[5057]: E1029 22:35:19.036726 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d2dnn" containerName="extractor" Oct 29 22:35:19.103882 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:19.103793 5057 generic.go:356] "Generic (PLEG): container finished" podID="af0e8682-1347-4724-b0a4-870a0130fd00" containerID="534a35847e5a1b683b1bcf4e588b625e3199cc9ab9d8f671d3bca0c479a18d7c" exitCode=0 Oct 29 22:35:19.103882 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:19.103828 5057 generic.go:356] "Generic (PLEG): container finished" podID="af0e8682-1347-4724-b0a4-870a0130fd00" containerID="4dc31581b1d0ee53683e89f0f03e48a74495c944fa7696dc878c09e7a29274a8" exitCode=0 Oct 29 22:35:19.103882 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:19.103836 5057 generic.go:356] "Generic (PLEG): container finished" podID="af0e8682-1347-4724-b0a4-870a0130fd00" containerID="877ec5d73ce266cb62cf4b6dfa9b9f2fb65b8aca6280dc8625734b5f6a362484" exitCode=0 Oct 29 22:35:19.103882 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:19.103842 5057 generic.go:356] "Generic (PLEG): container finished" podID="af0e8682-1347-4724-b0a4-870a0130fd00" containerID="95113b796d327d72ba495de8bfa833fa28add556419768c49a579e4f5507e9fc" exitCode=0 Oct 29 22:35:19.103882 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:19.103850 5057 generic.go:356] "Generic (PLEG): container finished" podID="af0e8682-1347-4724-b0a4-870a0130fd00" containerID="abb69760e74c80520fc09f6567ad98b02045f5b8b32abd44e52101dadaf64302" exitCode=0 Oct 29 22:35:19.104429 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:19.103918 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"af0e8682-1347-4724-b0a4-870a0130fd00","Type":"ContainerDied","Data":"534a35847e5a1b683b1bcf4e588b625e3199cc9ab9d8f671d3bca0c479a18d7c"} Oct 29 22:35:19.104429 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:19.103990 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"af0e8682-1347-4724-b0a4-870a0130fd00","Type":"ContainerDied","Data":"4dc31581b1d0ee53683e89f0f03e48a74495c944fa7696dc878c09e7a29274a8"} Oct 29 22:35:19.104429 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:19.104018 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"af0e8682-1347-4724-b0a4-870a0130fd00","Type":"ContainerDied","Data":"877ec5d73ce266cb62cf4b6dfa9b9f2fb65b8aca6280dc8625734b5f6a362484"} Oct 29 22:35:19.104429 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:19.104042 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"af0e8682-1347-4724-b0a4-870a0130fd00","Type":"ContainerDied","Data":"95113b796d327d72ba495de8bfa833fa28add556419768c49a579e4f5507e9fc"} Oct 29 22:35:19.104429 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:19.104065 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"af0e8682-1347-4724-b0a4-870a0130fd00","Type":"ContainerDied","Data":"abb69760e74c80520fc09f6567ad98b02045f5b8b32abd44e52101dadaf64302"} Oct 29 22:35:19.105736 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:19.105710 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-559f47964f-9wc6l" event={"ID":"47845ebc-dce7-4409-9fc9-ffe9ce036c77","Type":"ContainerStarted","Data":"e9ed7e320d4b94aab41bd91d497adda0142cc50920cbf84e750b7cebf64f9b3d"} Oct 29 22:35:19.783453 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:19.783429 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:35:19.838633 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:19.838559 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/af0e8682-1347-4724-b0a4-870a0130fd00-secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/af0e8682-1347-4724-b0a4-870a0130fd00-secret-prometheus-k8s-tls\") pod \"af0e8682-1347-4724-b0a4-870a0130fd00\" (UID: \"af0e8682-1347-4724-b0a4-870a0130fd00\") " Oct 29 22:35:19.838633 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:19.838602 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/af0e8682-1347-4724-b0a4-870a0130fd00-secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/af0e8682-1347-4724-b0a4-870a0130fd00-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"af0e8682-1347-4724-b0a4-870a0130fd00\" (UID: \"af0e8682-1347-4724-b0a4-870a0130fd00\") " Oct 29 22:35:19.838633 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:19.838627 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af0e8682-1347-4724-b0a4-870a0130fd00-prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/af0e8682-1347-4724-b0a4-870a0130fd00-prometheus-trusted-ca-bundle\") pod \"af0e8682-1347-4724-b0a4-870a0130fd00\" (UID: \"af0e8682-1347-4724-b0a4-870a0130fd00\") " Oct 29 22:35:19.838896 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:19.838675 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af0e8682-1347-4724-b0a4-870a0130fd00-configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/af0e8682-1347-4724-b0a4-870a0130fd00-configmap-metrics-client-ca\") pod \"af0e8682-1347-4724-b0a4-870a0130fd00\" (UID: \"af0e8682-1347-4724-b0a4-870a0130fd00\") " Oct 29 22:35:19.838896 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:19.838709 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/af0e8682-1347-4724-b0a4-870a0130fd00-secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/af0e8682-1347-4724-b0a4-870a0130fd00-secret-kube-rbac-proxy\") pod \"af0e8682-1347-4724-b0a4-870a0130fd00\" (UID: \"af0e8682-1347-4724-b0a4-870a0130fd00\") " Oct 29 22:35:19.838896 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:19.838744 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af0e8682-1347-4724-b0a4-870a0130fd00-prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/af0e8682-1347-4724-b0a4-870a0130fd00-prometheus-k8s-rulefiles-1\") pod \"af0e8682-1347-4724-b0a4-870a0130fd00\" (UID: \"af0e8682-1347-4724-b0a4-870a0130fd00\") " Oct 29 22:35:19.838896 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:19.838786 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af0e8682-1347-4724-b0a4-870a0130fd00-prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/af0e8682-1347-4724-b0a4-870a0130fd00-prometheus-k8s-rulefiles-0\") pod \"af0e8682-1347-4724-b0a4-870a0130fd00\" (UID: \"af0e8682-1347-4724-b0a4-870a0130fd00\") " Oct 29 22:35:19.838896 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:19.838814 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af0e8682-1347-4724-b0a4-870a0130fd00-configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/af0e8682-1347-4724-b0a4-870a0130fd00-configmap-serving-certs-ca-bundle\") pod \"af0e8682-1347-4724-b0a4-870a0130fd00\" (UID: \"af0e8682-1347-4724-b0a4-870a0130fd00\") " Oct 29 22:35:19.838896 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:19.838848 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af0e8682-1347-4724-b0a4-870a0130fd00-prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/af0e8682-1347-4724-b0a4-870a0130fd00-prometheus-k8s-rulefiles-2\") pod \"af0e8682-1347-4724-b0a4-870a0130fd00\" (UID: \"af0e8682-1347-4724-b0a4-870a0130fd00\") " Oct 29 22:35:19.838896 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:19.838878 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af0e8682-1347-4724-b0a4-870a0130fd00-tls-assets\" (UniqueName: \"kubernetes.io/projected/af0e8682-1347-4724-b0a4-870a0130fd00-tls-assets\") pod \"af0e8682-1347-4724-b0a4-870a0130fd00\" (UID: \"af0e8682-1347-4724-b0a4-870a0130fd00\") " Oct 29 22:35:19.839487 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:19.838905 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af0e8682-1347-4724-b0a4-870a0130fd00-config-out\" (UniqueName: \"kubernetes.io/empty-dir/af0e8682-1347-4724-b0a4-870a0130fd00-config-out\") pod \"af0e8682-1347-4724-b0a4-870a0130fd00\" (UID: \"af0e8682-1347-4724-b0a4-870a0130fd00\") " Oct 29 22:35:19.839487 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:19.838978 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af0e8682-1347-4724-b0a4-870a0130fd00-kube-api-access-g5wj4\" (UniqueName: \"kubernetes.io/projected/af0e8682-1347-4724-b0a4-870a0130fd00-kube-api-access-g5wj4\") pod \"af0e8682-1347-4724-b0a4-870a0130fd00\" (UID: \"af0e8682-1347-4724-b0a4-870a0130fd00\") " Oct 29 22:35:19.839487 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:19.839092 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af0e8682-1347-4724-b0a4-870a0130fd00-prometheus-trusted-ca-bundle" pod "af0e8682-1347-4724-b0a4-870a0130fd00" (UID: "af0e8682-1347-4724-b0a4-870a0130fd00"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:35:19.839487 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:19.839251 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af0e8682-1347-4724-b0a4-870a0130fd00-prometheus-k8s-rulefiles-1" pod "af0e8682-1347-4724-b0a4-870a0130fd00" (UID: "af0e8682-1347-4724-b0a4-870a0130fd00"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:35:19.839687 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:19.839585 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af0e8682-1347-4724-b0a4-870a0130fd00-prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/af0e8682-1347-4724-b0a4-870a0130fd00-prometheus-k8s-db\") pod \"af0e8682-1347-4724-b0a4-870a0130fd00\" (UID: \"af0e8682-1347-4724-b0a4-870a0130fd00\") " Oct 29 22:35:19.839687 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:19.839627 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/af0e8682-1347-4724-b0a4-870a0130fd00-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/af0e8682-1347-4724-b0a4-870a0130fd00-thanos-prometheus-http-client-file\") pod \"af0e8682-1347-4724-b0a4-870a0130fd00\" (UID: \"af0e8682-1347-4724-b0a4-870a0130fd00\") " Oct 29 22:35:19.839687 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:19.839658 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/af0e8682-1347-4724-b0a4-870a0130fd00-config\" (UniqueName: \"kubernetes.io/secret/af0e8682-1347-4724-b0a4-870a0130fd00-config\") pod \"af0e8682-1347-4724-b0a4-870a0130fd00\" (UID: \"af0e8682-1347-4724-b0a4-870a0130fd00\") " Oct 29 22:35:19.839840 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:19.839678 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af0e8682-1347-4724-b0a4-870a0130fd00-configmap-serving-certs-ca-bundle" pod "af0e8682-1347-4724-b0a4-870a0130fd00" (UID: "af0e8682-1347-4724-b0a4-870a0130fd00"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:35:19.839840 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:19.839711 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af0e8682-1347-4724-b0a4-870a0130fd00-configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/af0e8682-1347-4724-b0a4-870a0130fd00-configmap-kubelet-serving-ca-bundle\") pod \"af0e8682-1347-4724-b0a4-870a0130fd00\" (UID: \"af0e8682-1347-4724-b0a4-870a0130fd00\") " Oct 29 22:35:19.839840 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:19.839762 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/af0e8682-1347-4724-b0a4-870a0130fd00-web-config\" (UniqueName: \"kubernetes.io/secret/af0e8682-1347-4724-b0a4-870a0130fd00-web-config\") pod \"af0e8682-1347-4724-b0a4-870a0130fd00\" (UID: \"af0e8682-1347-4724-b0a4-870a0130fd00\") " Oct 29 22:35:19.839840 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:19.839796 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/af0e8682-1347-4724-b0a4-870a0130fd00-secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/af0e8682-1347-4724-b0a4-870a0130fd00-secret-metrics-client-certs\") pod \"af0e8682-1347-4724-b0a4-870a0130fd00\" (UID: \"af0e8682-1347-4724-b0a4-870a0130fd00\") " Oct 29 22:35:19.840029 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:19.839857 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/af0e8682-1347-4724-b0a4-870a0130fd00-secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/af0e8682-1347-4724-b0a4-870a0130fd00-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"af0e8682-1347-4724-b0a4-870a0130fd00\" (UID: \"af0e8682-1347-4724-b0a4-870a0130fd00\") " Oct 29 22:35:19.840029 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:19.839890 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/af0e8682-1347-4724-b0a4-870a0130fd00-secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/af0e8682-1347-4724-b0a4-870a0130fd00-secret-grpc-tls\") pod \"af0e8682-1347-4724-b0a4-870a0130fd00\" (UID: \"af0e8682-1347-4724-b0a4-870a0130fd00\") " Oct 29 22:35:19.841278 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:19.840219 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af0e8682-1347-4724-b0a4-870a0130fd00-prometheus-k8s-rulefiles-2" pod "af0e8682-1347-4724-b0a4-870a0130fd00" (UID: "af0e8682-1347-4724-b0a4-870a0130fd00"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:35:19.841278 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:19.840301 5057 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/af0e8682-1347-4724-b0a4-870a0130fd00-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 22:35:19.841278 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:19.840482 5057 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/af0e8682-1347-4724-b0a4-870a0130fd00-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 22:35:19.841278 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:19.840500 5057 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/af0e8682-1347-4724-b0a4-870a0130fd00-prometheus-trusted-ca-bundle\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 22:35:19.841278 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:19.840482 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af0e8682-1347-4724-b0a4-870a0130fd00-configmap-metrics-client-ca" pod "af0e8682-1347-4724-b0a4-870a0130fd00" (UID: "af0e8682-1347-4724-b0a4-870a0130fd00"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:35:19.841278 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:19.840806 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af0e8682-1347-4724-b0a4-870a0130fd00-configmap-kubelet-serving-ca-bundle" pod "af0e8682-1347-4724-b0a4-870a0130fd00" (UID: "af0e8682-1347-4724-b0a4-870a0130fd00"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:35:19.841278 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:19.841177 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af0e8682-1347-4724-b0a4-870a0130fd00-prometheus-k8s-rulefiles-0" pod "af0e8682-1347-4724-b0a4-870a0130fd00" (UID: "af0e8682-1347-4724-b0a4-870a0130fd00"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:35:19.841997 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:19.841971 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af0e8682-1347-4724-b0a4-870a0130fd00-prometheus-k8s-db" pod "af0e8682-1347-4724-b0a4-870a0130fd00" (UID: "af0e8682-1347-4724-b0a4-870a0130fd00"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:35:19.842481 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:19.842440 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af0e8682-1347-4724-b0a4-870a0130fd00-secret-prometheus-k8s-thanos-sidecar-tls" pod "af0e8682-1347-4724-b0a4-870a0130fd00" (UID: "af0e8682-1347-4724-b0a4-870a0130fd00"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:35:19.842925 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:19.842899 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af0e8682-1347-4724-b0a4-870a0130fd00-secret-prometheus-k8s-tls" pod "af0e8682-1347-4724-b0a4-870a0130fd00" (UID: "af0e8682-1347-4724-b0a4-870a0130fd00"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:35:19.843016 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:19.842967 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af0e8682-1347-4724-b0a4-870a0130fd00-kube-api-access-g5wj4" pod "af0e8682-1347-4724-b0a4-870a0130fd00" (UID: "af0e8682-1347-4724-b0a4-870a0130fd00"). InnerVolumeSpecName "kube-api-access-g5wj4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:35:19.844837 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:19.844773 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af0e8682-1347-4724-b0a4-870a0130fd00-secret-prometheus-k8s-kube-rbac-proxy-web" pod "af0e8682-1347-4724-b0a4-870a0130fd00" (UID: "af0e8682-1347-4724-b0a4-870a0130fd00"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:35:19.844933 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:19.844890 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af0e8682-1347-4724-b0a4-870a0130fd00-config-out" pod "af0e8682-1347-4724-b0a4-870a0130fd00" (UID: "af0e8682-1347-4724-b0a4-870a0130fd00"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:35:19.845128 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:19.845084 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af0e8682-1347-4724-b0a4-870a0130fd00-config" pod "af0e8682-1347-4724-b0a4-870a0130fd00" (UID: "af0e8682-1347-4724-b0a4-870a0130fd00"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:35:19.845128 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:19.845082 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af0e8682-1347-4724-b0a4-870a0130fd00-tls-assets" pod "af0e8682-1347-4724-b0a4-870a0130fd00" (UID: "af0e8682-1347-4724-b0a4-870a0130fd00"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:35:19.845263 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:19.845187 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af0e8682-1347-4724-b0a4-870a0130fd00-secret-grpc-tls" pod "af0e8682-1347-4724-b0a4-870a0130fd00" (UID: "af0e8682-1347-4724-b0a4-870a0130fd00"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:35:19.845323 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:19.845273 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af0e8682-1347-4724-b0a4-870a0130fd00-thanos-prometheus-http-client-file" pod "af0e8682-1347-4724-b0a4-870a0130fd00" (UID: "af0e8682-1347-4724-b0a4-870a0130fd00"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:35:19.845398 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:19.845372 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af0e8682-1347-4724-b0a4-870a0130fd00-secret-kube-rbac-proxy" pod "af0e8682-1347-4724-b0a4-870a0130fd00" (UID: "af0e8682-1347-4724-b0a4-870a0130fd00"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:35:19.846118 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:19.846092 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af0e8682-1347-4724-b0a4-870a0130fd00-secret-metrics-client-certs" pod "af0e8682-1347-4724-b0a4-870a0130fd00" (UID: "af0e8682-1347-4724-b0a4-870a0130fd00"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:35:19.862783 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:19.862760 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af0e8682-1347-4724-b0a4-870a0130fd00-web-config" pod "af0e8682-1347-4724-b0a4-870a0130fd00" (UID: "af0e8682-1347-4724-b0a4-870a0130fd00"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:35:19.940968 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:19.940939 5057 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/af0e8682-1347-4724-b0a4-870a0130fd00-configmap-metrics-client-ca\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 22:35:19.940968 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:19.940965 5057 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/af0e8682-1347-4724-b0a4-870a0130fd00-secret-kube-rbac-proxy\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 22:35:19.940968 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:19.940975 5057 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/af0e8682-1347-4724-b0a4-870a0130fd00-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 22:35:19.941174 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:19.940985 5057 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/af0e8682-1347-4724-b0a4-870a0130fd00-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 22:35:19.941174 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:19.940995 5057 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/af0e8682-1347-4724-b0a4-870a0130fd00-tls-assets\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 22:35:19.941174 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:19.941003 5057 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/af0e8682-1347-4724-b0a4-870a0130fd00-config-out\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 22:35:19.941174 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:19.941011 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g5wj4\" (UniqueName: \"kubernetes.io/projected/af0e8682-1347-4724-b0a4-870a0130fd00-kube-api-access-g5wj4\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 22:35:19.941174 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:19.941019 5057 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/af0e8682-1347-4724-b0a4-870a0130fd00-prometheus-k8s-db\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 22:35:19.941174 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:19.941028 5057 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/af0e8682-1347-4724-b0a4-870a0130fd00-thanos-prometheus-http-client-file\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 22:35:19.941174 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:19.941037 5057 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/af0e8682-1347-4724-b0a4-870a0130fd00-config\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 22:35:19.941174 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:19.941045 5057 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/af0e8682-1347-4724-b0a4-870a0130fd00-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 22:35:19.941174 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:19.941055 5057 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/af0e8682-1347-4724-b0a4-870a0130fd00-web-config\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 22:35:19.941174 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:19.941063 5057 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/af0e8682-1347-4724-b0a4-870a0130fd00-secret-metrics-client-certs\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 22:35:19.941174 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:19.941071 5057 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/af0e8682-1347-4724-b0a4-870a0130fd00-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 22:35:19.941174 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:19.941080 5057 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/af0e8682-1347-4724-b0a4-870a0130fd00-secret-grpc-tls\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 22:35:19.941174 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:19.941088 5057 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/af0e8682-1347-4724-b0a4-870a0130fd00-secret-prometheus-k8s-tls\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 22:35:19.941174 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:19.941097 5057 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/af0e8682-1347-4724-b0a4-870a0130fd00-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 22:35:20.116028 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:20.115989 5057 generic.go:356] "Generic (PLEG): container finished" podID="af0e8682-1347-4724-b0a4-870a0130fd00" containerID="38ebad248d161a5aea1cf5d3edb12d111de9a9851193124cb5419c726596116b" exitCode=0 Oct 29 22:35:20.116410 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:20.116110 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"af0e8682-1347-4724-b0a4-870a0130fd00","Type":"ContainerDied","Data":"38ebad248d161a5aea1cf5d3edb12d111de9a9851193124cb5419c726596116b"} Oct 29 22:35:20.116410 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:20.116182 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:35:20.116410 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:20.116259 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"af0e8682-1347-4724-b0a4-870a0130fd00","Type":"ContainerDied","Data":"def28858d816763418378a5405cc991bac32ce982d06832a69a1f71274c323ff"} Oct 29 22:35:20.116410 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:20.116319 5057 scope.go:122] "RemoveContainer" containerID="534a35847e5a1b683b1bcf4e588b625e3199cc9ab9d8f671d3bca0c479a18d7c" Oct 29 22:35:20.122515 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:20.122489 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-559f47964f-9wc6l" event={"ID":"47845ebc-dce7-4409-9fc9-ffe9ce036c77","Type":"ContainerStarted","Data":"31eb058f8d9c734985e064fee44002254987cb324e3278868a3f53b794da9f60"} Oct 29 22:35:20.122604 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:20.122536 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-559f47964f-9wc6l" event={"ID":"47845ebc-dce7-4409-9fc9-ffe9ce036c77","Type":"ContainerStarted","Data":"0beb885b56b992bdee1a28a52c430d138ab496f22c740aa5322ec7db1596ee28"} Oct 29 22:35:20.137029 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:20.137004 5057 scope.go:122] "RemoveContainer" containerID="4dc31581b1d0ee53683e89f0f03e48a74495c944fa7696dc878c09e7a29274a8" Oct 29 22:35:20.149380 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:20.149359 5057 scope.go:122] "RemoveContainer" containerID="38ebad248d161a5aea1cf5d3edb12d111de9a9851193124cb5419c726596116b" Oct 29 22:35:20.156992 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:20.156975 5057 scope.go:122] "RemoveContainer" containerID="877ec5d73ce266cb62cf4b6dfa9b9f2fb65b8aca6280dc8625734b5f6a362484" Oct 29 22:35:20.166093 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:20.166073 5057 scope.go:122] "RemoveContainer" containerID="95113b796d327d72ba495de8bfa833fa28add556419768c49a579e4f5507e9fc" Oct 29 22:35:20.168889 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:20.168870 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 29 22:35:20.174698 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:20.174663 5057 scope.go:122] "RemoveContainer" containerID="abb69760e74c80520fc09f6567ad98b02045f5b8b32abd44e52101dadaf64302" Oct 29 22:35:20.175086 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:20.175066 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 29 22:35:20.183954 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:20.183937 5057 scope.go:122] "RemoveContainer" containerID="64027ca342298dbcd74d94bc62ee02cdfdc65123eb490f95d4a53d35ec81f67b" Oct 29 22:35:20.193066 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:20.193044 5057 scope.go:122] "RemoveContainer" containerID="534a35847e5a1b683b1bcf4e588b625e3199cc9ab9d8f671d3bca0c479a18d7c" Oct 29 22:35:20.193365 ip-10-0-140-74 kubenswrapper[5057]: E1029 22:35:20.193343 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"534a35847e5a1b683b1bcf4e588b625e3199cc9ab9d8f671d3bca0c479a18d7c\": container with ID starting with 534a35847e5a1b683b1bcf4e588b625e3199cc9ab9d8f671d3bca0c479a18d7c not found: ID does not exist" containerID="534a35847e5a1b683b1bcf4e588b625e3199cc9ab9d8f671d3bca0c479a18d7c" Oct 29 22:35:20.193446 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:20.193376 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"534a35847e5a1b683b1bcf4e588b625e3199cc9ab9d8f671d3bca0c479a18d7c"} err="failed to get container status \"534a35847e5a1b683b1bcf4e588b625e3199cc9ab9d8f671d3bca0c479a18d7c\": rpc error: code = NotFound desc = could not find container \"534a35847e5a1b683b1bcf4e588b625e3199cc9ab9d8f671d3bca0c479a18d7c\": container with ID starting with 534a35847e5a1b683b1bcf4e588b625e3199cc9ab9d8f671d3bca0c479a18d7c not found: ID does not exist" Oct 29 22:35:20.193446 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:20.193394 5057 scope.go:122] "RemoveContainer" containerID="4dc31581b1d0ee53683e89f0f03e48a74495c944fa7696dc878c09e7a29274a8" Oct 29 22:35:20.193693 ip-10-0-140-74 kubenswrapper[5057]: E1029 22:35:20.193671 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4dc31581b1d0ee53683e89f0f03e48a74495c944fa7696dc878c09e7a29274a8\": container with ID starting with 4dc31581b1d0ee53683e89f0f03e48a74495c944fa7696dc878c09e7a29274a8 not found: ID does not exist" containerID="4dc31581b1d0ee53683e89f0f03e48a74495c944fa7696dc878c09e7a29274a8" Oct 29 22:35:20.193762 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:20.193707 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4dc31581b1d0ee53683e89f0f03e48a74495c944fa7696dc878c09e7a29274a8"} err="failed to get container status \"4dc31581b1d0ee53683e89f0f03e48a74495c944fa7696dc878c09e7a29274a8\": rpc error: code = NotFound desc = could not find container \"4dc31581b1d0ee53683e89f0f03e48a74495c944fa7696dc878c09e7a29274a8\": container with ID starting with 4dc31581b1d0ee53683e89f0f03e48a74495c944fa7696dc878c09e7a29274a8 not found: ID does not exist" Oct 29 22:35:20.193762 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:20.193729 5057 scope.go:122] "RemoveContainer" containerID="38ebad248d161a5aea1cf5d3edb12d111de9a9851193124cb5419c726596116b" Oct 29 22:35:20.193995 ip-10-0-140-74 kubenswrapper[5057]: E1029 22:35:20.193979 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38ebad248d161a5aea1cf5d3edb12d111de9a9851193124cb5419c726596116b\": container with ID starting with 38ebad248d161a5aea1cf5d3edb12d111de9a9851193124cb5419c726596116b not found: ID does not exist" containerID="38ebad248d161a5aea1cf5d3edb12d111de9a9851193124cb5419c726596116b" Oct 29 22:35:20.194058 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:20.194002 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38ebad248d161a5aea1cf5d3edb12d111de9a9851193124cb5419c726596116b"} err="failed to get container status \"38ebad248d161a5aea1cf5d3edb12d111de9a9851193124cb5419c726596116b\": rpc error: code = NotFound desc = could not find container \"38ebad248d161a5aea1cf5d3edb12d111de9a9851193124cb5419c726596116b\": container with ID starting with 38ebad248d161a5aea1cf5d3edb12d111de9a9851193124cb5419c726596116b not found: ID does not exist" Oct 29 22:35:20.194058 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:20.194021 5057 scope.go:122] "RemoveContainer" containerID="877ec5d73ce266cb62cf4b6dfa9b9f2fb65b8aca6280dc8625734b5f6a362484" Oct 29 22:35:20.194292 ip-10-0-140-74 kubenswrapper[5057]: E1029 22:35:20.194273 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"877ec5d73ce266cb62cf4b6dfa9b9f2fb65b8aca6280dc8625734b5f6a362484\": container with ID starting with 877ec5d73ce266cb62cf4b6dfa9b9f2fb65b8aca6280dc8625734b5f6a362484 not found: ID does not exist" containerID="877ec5d73ce266cb62cf4b6dfa9b9f2fb65b8aca6280dc8625734b5f6a362484" Oct 29 22:35:20.194364 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:20.194296 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"877ec5d73ce266cb62cf4b6dfa9b9f2fb65b8aca6280dc8625734b5f6a362484"} err="failed to get container status \"877ec5d73ce266cb62cf4b6dfa9b9f2fb65b8aca6280dc8625734b5f6a362484\": rpc error: code = NotFound desc = could not find container \"877ec5d73ce266cb62cf4b6dfa9b9f2fb65b8aca6280dc8625734b5f6a362484\": container with ID starting with 877ec5d73ce266cb62cf4b6dfa9b9f2fb65b8aca6280dc8625734b5f6a362484 not found: ID does not exist" Oct 29 22:35:20.194364 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:20.194310 5057 scope.go:122] "RemoveContainer" containerID="95113b796d327d72ba495de8bfa833fa28add556419768c49a579e4f5507e9fc" Oct 29 22:35:20.194642 ip-10-0-140-74 kubenswrapper[5057]: E1029 22:35:20.194623 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95113b796d327d72ba495de8bfa833fa28add556419768c49a579e4f5507e9fc\": container with ID starting with 95113b796d327d72ba495de8bfa833fa28add556419768c49a579e4f5507e9fc not found: ID does not exist" containerID="95113b796d327d72ba495de8bfa833fa28add556419768c49a579e4f5507e9fc" Oct 29 22:35:20.194706 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:20.194647 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95113b796d327d72ba495de8bfa833fa28add556419768c49a579e4f5507e9fc"} err="failed to get container status \"95113b796d327d72ba495de8bfa833fa28add556419768c49a579e4f5507e9fc\": rpc error: code = NotFound desc = could not find container \"95113b796d327d72ba495de8bfa833fa28add556419768c49a579e4f5507e9fc\": container with ID starting with 95113b796d327d72ba495de8bfa833fa28add556419768c49a579e4f5507e9fc not found: ID does not exist" Oct 29 22:35:20.194706 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:20.194662 5057 scope.go:122] "RemoveContainer" containerID="abb69760e74c80520fc09f6567ad98b02045f5b8b32abd44e52101dadaf64302" Oct 29 22:35:20.194980 ip-10-0-140-74 kubenswrapper[5057]: E1029 22:35:20.194959 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"abb69760e74c80520fc09f6567ad98b02045f5b8b32abd44e52101dadaf64302\": container with ID starting with abb69760e74c80520fc09f6567ad98b02045f5b8b32abd44e52101dadaf64302 not found: ID does not exist" containerID="abb69760e74c80520fc09f6567ad98b02045f5b8b32abd44e52101dadaf64302" Oct 29 22:35:20.195036 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:20.194987 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"abb69760e74c80520fc09f6567ad98b02045f5b8b32abd44e52101dadaf64302"} err="failed to get container status \"abb69760e74c80520fc09f6567ad98b02045f5b8b32abd44e52101dadaf64302\": rpc error: code = NotFound desc = could not find container \"abb69760e74c80520fc09f6567ad98b02045f5b8b32abd44e52101dadaf64302\": container with ID starting with abb69760e74c80520fc09f6567ad98b02045f5b8b32abd44e52101dadaf64302 not found: ID does not exist" Oct 29 22:35:20.195036 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:20.195009 5057 scope.go:122] "RemoveContainer" containerID="64027ca342298dbcd74d94bc62ee02cdfdc65123eb490f95d4a53d35ec81f67b" Oct 29 22:35:20.195305 ip-10-0-140-74 kubenswrapper[5057]: E1029 22:35:20.195282 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64027ca342298dbcd74d94bc62ee02cdfdc65123eb490f95d4a53d35ec81f67b\": container with ID starting with 64027ca342298dbcd74d94bc62ee02cdfdc65123eb490f95d4a53d35ec81f67b not found: ID does not exist" containerID="64027ca342298dbcd74d94bc62ee02cdfdc65123eb490f95d4a53d35ec81f67b" Oct 29 22:35:20.195374 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:20.195314 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64027ca342298dbcd74d94bc62ee02cdfdc65123eb490f95d4a53d35ec81f67b"} err="failed to get container status \"64027ca342298dbcd74d94bc62ee02cdfdc65123eb490f95d4a53d35ec81f67b\": rpc error: code = NotFound desc = could not find container \"64027ca342298dbcd74d94bc62ee02cdfdc65123eb490f95d4a53d35ec81f67b\": container with ID starting with 64027ca342298dbcd74d94bc62ee02cdfdc65123eb490f95d4a53d35ec81f67b not found: ID does not exist" Oct 29 22:35:20.205481 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:20.205460 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 29 22:35:20.205896 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:20.205877 5057 cpu_manager.go:414] "containerMap: removing container" podUID="af0e8682-1347-4724-b0a4-870a0130fd00" containerName="config-reloader" Oct 29 22:35:20.205896 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:20.205895 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af0e8682-1347-4724-b0a4-870a0130fd00" containerName="config-reloader" Oct 29 22:35:20.206043 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:20.205907 5057 cpu_manager.go:414] "containerMap: removing container" podUID="af0e8682-1347-4724-b0a4-870a0130fd00" containerName="kube-rbac-proxy-thanos" Oct 29 22:35:20.206043 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:20.205913 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af0e8682-1347-4724-b0a4-870a0130fd00" containerName="kube-rbac-proxy-thanos" Oct 29 22:35:20.206043 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:20.205923 5057 cpu_manager.go:414] "containerMap: removing container" podUID="af0e8682-1347-4724-b0a4-870a0130fd00" containerName="kube-rbac-proxy-web" Oct 29 22:35:20.206043 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:20.205929 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af0e8682-1347-4724-b0a4-870a0130fd00" containerName="kube-rbac-proxy-web" Oct 29 22:35:20.206043 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:20.205939 5057 cpu_manager.go:414] "containerMap: removing container" podUID="af0e8682-1347-4724-b0a4-870a0130fd00" containerName="init-config-reloader" Oct 29 22:35:20.206043 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:20.205944 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af0e8682-1347-4724-b0a4-870a0130fd00" containerName="init-config-reloader" Oct 29 22:35:20.206043 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:20.205951 5057 cpu_manager.go:414] "containerMap: removing container" podUID="af0e8682-1347-4724-b0a4-870a0130fd00" containerName="kube-rbac-proxy" Oct 29 22:35:20.206043 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:20.205956 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af0e8682-1347-4724-b0a4-870a0130fd00" containerName="kube-rbac-proxy" Oct 29 22:35:20.206043 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:20.205965 5057 cpu_manager.go:414] "containerMap: removing container" podUID="af0e8682-1347-4724-b0a4-870a0130fd00" containerName="thanos-sidecar" Oct 29 22:35:20.206043 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:20.205970 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af0e8682-1347-4724-b0a4-870a0130fd00" containerName="thanos-sidecar" Oct 29 22:35:20.206043 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:20.205980 5057 cpu_manager.go:414] "containerMap: removing container" podUID="af0e8682-1347-4724-b0a4-870a0130fd00" containerName="prometheus" Oct 29 22:35:20.206043 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:20.205985 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af0e8682-1347-4724-b0a4-870a0130fd00" containerName="prometheus" Oct 29 22:35:20.206450 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:20.206055 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="af0e8682-1347-4724-b0a4-870a0130fd00" containerName="config-reloader" Oct 29 22:35:20.206450 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:20.206063 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="af0e8682-1347-4724-b0a4-870a0130fd00" containerName="kube-rbac-proxy-thanos" Oct 29 22:35:20.206450 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:20.206069 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="af0e8682-1347-4724-b0a4-870a0130fd00" containerName="kube-rbac-proxy-web" Oct 29 22:35:20.206450 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:20.206076 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="af0e8682-1347-4724-b0a4-870a0130fd00" containerName="kube-rbac-proxy" Oct 29 22:35:20.206450 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:20.206083 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="af0e8682-1347-4724-b0a4-870a0130fd00" containerName="thanos-sidecar" Oct 29 22:35:20.206450 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:20.206090 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="af0e8682-1347-4724-b0a4-870a0130fd00" containerName="prometheus" Oct 29 22:35:20.209759 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:20.209743 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:35:20.223370 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:20.223348 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Oct 29 22:35:20.223445 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:20.223417 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Oct 29 22:35:20.223494 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:20.223480 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Oct 29 22:35:20.223546 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:20.223493 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Oct 29 22:35:20.223546 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:20.223501 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Oct 29 22:35:20.223546 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:20.223514 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Oct 29 22:35:20.223546 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:20.223522 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Oct 29 22:35:20.223702 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:20.223567 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-dc59t2jksncvi\"" Oct 29 22:35:20.223702 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:20.223572 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-cjpf4\"" Oct 29 22:35:20.223702 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:20.223567 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Oct 29 22:35:20.223702 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:20.223606 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Oct 29 22:35:20.228254 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:20.228231 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Oct 29 22:35:20.228254 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:20.228236 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Oct 29 22:35:20.228418 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:20.228310 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Oct 29 22:35:20.239188 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:20.239170 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Oct 29 22:35:20.239293 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:20.239254 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Oct 29 22:35:20.241322 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:20.241302 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 29 22:35:20.244169 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:20.244147 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ef110980-d6dc-44d9-8cbf-ec396e99db44-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"ef110980-d6dc-44d9-8cbf-ec396e99db44\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:35:20.244288 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:20.244178 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/ef110980-d6dc-44d9-8cbf-ec396e99db44-config\") pod \"prometheus-k8s-0\" (UID: \"ef110980-d6dc-44d9-8cbf-ec396e99db44\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:35:20.244288 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:20.244196 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ef110980-d6dc-44d9-8cbf-ec396e99db44-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"ef110980-d6dc-44d9-8cbf-ec396e99db44\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:35:20.244288 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:20.244221 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ef110980-d6dc-44d9-8cbf-ec396e99db44-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"ef110980-d6dc-44d9-8cbf-ec396e99db44\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:35:20.244478 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:20.244308 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ef110980-d6dc-44d9-8cbf-ec396e99db44-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"ef110980-d6dc-44d9-8cbf-ec396e99db44\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:35:20.244478 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:20.244396 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ef110980-d6dc-44d9-8cbf-ec396e99db44-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"ef110980-d6dc-44d9-8cbf-ec396e99db44\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:35:20.244478 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:20.244426 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ef110980-d6dc-44d9-8cbf-ec396e99db44-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"ef110980-d6dc-44d9-8cbf-ec396e99db44\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:35:20.244478 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:20.244447 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ef110980-d6dc-44d9-8cbf-ec396e99db44-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ef110980-d6dc-44d9-8cbf-ec396e99db44\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:35:20.244688 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:20.244505 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ef110980-d6dc-44d9-8cbf-ec396e99db44-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"ef110980-d6dc-44d9-8cbf-ec396e99db44\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:35:20.244688 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:20.244551 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ef110980-d6dc-44d9-8cbf-ec396e99db44-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"ef110980-d6dc-44d9-8cbf-ec396e99db44\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:35:20.244688 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:20.244614 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ef110980-d6dc-44d9-8cbf-ec396e99db44-web-config\") pod \"prometheus-k8s-0\" (UID: \"ef110980-d6dc-44d9-8cbf-ec396e99db44\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:35:20.244688 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:20.244635 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ef110980-d6dc-44d9-8cbf-ec396e99db44-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ef110980-d6dc-44d9-8cbf-ec396e99db44\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:35:20.244688 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:20.244655 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ef110980-d6dc-44d9-8cbf-ec396e99db44-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"ef110980-d6dc-44d9-8cbf-ec396e99db44\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:35:20.244931 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:20.244701 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ef110980-d6dc-44d9-8cbf-ec396e99db44-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ef110980-d6dc-44d9-8cbf-ec396e99db44\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:35:20.244931 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:20.244719 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zb6wx\" (UniqueName: \"kubernetes.io/projected/ef110980-d6dc-44d9-8cbf-ec396e99db44-kube-api-access-zb6wx\") pod \"prometheus-k8s-0\" (UID: \"ef110980-d6dc-44d9-8cbf-ec396e99db44\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:35:20.244931 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:20.244788 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ef110980-d6dc-44d9-8cbf-ec396e99db44-config-out\") pod \"prometheus-k8s-0\" (UID: \"ef110980-d6dc-44d9-8cbf-ec396e99db44\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:35:20.244931 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:20.244827 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ef110980-d6dc-44d9-8cbf-ec396e99db44-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"ef110980-d6dc-44d9-8cbf-ec396e99db44\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:35:20.244931 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:20.244868 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ef110980-d6dc-44d9-8cbf-ec396e99db44-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"ef110980-d6dc-44d9-8cbf-ec396e99db44\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:35:20.244931 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:20.244888 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ef110980-d6dc-44d9-8cbf-ec396e99db44-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"ef110980-d6dc-44d9-8cbf-ec396e99db44\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:35:20.244931 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:20.244907 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ef110980-d6dc-44d9-8cbf-ec396e99db44-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"ef110980-d6dc-44d9-8cbf-ec396e99db44\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:35:20.346404 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:20.346307 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ef110980-d6dc-44d9-8cbf-ec396e99db44-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"ef110980-d6dc-44d9-8cbf-ec396e99db44\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:35:20.346404 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:20.346371 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/ef110980-d6dc-44d9-8cbf-ec396e99db44-config\") pod \"prometheus-k8s-0\" (UID: \"ef110980-d6dc-44d9-8cbf-ec396e99db44\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:35:20.346404 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:20.346390 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ef110980-d6dc-44d9-8cbf-ec396e99db44-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"ef110980-d6dc-44d9-8cbf-ec396e99db44\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:35:20.346404 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:20.346408 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ef110980-d6dc-44d9-8cbf-ec396e99db44-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"ef110980-d6dc-44d9-8cbf-ec396e99db44\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:35:20.346694 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:20.346426 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ef110980-d6dc-44d9-8cbf-ec396e99db44-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"ef110980-d6dc-44d9-8cbf-ec396e99db44\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:35:20.346694 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:20.346446 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ef110980-d6dc-44d9-8cbf-ec396e99db44-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"ef110980-d6dc-44d9-8cbf-ec396e99db44\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:35:20.346694 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:20.346467 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ef110980-d6dc-44d9-8cbf-ec396e99db44-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"ef110980-d6dc-44d9-8cbf-ec396e99db44\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:35:20.346694 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:20.346599 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ef110980-d6dc-44d9-8cbf-ec396e99db44-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ef110980-d6dc-44d9-8cbf-ec396e99db44\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:35:20.346694 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:20.346645 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ef110980-d6dc-44d9-8cbf-ec396e99db44-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"ef110980-d6dc-44d9-8cbf-ec396e99db44\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:35:20.346694 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:20.346674 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ef110980-d6dc-44d9-8cbf-ec396e99db44-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"ef110980-d6dc-44d9-8cbf-ec396e99db44\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:35:20.346987 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:20.346704 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ef110980-d6dc-44d9-8cbf-ec396e99db44-web-config\") pod \"prometheus-k8s-0\" (UID: \"ef110980-d6dc-44d9-8cbf-ec396e99db44\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:35:20.346987 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:20.346729 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ef110980-d6dc-44d9-8cbf-ec396e99db44-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ef110980-d6dc-44d9-8cbf-ec396e99db44\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:35:20.346987 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:20.346755 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ef110980-d6dc-44d9-8cbf-ec396e99db44-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"ef110980-d6dc-44d9-8cbf-ec396e99db44\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:35:20.346987 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:20.346786 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ef110980-d6dc-44d9-8cbf-ec396e99db44-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ef110980-d6dc-44d9-8cbf-ec396e99db44\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:35:20.346987 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:20.346811 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zb6wx\" (UniqueName: \"kubernetes.io/projected/ef110980-d6dc-44d9-8cbf-ec396e99db44-kube-api-access-zb6wx\") pod \"prometheus-k8s-0\" (UID: \"ef110980-d6dc-44d9-8cbf-ec396e99db44\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:35:20.346987 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:20.346860 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ef110980-d6dc-44d9-8cbf-ec396e99db44-config-out\") pod \"prometheus-k8s-0\" (UID: \"ef110980-d6dc-44d9-8cbf-ec396e99db44\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:35:20.346987 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:20.346958 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ef110980-d6dc-44d9-8cbf-ec396e99db44-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"ef110980-d6dc-44d9-8cbf-ec396e99db44\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:35:20.346987 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:20.346974 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ef110980-d6dc-44d9-8cbf-ec396e99db44-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"ef110980-d6dc-44d9-8cbf-ec396e99db44\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:35:20.346987 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:20.346988 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ef110980-d6dc-44d9-8cbf-ec396e99db44-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"ef110980-d6dc-44d9-8cbf-ec396e99db44\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:35:20.347493 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:20.347044 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ef110980-d6dc-44d9-8cbf-ec396e99db44-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"ef110980-d6dc-44d9-8cbf-ec396e99db44\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:35:20.347493 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:20.347063 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ef110980-d6dc-44d9-8cbf-ec396e99db44-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"ef110980-d6dc-44d9-8cbf-ec396e99db44\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:35:20.347493 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:20.347104 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ef110980-d6dc-44d9-8cbf-ec396e99db44-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"ef110980-d6dc-44d9-8cbf-ec396e99db44\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:35:20.347493 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:20.347284 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ef110980-d6dc-44d9-8cbf-ec396e99db44-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"ef110980-d6dc-44d9-8cbf-ec396e99db44\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:35:20.348015 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:20.347986 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ef110980-d6dc-44d9-8cbf-ec396e99db44-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ef110980-d6dc-44d9-8cbf-ec396e99db44\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:35:20.349667 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:20.349633 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ef110980-d6dc-44d9-8cbf-ec396e99db44-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"ef110980-d6dc-44d9-8cbf-ec396e99db44\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:35:20.349667 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:20.349645 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/ef110980-d6dc-44d9-8cbf-ec396e99db44-config\") pod \"prometheus-k8s-0\" (UID: \"ef110980-d6dc-44d9-8cbf-ec396e99db44\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:35:20.349827 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:20.349771 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ef110980-d6dc-44d9-8cbf-ec396e99db44-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"ef110980-d6dc-44d9-8cbf-ec396e99db44\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:35:20.349898 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:20.349874 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ef110980-d6dc-44d9-8cbf-ec396e99db44-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"ef110980-d6dc-44d9-8cbf-ec396e99db44\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:35:20.350212 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:20.350185 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ef110980-d6dc-44d9-8cbf-ec396e99db44-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"ef110980-d6dc-44d9-8cbf-ec396e99db44\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:35:20.350284 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:20.350221 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ef110980-d6dc-44d9-8cbf-ec396e99db44-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"ef110980-d6dc-44d9-8cbf-ec396e99db44\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:35:20.350604 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:20.350583 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ef110980-d6dc-44d9-8cbf-ec396e99db44-config-out\") pod \"prometheus-k8s-0\" (UID: \"ef110980-d6dc-44d9-8cbf-ec396e99db44\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:35:20.350675 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:20.350622 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ef110980-d6dc-44d9-8cbf-ec396e99db44-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ef110980-d6dc-44d9-8cbf-ec396e99db44\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:35:20.351101 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:20.351073 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ef110980-d6dc-44d9-8cbf-ec396e99db44-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"ef110980-d6dc-44d9-8cbf-ec396e99db44\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:35:20.351208 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:20.351192 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ef110980-d6dc-44d9-8cbf-ec396e99db44-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ef110980-d6dc-44d9-8cbf-ec396e99db44\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:35:20.351273 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:20.351201 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ef110980-d6dc-44d9-8cbf-ec396e99db44-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"ef110980-d6dc-44d9-8cbf-ec396e99db44\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:35:20.352422 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:20.352401 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ef110980-d6dc-44d9-8cbf-ec396e99db44-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"ef110980-d6dc-44d9-8cbf-ec396e99db44\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:35:20.352599 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:20.352578 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ef110980-d6dc-44d9-8cbf-ec396e99db44-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"ef110980-d6dc-44d9-8cbf-ec396e99db44\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:35:20.352676 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:20.352659 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ef110980-d6dc-44d9-8cbf-ec396e99db44-web-config\") pod \"prometheus-k8s-0\" (UID: \"ef110980-d6dc-44d9-8cbf-ec396e99db44\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:35:20.352708 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:20.352665 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ef110980-d6dc-44d9-8cbf-ec396e99db44-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"ef110980-d6dc-44d9-8cbf-ec396e99db44\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:35:20.369305 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:20.369288 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zb6wx\" (UniqueName: \"kubernetes.io/projected/ef110980-d6dc-44d9-8cbf-ec396e99db44-kube-api-access-zb6wx\") pod \"prometheus-k8s-0\" (UID: \"ef110980-d6dc-44d9-8cbf-ec396e99db44\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:35:20.527768 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:20.527743 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:35:20.862120 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:20.862093 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 29 22:35:20.864358 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:35:20.864321 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef110980_d6dc_44d9_8cbf_ec396e99db44.slice/crio-baededc4005a75f44b84005e13efa0cfa18a88539bda651e46cb15384cb261a3 WatchSource:0}: Error finding container baededc4005a75f44b84005e13efa0cfa18a88539bda651e46cb15384cb261a3: Status 404 returned error can't find the container with id baededc4005a75f44b84005e13efa0cfa18a88539bda651e46cb15384cb261a3 Oct 29 22:35:21.131540 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:21.131461 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-559f47964f-9wc6l" event={"ID":"47845ebc-dce7-4409-9fc9-ffe9ce036c77","Type":"ContainerStarted","Data":"6ca0db511b04ac24adc3d4090c1846ee8ac3311d2ef6c16fb86779f738e717a2"} Oct 29 22:35:21.133087 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:21.133059 5057 generic.go:356] "Generic (PLEG): container finished" podID="ef110980-d6dc-44d9-8cbf-ec396e99db44" containerID="a9ddf1f3e7710c60e15b0248602a68cef290b72623e22b2fd72395a7dfac9eb3" exitCode=0 Oct 29 22:35:21.133201 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:21.133131 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ef110980-d6dc-44d9-8cbf-ec396e99db44","Type":"ContainerDied","Data":"a9ddf1f3e7710c60e15b0248602a68cef290b72623e22b2fd72395a7dfac9eb3"} Oct 29 22:35:21.133275 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:21.133196 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ef110980-d6dc-44d9-8cbf-ec396e99db44","Type":"ContainerStarted","Data":"baededc4005a75f44b84005e13efa0cfa18a88539bda651e46cb15384cb261a3"} Oct 29 22:35:21.177568 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:21.177524 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-559f47964f-9wc6l" podStartSLOduration=2.711675409 podStartE2EDuration="4.177509664s" totalImagesPullingTime="1.465834255s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:35:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:35:21.176358721 +0000 UTC m=+101.959572148" watchObservedRunningTime="2026-10-29 22:35:21.177509664 +0000 UTC m=+101.960723089" Oct 29 22:35:21.343717 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:21.343689 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af0e8682-1347-4724-b0a4-870a0130fd00" path="/var/lib/kubelet/pods/af0e8682-1347-4724-b0a4-870a0130fd00/volumes" Oct 29 22:35:21.385659 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:21.385632 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-765d49f595-55hx9"] Oct 29 22:35:21.394231 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:21.394208 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-765d49f595-55hx9" Oct 29 22:35:21.411025 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:21.410817 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-765d49f595-55hx9"] Oct 29 22:35:21.458593 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:21.458558 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vqsg\" (UniqueName: \"kubernetes.io/projected/e68006a8-7278-43c2-ba05-1fe673fe6c70-kube-api-access-8vqsg\") pod \"console-765d49f595-55hx9\" (UID: \"e68006a8-7278-43c2-ba05-1fe673fe6c70\") " pod="openshift-console/console-765d49f595-55hx9" Oct 29 22:35:21.458733 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:21.458615 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e68006a8-7278-43c2-ba05-1fe673fe6c70-console-config\") pod \"console-765d49f595-55hx9\" (UID: \"e68006a8-7278-43c2-ba05-1fe673fe6c70\") " pod="openshift-console/console-765d49f595-55hx9" Oct 29 22:35:21.459351 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:21.459300 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e68006a8-7278-43c2-ba05-1fe673fe6c70-service-ca\") pod \"console-765d49f595-55hx9\" (UID: \"e68006a8-7278-43c2-ba05-1fe673fe6c70\") " pod="openshift-console/console-765d49f595-55hx9" Oct 29 22:35:21.459470 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:21.459394 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e68006a8-7278-43c2-ba05-1fe673fe6c70-console-serving-cert\") pod \"console-765d49f595-55hx9\" (UID: \"e68006a8-7278-43c2-ba05-1fe673fe6c70\") " pod="openshift-console/console-765d49f595-55hx9" Oct 29 22:35:21.459470 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:21.459421 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e68006a8-7278-43c2-ba05-1fe673fe6c70-console-oauth-config\") pod \"console-765d49f595-55hx9\" (UID: \"e68006a8-7278-43c2-ba05-1fe673fe6c70\") " pod="openshift-console/console-765d49f595-55hx9" Oct 29 22:35:21.459470 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:21.459449 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e68006a8-7278-43c2-ba05-1fe673fe6c70-trusted-ca-bundle\") pod \"console-765d49f595-55hx9\" (UID: \"e68006a8-7278-43c2-ba05-1fe673fe6c70\") " pod="openshift-console/console-765d49f595-55hx9" Oct 29 22:35:21.459704 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:21.459490 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e68006a8-7278-43c2-ba05-1fe673fe6c70-tmp\") pod \"console-765d49f595-55hx9\" (UID: \"e68006a8-7278-43c2-ba05-1fe673fe6c70\") " pod="openshift-console/console-765d49f595-55hx9" Oct 29 22:35:21.459704 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:21.459544 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e68006a8-7278-43c2-ba05-1fe673fe6c70-oauth-serving-cert\") pod \"console-765d49f595-55hx9\" (UID: \"e68006a8-7278-43c2-ba05-1fe673fe6c70\") " pod="openshift-console/console-765d49f595-55hx9" Oct 29 22:35:21.523125 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:21.523089 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:35:21.560461 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:21.560439 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e68006a8-7278-43c2-ba05-1fe673fe6c70-service-ca\") pod \"console-765d49f595-55hx9\" (UID: \"e68006a8-7278-43c2-ba05-1fe673fe6c70\") " pod="openshift-console/console-765d49f595-55hx9" Oct 29 22:35:21.560553 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:21.560494 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e68006a8-7278-43c2-ba05-1fe673fe6c70-console-serving-cert\") pod \"console-765d49f595-55hx9\" (UID: \"e68006a8-7278-43c2-ba05-1fe673fe6c70\") " pod="openshift-console/console-765d49f595-55hx9" Oct 29 22:35:21.560553 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:21.560520 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e68006a8-7278-43c2-ba05-1fe673fe6c70-console-oauth-config\") pod \"console-765d49f595-55hx9\" (UID: \"e68006a8-7278-43c2-ba05-1fe673fe6c70\") " pod="openshift-console/console-765d49f595-55hx9" Oct 29 22:35:21.560553 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:21.560549 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e68006a8-7278-43c2-ba05-1fe673fe6c70-trusted-ca-bundle\") pod \"console-765d49f595-55hx9\" (UID: \"e68006a8-7278-43c2-ba05-1fe673fe6c70\") " pod="openshift-console/console-765d49f595-55hx9" Oct 29 22:35:21.560662 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:21.560588 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e68006a8-7278-43c2-ba05-1fe673fe6c70-tmp\") pod \"console-765d49f595-55hx9\" (UID: \"e68006a8-7278-43c2-ba05-1fe673fe6c70\") " pod="openshift-console/console-765d49f595-55hx9" Oct 29 22:35:21.560662 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:21.560639 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e68006a8-7278-43c2-ba05-1fe673fe6c70-oauth-serving-cert\") pod \"console-765d49f595-55hx9\" (UID: \"e68006a8-7278-43c2-ba05-1fe673fe6c70\") " pod="openshift-console/console-765d49f595-55hx9" Oct 29 22:35:21.560720 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:21.560689 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vqsg\" (UniqueName: \"kubernetes.io/projected/e68006a8-7278-43c2-ba05-1fe673fe6c70-kube-api-access-8vqsg\") pod \"console-765d49f595-55hx9\" (UID: \"e68006a8-7278-43c2-ba05-1fe673fe6c70\") " pod="openshift-console/console-765d49f595-55hx9" Oct 29 22:35:21.560772 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:21.560728 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e68006a8-7278-43c2-ba05-1fe673fe6c70-console-config\") pod \"console-765d49f595-55hx9\" (UID: \"e68006a8-7278-43c2-ba05-1fe673fe6c70\") " pod="openshift-console/console-765d49f595-55hx9" Oct 29 22:35:21.561188 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:21.561165 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e68006a8-7278-43c2-ba05-1fe673fe6c70-service-ca\") pod \"console-765d49f595-55hx9\" (UID: \"e68006a8-7278-43c2-ba05-1fe673fe6c70\") " pod="openshift-console/console-765d49f595-55hx9" Oct 29 22:35:21.561425 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:21.561404 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e68006a8-7278-43c2-ba05-1fe673fe6c70-console-config\") pod \"console-765d49f595-55hx9\" (UID: \"e68006a8-7278-43c2-ba05-1fe673fe6c70\") " pod="openshift-console/console-765d49f595-55hx9" Oct 29 22:35:21.561708 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:21.561689 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e68006a8-7278-43c2-ba05-1fe673fe6c70-tmp\") pod \"console-765d49f595-55hx9\" (UID: \"e68006a8-7278-43c2-ba05-1fe673fe6c70\") " pod="openshift-console/console-765d49f595-55hx9" Oct 29 22:35:21.561952 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:21.561933 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e68006a8-7278-43c2-ba05-1fe673fe6c70-trusted-ca-bundle\") pod \"console-765d49f595-55hx9\" (UID: \"e68006a8-7278-43c2-ba05-1fe673fe6c70\") " pod="openshift-console/console-765d49f595-55hx9" Oct 29 22:35:21.562255 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:21.562235 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e68006a8-7278-43c2-ba05-1fe673fe6c70-oauth-serving-cert\") pod \"console-765d49f595-55hx9\" (UID: \"e68006a8-7278-43c2-ba05-1fe673fe6c70\") " pod="openshift-console/console-765d49f595-55hx9" Oct 29 22:35:21.564241 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:21.564219 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e68006a8-7278-43c2-ba05-1fe673fe6c70-console-serving-cert\") pod \"console-765d49f595-55hx9\" (UID: \"e68006a8-7278-43c2-ba05-1fe673fe6c70\") " pod="openshift-console/console-765d49f595-55hx9" Oct 29 22:35:21.564376 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:21.564302 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e68006a8-7278-43c2-ba05-1fe673fe6c70-console-oauth-config\") pod \"console-765d49f595-55hx9\" (UID: \"e68006a8-7278-43c2-ba05-1fe673fe6c70\") " pod="openshift-console/console-765d49f595-55hx9" Oct 29 22:35:21.583548 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:21.583521 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vqsg\" (UniqueName: \"kubernetes.io/projected/e68006a8-7278-43c2-ba05-1fe673fe6c70-kube-api-access-8vqsg\") pod \"console-765d49f595-55hx9\" (UID: \"e68006a8-7278-43c2-ba05-1fe673fe6c70\") " pod="openshift-console/console-765d49f595-55hx9" Oct 29 22:35:21.630438 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:21.630410 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:35:21.727827 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:21.727785 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-765d49f595-55hx9" Oct 29 22:35:21.792684 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:21.792650 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:35:21.901829 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:21.901795 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:35:22.023395 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:22.023095 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-765d49f595-55hx9"] Oct 29 22:35:22.025900 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:35:22.025875 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode68006a8_7278_43c2_ba05_1fe673fe6c70.slice/crio-8048cf004232ff95dae3eb83f15c1b76aa731c52589ab3142e56b4d1aa496cf3 WatchSource:0}: Error finding container 8048cf004232ff95dae3eb83f15c1b76aa731c52589ab3142e56b4d1aa496cf3: Status 404 returned error can't find the container with id 8048cf004232ff95dae3eb83f15c1b76aa731c52589ab3142e56b4d1aa496cf3 Oct 29 22:35:22.141687 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:22.140591 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-765d49f595-55hx9" event={"ID":"e68006a8-7278-43c2-ba05-1fe673fe6c70","Type":"ContainerStarted","Data":"b6866cc7d34b9eaea771d015cff2f4191184c7bccf692fca5c970a44db2df792"} Oct 29 22:35:22.141687 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:22.140667 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-765d49f595-55hx9" event={"ID":"e68006a8-7278-43c2-ba05-1fe673fe6c70","Type":"ContainerStarted","Data":"8048cf004232ff95dae3eb83f15c1b76aa731c52589ab3142e56b4d1aa496cf3"} Oct 29 22:35:22.144561 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:22.144534 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ef110980-d6dc-44d9-8cbf-ec396e99db44","Type":"ContainerStarted","Data":"44eec5e9b3f1695f8bff0a0923d387a0565132b09a2e9021df5f9d74304f0fe8"} Oct 29 22:35:22.144674 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:22.144593 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ef110980-d6dc-44d9-8cbf-ec396e99db44","Type":"ContainerStarted","Data":"4721e0cc64fd7c983087bd7b1b6f9fe07ebdefc9788af887fef0e6fd580111dc"} Oct 29 22:35:22.144674 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:22.144620 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ef110980-d6dc-44d9-8cbf-ec396e99db44","Type":"ContainerStarted","Data":"c717f8760debbb58d8e5e254ff37256c4bb932f856c1ff99548018b8d0ffadc5"} Oct 29 22:35:22.144674 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:22.144648 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ef110980-d6dc-44d9-8cbf-ec396e99db44","Type":"ContainerStarted","Data":"8d1827e10b5097fd3f3748a7e7a8a3cd76054a508f8d5de383b0ab7b92198033"} Oct 29 22:35:22.144674 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:22.144673 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ef110980-d6dc-44d9-8cbf-ec396e99db44","Type":"ContainerStarted","Data":"ca96d7c8e753e9e77477212ef69bea7b79189882c4aed0c787984c7d52851fa5"} Oct 29 22:35:22.144825 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:22.144695 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ef110980-d6dc-44d9-8cbf-ec396e99db44","Type":"ContainerStarted","Data":"e3771ec5bc0fb330820f94f676b43dc25ac807b5e21750390f14d9f2db2db75f"} Oct 29 22:35:22.181201 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:22.181154 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-765d49f595-55hx9" podStartSLOduration=1.181140217 podStartE2EDuration="1.181140217s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:35:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:35:22.179836076 +0000 UTC m=+102.963049503" watchObservedRunningTime="2026-10-29 22:35:22.181140217 +0000 UTC m=+102.964353642" Oct 29 22:35:22.234667 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:22.234611 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.213530214 podStartE2EDuration="2.234593332s" totalImagesPullingTime="0s" totalInitContainerRuntime="21.063118ms" isStatefulPod=false podCreationTimestamp="2026-10-29 22:35:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:35:22.231849021 +0000 UTC m=+103.015062465" watchObservedRunningTime="2026-10-29 22:35:22.234593332 +0000 UTC m=+103.017806759" Oct 29 22:35:24.588536 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:24.588496 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:35:24.708255 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:24.708226 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:35:25.528528 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:25.528496 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:35:28.862280 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:28.862244 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:35:31.728221 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:31.728185 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-765d49f595-55hx9" Oct 29 22:35:31.728678 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:31.728238 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-765d49f595-55hx9" Oct 29 22:35:31.732815 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:31.732794 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-765d49f595-55hx9" Oct 29 22:35:32.196531 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:32.196463 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-765d49f595-55hx9" Oct 29 22:35:32.266485 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:32.266456 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-f8f65dd7c-j9qgf"] Oct 29 22:35:49.027002 ip-10-0-140-74 kubenswrapper[5057]: E1029 22:35:49.026961 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d2dnn" containerName="extractor" Oct 29 22:35:57.293371 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:57.293243 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-f8f65dd7c-j9qgf" podUID="9fdd36b9-cabf-4104-9d4e-bfe0e6aa083d" containerName="console" containerID="cri-o://9161f995b8554f4488030c35b87226fa8fbb72dec5b99ac5b2e8dbf6195c545e" gracePeriod=15 Oct 29 22:35:57.570693 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:57.570666 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-f8f65dd7c-j9qgf" Oct 29 22:35:57.633803 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:57.633767 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9fdd36b9-cabf-4104-9d4e-bfe0e6aa083d-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9fdd36b9-cabf-4104-9d4e-bfe0e6aa083d-console-serving-cert\") pod \"9fdd36b9-cabf-4104-9d4e-bfe0e6aa083d\" (UID: \"9fdd36b9-cabf-4104-9d4e-bfe0e6aa083d\") " Oct 29 22:35:57.633949 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:57.633856 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9fdd36b9-cabf-4104-9d4e-bfe0e6aa083d-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9fdd36b9-cabf-4104-9d4e-bfe0e6aa083d-oauth-serving-cert\") pod \"9fdd36b9-cabf-4104-9d4e-bfe0e6aa083d\" (UID: \"9fdd36b9-cabf-4104-9d4e-bfe0e6aa083d\") " Oct 29 22:35:57.633949 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:57.633888 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9fdd36b9-cabf-4104-9d4e-bfe0e6aa083d-service-ca\" (UniqueName: \"kubernetes.io/configmap/9fdd36b9-cabf-4104-9d4e-bfe0e6aa083d-service-ca\") pod \"9fdd36b9-cabf-4104-9d4e-bfe0e6aa083d\" (UID: \"9fdd36b9-cabf-4104-9d4e-bfe0e6aa083d\") " Oct 29 22:35:57.634029 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:57.633961 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fdd36b9-cabf-4104-9d4e-bfe0e6aa083d-tmp\" (UniqueName: \"kubernetes.io/empty-dir/9fdd36b9-cabf-4104-9d4e-bfe0e6aa083d-tmp\") pod \"9fdd36b9-cabf-4104-9d4e-bfe0e6aa083d\" (UID: \"9fdd36b9-cabf-4104-9d4e-bfe0e6aa083d\") " Oct 29 22:35:57.634029 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:57.633997 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9fdd36b9-cabf-4104-9d4e-bfe0e6aa083d-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9fdd36b9-cabf-4104-9d4e-bfe0e6aa083d-console-oauth-config\") pod \"9fdd36b9-cabf-4104-9d4e-bfe0e6aa083d\" (UID: \"9fdd36b9-cabf-4104-9d4e-bfe0e6aa083d\") " Oct 29 22:35:57.634138 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:57.634028 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9fdd36b9-cabf-4104-9d4e-bfe0e6aa083d-kube-api-access-52qmj\" (UniqueName: \"kubernetes.io/projected/9fdd36b9-cabf-4104-9d4e-bfe0e6aa083d-kube-api-access-52qmj\") pod \"9fdd36b9-cabf-4104-9d4e-bfe0e6aa083d\" (UID: \"9fdd36b9-cabf-4104-9d4e-bfe0e6aa083d\") " Oct 29 22:35:57.634138 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:57.634093 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9fdd36b9-cabf-4104-9d4e-bfe0e6aa083d-console-config\" (UniqueName: \"kubernetes.io/configmap/9fdd36b9-cabf-4104-9d4e-bfe0e6aa083d-console-config\") pod \"9fdd36b9-cabf-4104-9d4e-bfe0e6aa083d\" (UID: \"9fdd36b9-cabf-4104-9d4e-bfe0e6aa083d\") " Oct 29 22:35:57.634138 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:57.634132 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9fdd36b9-cabf-4104-9d4e-bfe0e6aa083d-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9fdd36b9-cabf-4104-9d4e-bfe0e6aa083d-trusted-ca-bundle\") pod \"9fdd36b9-cabf-4104-9d4e-bfe0e6aa083d\" (UID: \"9fdd36b9-cabf-4104-9d4e-bfe0e6aa083d\") " Oct 29 22:35:57.634324 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:57.634292 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fdd36b9-cabf-4104-9d4e-bfe0e6aa083d-oauth-serving-cert" pod "9fdd36b9-cabf-4104-9d4e-bfe0e6aa083d" (UID: "9fdd36b9-cabf-4104-9d4e-bfe0e6aa083d"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:35:57.634466 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:57.634365 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fdd36b9-cabf-4104-9d4e-bfe0e6aa083d-service-ca" pod "9fdd36b9-cabf-4104-9d4e-bfe0e6aa083d" (UID: "9fdd36b9-cabf-4104-9d4e-bfe0e6aa083d"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:35:57.634697 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:57.634656 5057 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9fdd36b9-cabf-4104-9d4e-bfe0e6aa083d-oauth-serving-cert\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 22:35:57.634697 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:57.634681 5057 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9fdd36b9-cabf-4104-9d4e-bfe0e6aa083d-service-ca\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 22:35:57.634842 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:57.634716 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fdd36b9-cabf-4104-9d4e-bfe0e6aa083d-tmp" pod "9fdd36b9-cabf-4104-9d4e-bfe0e6aa083d" (UID: "9fdd36b9-cabf-4104-9d4e-bfe0e6aa083d"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:35:57.634842 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:57.634758 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fdd36b9-cabf-4104-9d4e-bfe0e6aa083d-trusted-ca-bundle" pod "9fdd36b9-cabf-4104-9d4e-bfe0e6aa083d" (UID: "9fdd36b9-cabf-4104-9d4e-bfe0e6aa083d"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:35:57.634842 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:57.634830 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fdd36b9-cabf-4104-9d4e-bfe0e6aa083d-console-config" pod "9fdd36b9-cabf-4104-9d4e-bfe0e6aa083d" (UID: "9fdd36b9-cabf-4104-9d4e-bfe0e6aa083d"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:35:57.636041 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:57.636018 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9fdd36b9-cabf-4104-9d4e-bfe0e6aa083d-console-serving-cert" pod "9fdd36b9-cabf-4104-9d4e-bfe0e6aa083d" (UID: "9fdd36b9-cabf-4104-9d4e-bfe0e6aa083d"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:35:57.636631 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:57.636605 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9fdd36b9-cabf-4104-9d4e-bfe0e6aa083d-console-oauth-config" pod "9fdd36b9-cabf-4104-9d4e-bfe0e6aa083d" (UID: "9fdd36b9-cabf-4104-9d4e-bfe0e6aa083d"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:35:57.636693 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:57.636638 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fdd36b9-cabf-4104-9d4e-bfe0e6aa083d-kube-api-access-52qmj" pod "9fdd36b9-cabf-4104-9d4e-bfe0e6aa083d" (UID: "9fdd36b9-cabf-4104-9d4e-bfe0e6aa083d"). InnerVolumeSpecName "kube-api-access-52qmj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:35:57.735157 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:57.735131 5057 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9fdd36b9-cabf-4104-9d4e-bfe0e6aa083d-tmp\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 22:35:57.735157 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:57.735153 5057 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9fdd36b9-cabf-4104-9d4e-bfe0e6aa083d-console-oauth-config\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 22:35:57.735313 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:57.735163 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-52qmj\" (UniqueName: \"kubernetes.io/projected/9fdd36b9-cabf-4104-9d4e-bfe0e6aa083d-kube-api-access-52qmj\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 22:35:57.735313 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:57.735174 5057 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9fdd36b9-cabf-4104-9d4e-bfe0e6aa083d-console-config\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 22:35:57.735313 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:57.735182 5057 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9fdd36b9-cabf-4104-9d4e-bfe0e6aa083d-trusted-ca-bundle\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 22:35:57.735313 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:57.735191 5057 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9fdd36b9-cabf-4104-9d4e-bfe0e6aa083d-console-serving-cert\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 22:35:58.348770 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:58.348735 5057 generic.go:356] "Generic (PLEG): container finished" podID="9fdd36b9-cabf-4104-9d4e-bfe0e6aa083d" containerID="9161f995b8554f4488030c35b87226fa8fbb72dec5b99ac5b2e8dbf6195c545e" exitCode=0 Oct 29 22:35:58.349199 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:58.348778 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-f8f65dd7c-j9qgf" Oct 29 22:35:58.349199 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:58.348787 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-f8f65dd7c-j9qgf" event={"ID":"9fdd36b9-cabf-4104-9d4e-bfe0e6aa083d","Type":"ContainerDied","Data":"9161f995b8554f4488030c35b87226fa8fbb72dec5b99ac5b2e8dbf6195c545e"} Oct 29 22:35:58.349199 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:58.348830 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-f8f65dd7c-j9qgf" event={"ID":"9fdd36b9-cabf-4104-9d4e-bfe0e6aa083d","Type":"ContainerDied","Data":"1e324881c5c7d684ea1a11595b7e9ee3dc697b71c2ed68a6715a83e2283c757f"} Oct 29 22:35:58.349199 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:58.348856 5057 scope.go:122] "RemoveContainer" containerID="9161f995b8554f4488030c35b87226fa8fbb72dec5b99ac5b2e8dbf6195c545e" Oct 29 22:35:58.358525 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:58.358504 5057 scope.go:122] "RemoveContainer" containerID="9161f995b8554f4488030c35b87226fa8fbb72dec5b99ac5b2e8dbf6195c545e" Oct 29 22:35:58.358804 ip-10-0-140-74 kubenswrapper[5057]: E1029 22:35:58.358784 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9161f995b8554f4488030c35b87226fa8fbb72dec5b99ac5b2e8dbf6195c545e\": container with ID starting with 9161f995b8554f4488030c35b87226fa8fbb72dec5b99ac5b2e8dbf6195c545e not found: ID does not exist" containerID="9161f995b8554f4488030c35b87226fa8fbb72dec5b99ac5b2e8dbf6195c545e" Oct 29 22:35:58.358849 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:58.358813 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9161f995b8554f4488030c35b87226fa8fbb72dec5b99ac5b2e8dbf6195c545e"} err="failed to get container status \"9161f995b8554f4488030c35b87226fa8fbb72dec5b99ac5b2e8dbf6195c545e\": rpc error: code = NotFound desc = could not find container \"9161f995b8554f4488030c35b87226fa8fbb72dec5b99ac5b2e8dbf6195c545e\": container with ID starting with 9161f995b8554f4488030c35b87226fa8fbb72dec5b99ac5b2e8dbf6195c545e not found: ID does not exist" Oct 29 22:35:58.394126 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:58.394099 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-f8f65dd7c-j9qgf"] Oct 29 22:35:58.398677 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:58.398655 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-f8f65dd7c-j9qgf"] Oct 29 22:35:59.326139 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:35:59.326106 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9fdd36b9-cabf-4104-9d4e-bfe0e6aa083d" path="/var/lib/kubelet/pods/9fdd36b9-cabf-4104-9d4e-bfe0e6aa083d/volumes" Oct 29 22:36:19.026663 ip-10-0-140-74 kubenswrapper[5057]: E1029 22:36:19.026624 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d2dnn" containerName="extractor" Oct 29 22:36:20.528385 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:36:20.528354 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:36:20.544572 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:36:20.544542 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:36:21.480906 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:36:21.480879 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:36:28.428958 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:36:28.428918 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:36:28.535475 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:36:28.535433 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:36:28.997124 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:36:28.997092 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:36:33.573432 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:36:33.573395 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-9bbcb6fd8-w6wx5"] Oct 29 22:36:33.573851 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:36:33.573835 5057 cpu_manager.go:414] "containerMap: removing container" podUID="9fdd36b9-cabf-4104-9d4e-bfe0e6aa083d" containerName="console" Oct 29 22:36:33.573894 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:36:33.573852 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9fdd36b9-cabf-4104-9d4e-bfe0e6aa083d" containerName="console" Oct 29 22:36:33.573984 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:36:33.573975 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="9fdd36b9-cabf-4104-9d4e-bfe0e6aa083d" containerName="console" Oct 29 22:36:33.578451 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:36:33.578431 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-9bbcb6fd8-w6wx5" Oct 29 22:36:33.603479 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:36:33.603454 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-9bbcb6fd8-w6wx5"] Oct 29 22:36:33.702305 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:36:33.702268 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:36:33.754817 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:36:33.754788 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/cd69f88c-fed7-4f6a-8c88-8d5ac7642061-console-serving-cert\") pod \"console-9bbcb6fd8-w6wx5\" (UID: \"cd69f88c-fed7-4f6a-8c88-8d5ac7642061\") " pod="openshift-console/console-9bbcb6fd8-w6wx5" Oct 29 22:36:33.754817 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:36:33.754819 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/cd69f88c-fed7-4f6a-8c88-8d5ac7642061-console-oauth-config\") pod \"console-9bbcb6fd8-w6wx5\" (UID: \"cd69f88c-fed7-4f6a-8c88-8d5ac7642061\") " pod="openshift-console/console-9bbcb6fd8-w6wx5" Oct 29 22:36:33.754973 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:36:33.754836 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95msm\" (UniqueName: \"kubernetes.io/projected/cd69f88c-fed7-4f6a-8c88-8d5ac7642061-kube-api-access-95msm\") pod \"console-9bbcb6fd8-w6wx5\" (UID: \"cd69f88c-fed7-4f6a-8c88-8d5ac7642061\") " pod="openshift-console/console-9bbcb6fd8-w6wx5" Oct 29 22:36:33.754973 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:36:33.754907 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/cd69f88c-fed7-4f6a-8c88-8d5ac7642061-service-ca\") pod \"console-9bbcb6fd8-w6wx5\" (UID: \"cd69f88c-fed7-4f6a-8c88-8d5ac7642061\") " pod="openshift-console/console-9bbcb6fd8-w6wx5" Oct 29 22:36:33.754973 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:36:33.754969 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/cd69f88c-fed7-4f6a-8c88-8d5ac7642061-oauth-serving-cert\") pod \"console-9bbcb6fd8-w6wx5\" (UID: \"cd69f88c-fed7-4f6a-8c88-8d5ac7642061\") " pod="openshift-console/console-9bbcb6fd8-w6wx5" Oct 29 22:36:33.755066 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:36:33.754991 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cd69f88c-fed7-4f6a-8c88-8d5ac7642061-tmp\") pod \"console-9bbcb6fd8-w6wx5\" (UID: \"cd69f88c-fed7-4f6a-8c88-8d5ac7642061\") " pod="openshift-console/console-9bbcb6fd8-w6wx5" Oct 29 22:36:33.755100 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:36:33.755072 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/cd69f88c-fed7-4f6a-8c88-8d5ac7642061-console-config\") pod \"console-9bbcb6fd8-w6wx5\" (UID: \"cd69f88c-fed7-4f6a-8c88-8d5ac7642061\") " pod="openshift-console/console-9bbcb6fd8-w6wx5" Oct 29 22:36:33.755136 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:36:33.755129 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cd69f88c-fed7-4f6a-8c88-8d5ac7642061-trusted-ca-bundle\") pod \"console-9bbcb6fd8-w6wx5\" (UID: \"cd69f88c-fed7-4f6a-8c88-8d5ac7642061\") " pod="openshift-console/console-9bbcb6fd8-w6wx5" Oct 29 22:36:33.814954 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:36:33.814896 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:36:33.855957 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:36:33.855898 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/cd69f88c-fed7-4f6a-8c88-8d5ac7642061-console-serving-cert\") pod \"console-9bbcb6fd8-w6wx5\" (UID: \"cd69f88c-fed7-4f6a-8c88-8d5ac7642061\") " pod="openshift-console/console-9bbcb6fd8-w6wx5" Oct 29 22:36:33.855957 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:36:33.855931 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/cd69f88c-fed7-4f6a-8c88-8d5ac7642061-console-oauth-config\") pod \"console-9bbcb6fd8-w6wx5\" (UID: \"cd69f88c-fed7-4f6a-8c88-8d5ac7642061\") " pod="openshift-console/console-9bbcb6fd8-w6wx5" Oct 29 22:36:33.855957 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:36:33.855948 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95msm\" (UniqueName: \"kubernetes.io/projected/cd69f88c-fed7-4f6a-8c88-8d5ac7642061-kube-api-access-95msm\") pod \"console-9bbcb6fd8-w6wx5\" (UID: \"cd69f88c-fed7-4f6a-8c88-8d5ac7642061\") " pod="openshift-console/console-9bbcb6fd8-w6wx5" Oct 29 22:36:33.856150 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:36:33.855977 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/cd69f88c-fed7-4f6a-8c88-8d5ac7642061-service-ca\") pod \"console-9bbcb6fd8-w6wx5\" (UID: \"cd69f88c-fed7-4f6a-8c88-8d5ac7642061\") " pod="openshift-console/console-9bbcb6fd8-w6wx5" Oct 29 22:36:33.856150 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:36:33.856010 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/cd69f88c-fed7-4f6a-8c88-8d5ac7642061-oauth-serving-cert\") pod \"console-9bbcb6fd8-w6wx5\" (UID: \"cd69f88c-fed7-4f6a-8c88-8d5ac7642061\") " pod="openshift-console/console-9bbcb6fd8-w6wx5" Oct 29 22:36:33.856150 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:36:33.856034 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cd69f88c-fed7-4f6a-8c88-8d5ac7642061-tmp\") pod \"console-9bbcb6fd8-w6wx5\" (UID: \"cd69f88c-fed7-4f6a-8c88-8d5ac7642061\") " pod="openshift-console/console-9bbcb6fd8-w6wx5" Oct 29 22:36:33.856150 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:36:33.856096 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/cd69f88c-fed7-4f6a-8c88-8d5ac7642061-console-config\") pod \"console-9bbcb6fd8-w6wx5\" (UID: \"cd69f88c-fed7-4f6a-8c88-8d5ac7642061\") " pod="openshift-console/console-9bbcb6fd8-w6wx5" Oct 29 22:36:33.856371 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:36:33.856232 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cd69f88c-fed7-4f6a-8c88-8d5ac7642061-trusted-ca-bundle\") pod \"console-9bbcb6fd8-w6wx5\" (UID: \"cd69f88c-fed7-4f6a-8c88-8d5ac7642061\") " pod="openshift-console/console-9bbcb6fd8-w6wx5" Oct 29 22:36:33.856566 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:36:33.856534 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cd69f88c-fed7-4f6a-8c88-8d5ac7642061-tmp\") pod \"console-9bbcb6fd8-w6wx5\" (UID: \"cd69f88c-fed7-4f6a-8c88-8d5ac7642061\") " pod="openshift-console/console-9bbcb6fd8-w6wx5" Oct 29 22:36:33.856949 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:36:33.856918 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/cd69f88c-fed7-4f6a-8c88-8d5ac7642061-oauth-serving-cert\") pod \"console-9bbcb6fd8-w6wx5\" (UID: \"cd69f88c-fed7-4f6a-8c88-8d5ac7642061\") " pod="openshift-console/console-9bbcb6fd8-w6wx5" Oct 29 22:36:33.856949 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:36:33.856936 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/cd69f88c-fed7-4f6a-8c88-8d5ac7642061-console-config\") pod \"console-9bbcb6fd8-w6wx5\" (UID: \"cd69f88c-fed7-4f6a-8c88-8d5ac7642061\") " pod="openshift-console/console-9bbcb6fd8-w6wx5" Oct 29 22:36:33.857101 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:36:33.856978 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/cd69f88c-fed7-4f6a-8c88-8d5ac7642061-service-ca\") pod \"console-9bbcb6fd8-w6wx5\" (UID: \"cd69f88c-fed7-4f6a-8c88-8d5ac7642061\") " pod="openshift-console/console-9bbcb6fd8-w6wx5" Oct 29 22:36:33.857101 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:36:33.857032 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cd69f88c-fed7-4f6a-8c88-8d5ac7642061-trusted-ca-bundle\") pod \"console-9bbcb6fd8-w6wx5\" (UID: \"cd69f88c-fed7-4f6a-8c88-8d5ac7642061\") " pod="openshift-console/console-9bbcb6fd8-w6wx5" Oct 29 22:36:33.858535 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:36:33.858515 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/cd69f88c-fed7-4f6a-8c88-8d5ac7642061-console-serving-cert\") pod \"console-9bbcb6fd8-w6wx5\" (UID: \"cd69f88c-fed7-4f6a-8c88-8d5ac7642061\") " pod="openshift-console/console-9bbcb6fd8-w6wx5" Oct 29 22:36:33.858661 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:36:33.858642 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/cd69f88c-fed7-4f6a-8c88-8d5ac7642061-console-oauth-config\") pod \"console-9bbcb6fd8-w6wx5\" (UID: \"cd69f88c-fed7-4f6a-8c88-8d5ac7642061\") " pod="openshift-console/console-9bbcb6fd8-w6wx5" Oct 29 22:36:33.876845 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:36:33.876828 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-95msm\" (UniqueName: \"kubernetes.io/projected/cd69f88c-fed7-4f6a-8c88-8d5ac7642061-kube-api-access-95msm\") pod \"console-9bbcb6fd8-w6wx5\" (UID: \"cd69f88c-fed7-4f6a-8c88-8d5ac7642061\") " pod="openshift-console/console-9bbcb6fd8-w6wx5" Oct 29 22:36:33.890691 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:36:33.890671 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-9bbcb6fd8-w6wx5" Oct 29 22:36:34.009199 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:36:34.009166 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:36:34.116754 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:36:34.116666 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:36:34.226406 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:36:34.226371 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-9bbcb6fd8-w6wx5"] Oct 29 22:36:34.228725 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:36:34.228697 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd69f88c_fed7_4f6a_8c88_8d5ac7642061.slice/crio-c56b03e7b2d826e83ccac42d736178a644ce621c61396257e7216698552707ff WatchSource:0}: Error finding container c56b03e7b2d826e83ccac42d736178a644ce621c61396257e7216698552707ff: Status 404 returned error can't find the container with id c56b03e7b2d826e83ccac42d736178a644ce621c61396257e7216698552707ff Oct 29 22:36:34.524806 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:36:34.524767 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-9bbcb6fd8-w6wx5" event={"ID":"cd69f88c-fed7-4f6a-8c88-8d5ac7642061","Type":"ContainerStarted","Data":"e91b1b8f7cc1be399511625263854f291eb2a52f6efa7c4f1ff3e8c6782e16bd"} Oct 29 22:36:34.525001 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:36:34.524818 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-9bbcb6fd8-w6wx5" event={"ID":"cd69f88c-fed7-4f6a-8c88-8d5ac7642061","Type":"ContainerStarted","Data":"c56b03e7b2d826e83ccac42d736178a644ce621c61396257e7216698552707ff"} Oct 29 22:36:34.563050 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:36:34.563010 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-9bbcb6fd8-w6wx5" podStartSLOduration=1.562997537 podStartE2EDuration="1.562997537s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:36:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:36:34.562901335 +0000 UTC m=+175.346114761" watchObservedRunningTime="2026-10-29 22:36:34.562997537 +0000 UTC m=+175.346210963" Oct 29 22:36:41.157337 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:36:41.157303 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:36:41.268536 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:36:41.268494 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:36:41.558105 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:36:41.558069 5057 generic.go:356] "Generic (PLEG): container finished" podID="84569698-cff8-449e-9acc-2c8408581d52" containerID="e0eaf237b3d7efe83083caa49c60993203bbbeeb68c952a53b13ecc5f457c211" exitCode=0 Oct 29 22:36:41.558280 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:36:41.558141 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-ktp5j-2qdc5" event={"ID":"84569698-cff8-449e-9acc-2c8408581d52","Type":"ContainerDied","Data":"e0eaf237b3d7efe83083caa49c60993203bbbeeb68c952a53b13ecc5f457c211"} Oct 29 22:36:42.695610 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:36:42.695587 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-ktp5j-2qdc5" Oct 29 22:36:42.748067 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:36:42.748041 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84569698-cff8-449e-9acc-2c8408581d52-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/84569698-cff8-449e-9acc-2c8408581d52-archives-path\") pod \"84569698-cff8-449e-9acc-2c8408581d52\" (UID: \"84569698-cff8-449e-9acc-2c8408581d52\") " Oct 29 22:36:42.748212 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:36:42.748155 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84569698-cff8-449e-9acc-2c8408581d52-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/84569698-cff8-449e-9acc-2c8408581d52-service-ca-bundle\") pod \"84569698-cff8-449e-9acc-2c8408581d52\" (UID: \"84569698-cff8-449e-9acc-2c8408581d52\") " Oct 29 22:36:42.748276 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:36:42.748245 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84569698-cff8-449e-9acc-2c8408581d52-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/84569698-cff8-449e-9acc-2c8408581d52-trusted-ca-bundle\") pod \"84569698-cff8-449e-9acc-2c8408581d52\" (UID: \"84569698-cff8-449e-9acc-2c8408581d52\") " Oct 29 22:36:42.748350 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:36:42.748276 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84569698-cff8-449e-9acc-2c8408581d52-kube-api-access-mpdqq\" (UniqueName: \"kubernetes.io/projected/84569698-cff8-449e-9acc-2c8408581d52-kube-api-access-mpdqq\") pod \"84569698-cff8-449e-9acc-2c8408581d52\" (UID: \"84569698-cff8-449e-9acc-2c8408581d52\") " Oct 29 22:36:42.748487 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:36:42.748447 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84569698-cff8-449e-9acc-2c8408581d52-archives-path" pod "84569698-cff8-449e-9acc-2c8408581d52" (UID: "84569698-cff8-449e-9acc-2c8408581d52"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:36:42.748548 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:36:42.748516 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84569698-cff8-449e-9acc-2c8408581d52-service-ca-bundle" pod "84569698-cff8-449e-9acc-2c8408581d52" (UID: "84569698-cff8-449e-9acc-2c8408581d52"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:36:42.748626 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:36:42.748609 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84569698-cff8-449e-9acc-2c8408581d52-trusted-ca-bundle" pod "84569698-cff8-449e-9acc-2c8408581d52" (UID: "84569698-cff8-449e-9acc-2c8408581d52"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:36:42.748661 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:36:42.748653 5057 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/84569698-cff8-449e-9acc-2c8408581d52-service-ca-bundle\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 22:36:42.748695 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:36:42.748666 5057 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/84569698-cff8-449e-9acc-2c8408581d52-archives-path\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 22:36:42.750464 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:36:42.750439 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84569698-cff8-449e-9acc-2c8408581d52-kube-api-access-mpdqq" pod "84569698-cff8-449e-9acc-2c8408581d52" (UID: "84569698-cff8-449e-9acc-2c8408581d52"). InnerVolumeSpecName "kube-api-access-mpdqq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:36:42.849520 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:36:42.849466 5057 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/84569698-cff8-449e-9acc-2c8408581d52-trusted-ca-bundle\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 22:36:42.849520 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:36:42.849489 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mpdqq\" (UniqueName: \"kubernetes.io/projected/84569698-cff8-449e-9acc-2c8408581d52-kube-api-access-mpdqq\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 22:36:43.569127 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:36:43.569091 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-ktp5j-2qdc5" Oct 29 22:36:43.569291 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:36:43.569118 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-ktp5j-2qdc5" event={"ID":"84569698-cff8-449e-9acc-2c8408581d52","Type":"ContainerDied","Data":"8bf2b9d5a0188dc1cfc3f93bf266d881b6b2d307616bb6a6c859188aaf9622e1"} Oct 29 22:36:43.569291 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:36:43.569174 5057 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8bf2b9d5a0188dc1cfc3f93bf266d881b6b2d307616bb6a6c859188aaf9622e1" Oct 29 22:36:43.891293 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:36:43.891260 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-9bbcb6fd8-w6wx5" Oct 29 22:36:43.891692 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:36:43.891318 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-9bbcb6fd8-w6wx5" Oct 29 22:36:43.896254 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:36:43.896226 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-9bbcb6fd8-w6wx5" Oct 29 22:36:44.527964 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:36:44.527930 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:36:44.578788 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:36:44.578757 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-9bbcb6fd8-w6wx5" Oct 29 22:36:44.647505 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:36:44.647466 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:36:44.648904 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:36:44.648876 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-765d49f595-55hx9"] Oct 29 22:36:49.026157 ip-10-0-140-74 kubenswrapper[5057]: E1029 22:36:49.026120 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d2dnn" containerName="extractor" Oct 29 22:37:09.676154 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:37:09.676110 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-765d49f595-55hx9" podUID="e68006a8-7278-43c2-ba05-1fe673fe6c70" containerName="console" containerID="cri-o://b6866cc7d34b9eaea771d015cff2f4191184c7bccf692fca5c970a44db2df792" gracePeriod=15 Oct 29 22:37:09.933941 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:37:09.933887 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-765d49f595-55hx9" Oct 29 22:37:10.097447 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:37:10.097418 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e68006a8-7278-43c2-ba05-1fe673fe6c70-service-ca\" (UniqueName: \"kubernetes.io/configmap/e68006a8-7278-43c2-ba05-1fe673fe6c70-service-ca\") pod \"e68006a8-7278-43c2-ba05-1fe673fe6c70\" (UID: \"e68006a8-7278-43c2-ba05-1fe673fe6c70\") " Oct 29 22:37:10.097599 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:37:10.097483 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e68006a8-7278-43c2-ba05-1fe673fe6c70-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e68006a8-7278-43c2-ba05-1fe673fe6c70-console-oauth-config\") pod \"e68006a8-7278-43c2-ba05-1fe673fe6c70\" (UID: \"e68006a8-7278-43c2-ba05-1fe673fe6c70\") " Oct 29 22:37:10.097599 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:37:10.097524 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e68006a8-7278-43c2-ba05-1fe673fe6c70-console-config\" (UniqueName: \"kubernetes.io/configmap/e68006a8-7278-43c2-ba05-1fe673fe6c70-console-config\") pod \"e68006a8-7278-43c2-ba05-1fe673fe6c70\" (UID: \"e68006a8-7278-43c2-ba05-1fe673fe6c70\") " Oct 29 22:37:10.097599 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:37:10.097548 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e68006a8-7278-43c2-ba05-1fe673fe6c70-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e68006a8-7278-43c2-ba05-1fe673fe6c70-trusted-ca-bundle\") pod \"e68006a8-7278-43c2-ba05-1fe673fe6c70\" (UID: \"e68006a8-7278-43c2-ba05-1fe673fe6c70\") " Oct 29 22:37:10.097720 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:37:10.097638 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e68006a8-7278-43c2-ba05-1fe673fe6c70-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e68006a8-7278-43c2-ba05-1fe673fe6c70-oauth-serving-cert\") pod \"e68006a8-7278-43c2-ba05-1fe673fe6c70\" (UID: \"e68006a8-7278-43c2-ba05-1fe673fe6c70\") " Oct 29 22:37:10.097720 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:37:10.097681 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e68006a8-7278-43c2-ba05-1fe673fe6c70-tmp\" (UniqueName: \"kubernetes.io/empty-dir/e68006a8-7278-43c2-ba05-1fe673fe6c70-tmp\") pod \"e68006a8-7278-43c2-ba05-1fe673fe6c70\" (UID: \"e68006a8-7278-43c2-ba05-1fe673fe6c70\") " Oct 29 22:37:10.097720 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:37:10.097715 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e68006a8-7278-43c2-ba05-1fe673fe6c70-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e68006a8-7278-43c2-ba05-1fe673fe6c70-console-serving-cert\") pod \"e68006a8-7278-43c2-ba05-1fe673fe6c70\" (UID: \"e68006a8-7278-43c2-ba05-1fe673fe6c70\") " Oct 29 22:37:10.097869 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:37:10.097747 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e68006a8-7278-43c2-ba05-1fe673fe6c70-kube-api-access-8vqsg\" (UniqueName: \"kubernetes.io/projected/e68006a8-7278-43c2-ba05-1fe673fe6c70-kube-api-access-8vqsg\") pod \"e68006a8-7278-43c2-ba05-1fe673fe6c70\" (UID: \"e68006a8-7278-43c2-ba05-1fe673fe6c70\") " Oct 29 22:37:10.098060 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:37:10.097880 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e68006a8-7278-43c2-ba05-1fe673fe6c70-service-ca" pod "e68006a8-7278-43c2-ba05-1fe673fe6c70" (UID: "e68006a8-7278-43c2-ba05-1fe673fe6c70"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:37:10.098127 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:37:10.098000 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e68006a8-7278-43c2-ba05-1fe673fe6c70-tmp" pod "e68006a8-7278-43c2-ba05-1fe673fe6c70" (UID: "e68006a8-7278-43c2-ba05-1fe673fe6c70"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:37:10.098127 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:37:10.098054 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e68006a8-7278-43c2-ba05-1fe673fe6c70-oauth-serving-cert" pod "e68006a8-7278-43c2-ba05-1fe673fe6c70" (UID: "e68006a8-7278-43c2-ba05-1fe673fe6c70"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:37:10.098127 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:37:10.098068 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e68006a8-7278-43c2-ba05-1fe673fe6c70-trusted-ca-bundle" pod "e68006a8-7278-43c2-ba05-1fe673fe6c70" (UID: "e68006a8-7278-43c2-ba05-1fe673fe6c70"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:37:10.098127 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:37:10.098014 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e68006a8-7278-43c2-ba05-1fe673fe6c70-console-config" pod "e68006a8-7278-43c2-ba05-1fe673fe6c70" (UID: "e68006a8-7278-43c2-ba05-1fe673fe6c70"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:37:10.099703 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:37:10.099682 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e68006a8-7278-43c2-ba05-1fe673fe6c70-console-oauth-config" pod "e68006a8-7278-43c2-ba05-1fe673fe6c70" (UID: "e68006a8-7278-43c2-ba05-1fe673fe6c70"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:37:10.099791 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:37:10.099714 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e68006a8-7278-43c2-ba05-1fe673fe6c70-console-serving-cert" pod "e68006a8-7278-43c2-ba05-1fe673fe6c70" (UID: "e68006a8-7278-43c2-ba05-1fe673fe6c70"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:37:10.099933 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:37:10.099918 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e68006a8-7278-43c2-ba05-1fe673fe6c70-kube-api-access-8vqsg" pod "e68006a8-7278-43c2-ba05-1fe673fe6c70" (UID: "e68006a8-7278-43c2-ba05-1fe673fe6c70"). InnerVolumeSpecName "kube-api-access-8vqsg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:37:10.198599 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:37:10.198571 5057 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e68006a8-7278-43c2-ba05-1fe673fe6c70-oauth-serving-cert\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 22:37:10.198599 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:37:10.198597 5057 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e68006a8-7278-43c2-ba05-1fe673fe6c70-tmp\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 22:37:10.198829 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:37:10.198609 5057 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e68006a8-7278-43c2-ba05-1fe673fe6c70-console-serving-cert\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 22:37:10.198829 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:37:10.198624 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8vqsg\" (UniqueName: \"kubernetes.io/projected/e68006a8-7278-43c2-ba05-1fe673fe6c70-kube-api-access-8vqsg\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 22:37:10.198829 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:37:10.198638 5057 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e68006a8-7278-43c2-ba05-1fe673fe6c70-service-ca\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 22:37:10.198829 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:37:10.198650 5057 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e68006a8-7278-43c2-ba05-1fe673fe6c70-console-oauth-config\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 22:37:10.198829 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:37:10.198661 5057 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e68006a8-7278-43c2-ba05-1fe673fe6c70-console-config\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 22:37:10.198829 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:37:10.198674 5057 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e68006a8-7278-43c2-ba05-1fe673fe6c70-trusted-ca-bundle\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 22:37:10.695298 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:37:10.695214 5057 generic.go:356] "Generic (PLEG): container finished" podID="e68006a8-7278-43c2-ba05-1fe673fe6c70" containerID="b6866cc7d34b9eaea771d015cff2f4191184c7bccf692fca5c970a44db2df792" exitCode=0 Oct 29 22:37:10.695298 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:37:10.695263 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-765d49f595-55hx9" Oct 29 22:37:10.695748 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:37:10.695293 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-765d49f595-55hx9" event={"ID":"e68006a8-7278-43c2-ba05-1fe673fe6c70","Type":"ContainerDied","Data":"b6866cc7d34b9eaea771d015cff2f4191184c7bccf692fca5c970a44db2df792"} Oct 29 22:37:10.695748 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:37:10.695353 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-765d49f595-55hx9" event={"ID":"e68006a8-7278-43c2-ba05-1fe673fe6c70","Type":"ContainerDied","Data":"8048cf004232ff95dae3eb83f15c1b76aa731c52589ab3142e56b4d1aa496cf3"} Oct 29 22:37:10.695748 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:37:10.695375 5057 scope.go:122] "RemoveContainer" containerID="b6866cc7d34b9eaea771d015cff2f4191184c7bccf692fca5c970a44db2df792" Oct 29 22:37:10.704651 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:37:10.704633 5057 scope.go:122] "RemoveContainer" containerID="b6866cc7d34b9eaea771d015cff2f4191184c7bccf692fca5c970a44db2df792" Oct 29 22:37:10.704912 ip-10-0-140-74 kubenswrapper[5057]: E1029 22:37:10.704894 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6866cc7d34b9eaea771d015cff2f4191184c7bccf692fca5c970a44db2df792\": container with ID starting with b6866cc7d34b9eaea771d015cff2f4191184c7bccf692fca5c970a44db2df792 not found: ID does not exist" containerID="b6866cc7d34b9eaea771d015cff2f4191184c7bccf692fca5c970a44db2df792" Oct 29 22:37:10.704950 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:37:10.704921 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6866cc7d34b9eaea771d015cff2f4191184c7bccf692fca5c970a44db2df792"} err="failed to get container status \"b6866cc7d34b9eaea771d015cff2f4191184c7bccf692fca5c970a44db2df792\": rpc error: code = NotFound desc = could not find container \"b6866cc7d34b9eaea771d015cff2f4191184c7bccf692fca5c970a44db2df792\": container with ID starting with b6866cc7d34b9eaea771d015cff2f4191184c7bccf692fca5c970a44db2df792 not found: ID does not exist" Oct 29 22:37:10.741563 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:37:10.741540 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-765d49f595-55hx9"] Oct 29 22:37:10.751953 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:37:10.751932 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-765d49f595-55hx9"] Oct 29 22:37:11.326010 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:37:11.325976 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e68006a8-7278-43c2-ba05-1fe673fe6c70" path="/var/lib/kubelet/pods/e68006a8-7278-43c2-ba05-1fe673fe6c70/volumes" Oct 29 22:37:14.721410 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:37:14.721378 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:37:19.026732 ip-10-0-140-74 kubenswrapper[5057]: E1029 22:37:19.026694 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d2dnn" containerName="extractor" Oct 29 22:37:29.128170 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:37:29.128136 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:37:49.026160 ip-10-0-140-74 kubenswrapper[5057]: E1029 22:37:49.026116 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d2dnn" containerName="extractor" Oct 29 22:38:19.026458 ip-10-0-140-74 kubenswrapper[5057]: E1029 22:38:19.026420 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d2dnn" containerName="extractor" Oct 29 22:38:29.250928 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:38:29.250888 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:38:39.361982 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:38:39.361957 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-vj6md_aa6a190e-3283-458a-9dfd-2de7cb193fc5/console-operator/2.log" Oct 29 22:38:39.365197 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:38:39.365172 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-vj6md_aa6a190e-3283-458a-9dfd-2de7cb193fc5/console-operator/2.log" Oct 29 22:38:49.025942 ip-10-0-140-74 kubenswrapper[5057]: E1029 22:38:49.025904 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d2dnn" containerName="extractor" Oct 29 22:39:15.060553 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:39:15.060476 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:39:16.241594 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:39:16.241554 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:39:16.347963 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:39:16.347930 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:39:19.026373 ip-10-0-140-74 kubenswrapper[5057]: E1029 22:39:19.026318 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d2dnn" containerName="extractor" Oct 29 22:39:19.193985 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:39:19.193946 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:39:19.311431 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:39:19.311349 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:39:29.373703 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:39:29.373657 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:39:44.204654 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:39:44.204613 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:39:49.025758 ip-10-0-140-74 kubenswrapper[5057]: E1029 22:39:49.025725 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d2dnn" containerName="extractor" Oct 29 22:40:19.026450 ip-10-0-140-74 kubenswrapper[5057]: E1029 22:40:19.026412 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d2dnn" containerName="extractor" Oct 29 22:40:29.502854 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:40:29.502817 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:40:38.088699 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:40:38.088644 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:40:38.197633 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:40:38.197595 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:40:49.025971 ip-10-0-140-74 kubenswrapper[5057]: E1029 22:40:49.025934 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d2dnn" containerName="extractor" Oct 29 22:41:19.026382 ip-10-0-140-74 kubenswrapper[5057]: E1029 22:41:19.026346 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d2dnn" containerName="extractor" Oct 29 22:41:29.636044 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:41:29.636009 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:41:46.768700 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:41:46.768664 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:41:46.894372 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:41:46.894310 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:41:49.026155 ip-10-0-140-74 kubenswrapper[5057]: E1029 22:41:49.026116 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d2dnn" containerName="extractor" Oct 29 22:41:56.978248 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:41:56.978160 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:41:57.098707 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:41:57.098669 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:42:05.193080 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:42:05.193044 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:42:05.442043 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:42:05.442006 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:42:05.552615 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:42:05.552585 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:42:06.131354 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:42:06.131297 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:42:16.791697 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:42:16.791657 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:42:16.917200 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:42:16.917161 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:42:19.030519 ip-10-0-140-74 kubenswrapper[5057]: E1029 22:42:19.030472 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d2dnn" containerName="extractor" Oct 29 22:42:29.759075 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:42:29.759042 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:42:31.366122 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:42:31.366085 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:42:42.217071 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:42:42.217021 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:42:42.330839 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:42:42.330803 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:42:48.352902 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:42:48.352866 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:42:49.026881 ip-10-0-140-74 kubenswrapper[5057]: E1029 22:42:49.026845 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d2dnn" containerName="extractor" Oct 29 22:43:19.026340 ip-10-0-140-74 kubenswrapper[5057]: E1029 22:43:19.026287 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d2dnn" containerName="extractor" Oct 29 22:43:29.892282 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:43:29.892249 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:43:39.399320 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:43:39.399294 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-vj6md_aa6a190e-3283-458a-9dfd-2de7cb193fc5/console-operator/2.log" Oct 29 22:43:39.402869 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:43:39.402848 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-vj6md_aa6a190e-3283-458a-9dfd-2de7cb193fc5/console-operator/2.log" Oct 29 22:43:49.026972 ip-10-0-140-74 kubenswrapper[5057]: E1029 22:43:49.026938 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d2dnn" containerName="extractor" Oct 29 22:43:52.324553 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:43:52.324514 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:44:19.026399 ip-10-0-140-74 kubenswrapper[5057]: E1029 22:44:19.026359 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d2dnn" containerName="extractor" Oct 29 22:44:30.030814 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:44:30.030777 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:44:33.368886 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:44:33.368849 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:44:34.824869 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:44:34.824828 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:44:34.961115 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:44:34.961077 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:44:36.218827 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:44:36.218797 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:44:38.788399 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:44:38.788361 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:44:38.897152 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:44:38.897118 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:44:40.053922 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:44:40.053887 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:44:40.200459 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:44:40.200424 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:44:40.312290 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:44:40.312209 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:44:43.163191 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:44:43.163159 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:44:49.026356 ip-10-0-140-74 kubenswrapper[5057]: E1029 22:44:49.026304 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d2dnn" containerName="extractor" Oct 29 22:44:51.769836 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:44:51.769756 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:45:14.893547 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:45:14.893510 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:45:15.077006 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:45:15.076974 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:45:15.563203 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:45:15.563175 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:45:15.733562 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:45:15.733522 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-855c4cdcb6-s4nbq"] Oct 29 22:45:15.734049 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:45:15.734022 5057 cpu_manager.go:414] "containerMap: removing container" podUID="e68006a8-7278-43c2-ba05-1fe673fe6c70" containerName="console" Oct 29 22:45:15.734049 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:45:15.734043 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e68006a8-7278-43c2-ba05-1fe673fe6c70" containerName="console" Oct 29 22:45:15.734234 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:45:15.734064 5057 cpu_manager.go:414] "containerMap: removing container" podUID="84569698-cff8-449e-9acc-2c8408581d52" containerName="insights-gathering" Oct 29 22:45:15.734234 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:45:15.734073 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84569698-cff8-449e-9acc-2c8408581d52" containerName="insights-gathering" Oct 29 22:45:15.734388 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:45:15.734368 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="e68006a8-7278-43c2-ba05-1fe673fe6c70" containerName="console" Oct 29 22:45:15.734486 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:45:15.734390 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="84569698-cff8-449e-9acc-2c8408581d52" containerName="insights-gathering" Oct 29 22:45:15.738029 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:45:15.738007 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-855c4cdcb6-s4nbq" Oct 29 22:45:15.767283 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:45:15.767256 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-855c4cdcb6-s4nbq"] Oct 29 22:45:15.814798 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:45:15.814708 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Oct 29 22:45:15.814798 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:45:15.814726 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Oct 29 22:45:15.814798 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:45:15.814709 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Oct 29 22:45:15.814798 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:45:15.814762 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-6q6mp\"" Oct 29 22:45:15.814798 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:45:15.814797 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Oct 29 22:45:15.815045 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:45:15.814802 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Oct 29 22:45:15.869532 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:45:15.869504 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/8541fa6d-cf09-4237-9a0d-7614c40099eb-metrics-cert\") pod \"opendatahub-operator-controller-manager-855c4cdcb6-s4nbq\" (UID: \"8541fa6d-cf09-4237-9a0d-7614c40099eb\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-855c4cdcb6-s4nbq" Oct 29 22:45:15.869675 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:45:15.869539 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8541fa6d-cf09-4237-9a0d-7614c40099eb-tmp\") pod \"opendatahub-operator-controller-manager-855c4cdcb6-s4nbq\" (UID: \"8541fa6d-cf09-4237-9a0d-7614c40099eb\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-855c4cdcb6-s4nbq" Oct 29 22:45:15.869675 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:45:15.869567 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/8541fa6d-cf09-4237-9a0d-7614c40099eb-manifests\") pod \"opendatahub-operator-controller-manager-855c4cdcb6-s4nbq\" (UID: \"8541fa6d-cf09-4237-9a0d-7614c40099eb\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-855c4cdcb6-s4nbq" Oct 29 22:45:15.869746 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:45:15.869720 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/8541fa6d-cf09-4237-9a0d-7614c40099eb-apiservice-cert\") pod \"opendatahub-operator-controller-manager-855c4cdcb6-s4nbq\" (UID: \"8541fa6d-cf09-4237-9a0d-7614c40099eb\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-855c4cdcb6-s4nbq" Oct 29 22:45:15.869783 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:45:15.869763 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfbc6\" (UniqueName: \"kubernetes.io/projected/8541fa6d-cf09-4237-9a0d-7614c40099eb-kube-api-access-vfbc6\") pod \"opendatahub-operator-controller-manager-855c4cdcb6-s4nbq\" (UID: \"8541fa6d-cf09-4237-9a0d-7614c40099eb\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-855c4cdcb6-s4nbq" Oct 29 22:45:15.869818 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:45:15.869786 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8541fa6d-cf09-4237-9a0d-7614c40099eb-webhook-cert\") pod \"opendatahub-operator-controller-manager-855c4cdcb6-s4nbq\" (UID: \"8541fa6d-cf09-4237-9a0d-7614c40099eb\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-855c4cdcb6-s4nbq" Oct 29 22:45:15.892740 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:45:15.892710 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:45:15.970945 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:45:15.970904 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/8541fa6d-cf09-4237-9a0d-7614c40099eb-apiservice-cert\") pod \"opendatahub-operator-controller-manager-855c4cdcb6-s4nbq\" (UID: \"8541fa6d-cf09-4237-9a0d-7614c40099eb\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-855c4cdcb6-s4nbq" Oct 29 22:45:15.971454 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:45:15.970963 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfbc6\" (UniqueName: \"kubernetes.io/projected/8541fa6d-cf09-4237-9a0d-7614c40099eb-kube-api-access-vfbc6\") pod \"opendatahub-operator-controller-manager-855c4cdcb6-s4nbq\" (UID: \"8541fa6d-cf09-4237-9a0d-7614c40099eb\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-855c4cdcb6-s4nbq" Oct 29 22:45:15.971454 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:45:15.970991 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8541fa6d-cf09-4237-9a0d-7614c40099eb-webhook-cert\") pod \"opendatahub-operator-controller-manager-855c4cdcb6-s4nbq\" (UID: \"8541fa6d-cf09-4237-9a0d-7614c40099eb\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-855c4cdcb6-s4nbq" Oct 29 22:45:15.971454 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:45:15.971059 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/8541fa6d-cf09-4237-9a0d-7614c40099eb-metrics-cert\") pod \"opendatahub-operator-controller-manager-855c4cdcb6-s4nbq\" (UID: \"8541fa6d-cf09-4237-9a0d-7614c40099eb\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-855c4cdcb6-s4nbq" Oct 29 22:45:15.971454 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:45:15.971094 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8541fa6d-cf09-4237-9a0d-7614c40099eb-tmp\") pod \"opendatahub-operator-controller-manager-855c4cdcb6-s4nbq\" (UID: \"8541fa6d-cf09-4237-9a0d-7614c40099eb\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-855c4cdcb6-s4nbq" Oct 29 22:45:15.971454 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:45:15.971132 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/8541fa6d-cf09-4237-9a0d-7614c40099eb-manifests\") pod \"opendatahub-operator-controller-manager-855c4cdcb6-s4nbq\" (UID: \"8541fa6d-cf09-4237-9a0d-7614c40099eb\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-855c4cdcb6-s4nbq" Oct 29 22:45:15.971716 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:45:15.971588 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/8541fa6d-cf09-4237-9a0d-7614c40099eb-manifests\") pod \"opendatahub-operator-controller-manager-855c4cdcb6-s4nbq\" (UID: \"8541fa6d-cf09-4237-9a0d-7614c40099eb\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-855c4cdcb6-s4nbq" Oct 29 22:45:15.973346 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:45:15.973309 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8541fa6d-cf09-4237-9a0d-7614c40099eb-tmp\") pod \"opendatahub-operator-controller-manager-855c4cdcb6-s4nbq\" (UID: \"8541fa6d-cf09-4237-9a0d-7614c40099eb\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-855c4cdcb6-s4nbq" Oct 29 22:45:15.973905 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:45:15.973882 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/8541fa6d-cf09-4237-9a0d-7614c40099eb-apiservice-cert\") pod \"opendatahub-operator-controller-manager-855c4cdcb6-s4nbq\" (UID: \"8541fa6d-cf09-4237-9a0d-7614c40099eb\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-855c4cdcb6-s4nbq" Oct 29 22:45:15.973990 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:45:15.973906 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/8541fa6d-cf09-4237-9a0d-7614c40099eb-metrics-cert\") pod \"opendatahub-operator-controller-manager-855c4cdcb6-s4nbq\" (UID: \"8541fa6d-cf09-4237-9a0d-7614c40099eb\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-855c4cdcb6-s4nbq" Oct 29 22:45:15.974087 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:45:15.974068 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8541fa6d-cf09-4237-9a0d-7614c40099eb-webhook-cert\") pod \"opendatahub-operator-controller-manager-855c4cdcb6-s4nbq\" (UID: \"8541fa6d-cf09-4237-9a0d-7614c40099eb\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-855c4cdcb6-s4nbq" Oct 29 22:45:15.995625 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:45:15.995597 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfbc6\" (UniqueName: \"kubernetes.io/projected/8541fa6d-cf09-4237-9a0d-7614c40099eb-kube-api-access-vfbc6\") pod \"opendatahub-operator-controller-manager-855c4cdcb6-s4nbq\" (UID: \"8541fa6d-cf09-4237-9a0d-7614c40099eb\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-855c4cdcb6-s4nbq" Oct 29 22:45:16.028575 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:45:16.028541 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:45:16.060044 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:45:16.060008 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-855c4cdcb6-s4nbq" Oct 29 22:45:16.147951 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:45:16.147919 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:45:16.417597 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:45:16.417559 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-855c4cdcb6-s4nbq"] Oct 29 22:45:16.419739 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:45:16.419711 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8541fa6d_cf09_4237_9a0d_7614c40099eb.slice/crio-908e6d284b3f7456dfee7917b33c067e2c6e01a8e4726f325211e28123df42d2 WatchSource:0}: Error finding container 908e6d284b3f7456dfee7917b33c067e2c6e01a8e4726f325211e28123df42d2: Status 404 returned error can't find the container with id 908e6d284b3f7456dfee7917b33c067e2c6e01a8e4726f325211e28123df42d2 Oct 29 22:45:16.421257 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:45:16.421240 5057 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 29 22:45:16.701677 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:45:16.701643 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:45:16.818504 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:45:16.818464 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:45:16.995735 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:45:16.995646 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-855c4cdcb6-s4nbq" event={"ID":"8541fa6d-cf09-4237-9a0d-7614c40099eb","Type":"ContainerStarted","Data":"908e6d284b3f7456dfee7917b33c067e2c6e01a8e4726f325211e28123df42d2"} Oct 29 22:45:18.694204 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:45:18.694171 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:45:18.834030 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:45:18.833997 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:45:19.031240 ip-10-0-140-74 kubenswrapper[5057]: E1029 22:45:19.031201 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d2dnn" containerName="extractor" Oct 29 22:45:19.096062 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:45:19.096028 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:45:19.220374 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:45:19.220319 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:45:20.013179 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:45:20.013142 5057 generic.go:356] "Generic (PLEG): container finished" podID="8541fa6d-cf09-4237-9a0d-7614c40099eb" containerID="42670f7c6d6ef77aaa8e187fa3691162101d68264106e3f16d1ef53e4843f576" exitCode=0 Oct 29 22:45:20.013573 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:45:20.013191 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-855c4cdcb6-s4nbq" event={"ID":"8541fa6d-cf09-4237-9a0d-7614c40099eb","Type":"ContainerDied","Data":"42670f7c6d6ef77aaa8e187fa3691162101d68264106e3f16d1ef53e4843f576"} Oct 29 22:45:21.019554 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:45:21.019518 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-855c4cdcb6-s4nbq" event={"ID":"8541fa6d-cf09-4237-9a0d-7614c40099eb","Type":"ContainerStarted","Data":"a259f590f04f83beed16d4a8d5f0abf876e04fbf6674644c213b93b6c5fd97cc"} Oct 29 22:45:21.019942 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:45:21.019579 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-855c4cdcb6-s4nbq" Oct 29 22:45:21.100033 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:45:21.099984 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-855c4cdcb6-s4nbq" podStartSLOduration=2.50270314 podStartE2EDuration="6.099971249s" totalImagesPullingTime="3.097278155s" totalInitContainerRuntime="499.989954ms" isStatefulPod=false podCreationTimestamp="2026-10-29 22:45:15 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:45:21.089735706 +0000 UTC m=+701.872949131" watchObservedRunningTime="2026-10-29 22:45:21.099971249 +0000 UTC m=+701.883184653" Oct 29 22:45:23.473573 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:45:23.473531 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:45:28.009636 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:45:28.009598 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:45:29.490447 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:45:29.490379 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:45:29.612796 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:45:29.612760 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:45:30.154820 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:45:30.154786 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:45:31.329685 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:45:31.329646 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:45:31.473738 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:45:31.473708 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:45:31.893878 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:45:31.893842 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:45:32.027389 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:45:32.027365 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-855c4cdcb6-s4nbq" Oct 29 22:45:32.922201 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:45:32.922166 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:45:33.850611 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:45:33.850577 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:45:34.021820 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:45:34.021786 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:45:34.197806 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:45:34.197774 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:45:39.096929 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:45:39.096895 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:45:39.221768 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:45:39.221732 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:45:40.876598 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:45:40.876563 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:45:41.013654 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:45:41.013624 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:45:41.130188 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:45:41.130107 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:45:41.340679 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:45:41.340650 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:45:41.658986 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:45:41.658954 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:45:42.739056 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:45:42.739022 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:45:42.878347 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:45:42.878305 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:45:43.873979 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:45:43.873941 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:45:44.673156 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:45:44.673125 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:45:45.451114 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:45:45.451072 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:45:45.639082 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:45:45.639053 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:45:45.773382 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:45:45.773307 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:45:45.899393 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:45:45.899362 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:45:46.128843 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:45:46.128764 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:45:46.253814 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:45:46.253779 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:45:46.397581 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:45:46.397495 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:45:46.511039 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:45:46.510999 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:45:47.889540 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:45:47.889502 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:45:48.550206 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:45:48.550174 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:45:48.947383 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:45:48.947346 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:45:49.026794 ip-10-0-140-74 kubenswrapper[5057]: E1029 22:45:49.026761 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d2dnn" containerName="extractor" Oct 29 22:45:51.129639 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:45:51.129604 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:45:53.335590 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:45:53.335517 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:45:53.446831 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:45:53.446801 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:45:54.291913 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:45:54.291874 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:45:54.401210 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:45:54.401172 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:45:57.775786 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:45:57.775753 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-2hpht"] Oct 29 22:45:57.779558 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:45:57.779541 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-2hpht" Oct 29 22:45:57.804729 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:45:57.804706 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Oct 29 22:45:57.804874 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:45:57.804791 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Oct 29 22:45:57.804874 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:45:57.804804 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-wfqwf\"" Oct 29 22:45:57.806756 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:45:57.806738 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-2hpht"] Oct 29 22:45:57.855450 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:45:57.855420 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/bd4e5443-0ff9-4a71-b6d1-bbddbbe457ab-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-2hpht\" (UID: \"bd4e5443-0ff9-4a71-b6d1-bbddbbe457ab\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-2hpht" Oct 29 22:45:57.855557 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:45:57.855475 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xv2nh\" (UniqueName: \"kubernetes.io/projected/bd4e5443-0ff9-4a71-b6d1-bbddbbe457ab-kube-api-access-xv2nh\") pod \"kuadrant-operator-controller-manager-678d87766f-2hpht\" (UID: \"bd4e5443-0ff9-4a71-b6d1-bbddbbe457ab\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-2hpht" Oct 29 22:45:57.956319 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:45:57.956290 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/bd4e5443-0ff9-4a71-b6d1-bbddbbe457ab-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-2hpht\" (UID: \"bd4e5443-0ff9-4a71-b6d1-bbddbbe457ab\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-2hpht" Oct 29 22:45:57.956499 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:45:57.956353 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xv2nh\" (UniqueName: \"kubernetes.io/projected/bd4e5443-0ff9-4a71-b6d1-bbddbbe457ab-kube-api-access-xv2nh\") pod \"kuadrant-operator-controller-manager-678d87766f-2hpht\" (UID: \"bd4e5443-0ff9-4a71-b6d1-bbddbbe457ab\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-2hpht" Oct 29 22:45:57.956677 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:45:57.956655 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/bd4e5443-0ff9-4a71-b6d1-bbddbbe457ab-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-2hpht\" (UID: \"bd4e5443-0ff9-4a71-b6d1-bbddbbe457ab\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-2hpht" Oct 29 22:45:57.978138 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:45:57.978103 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xv2nh\" (UniqueName: \"kubernetes.io/projected/bd4e5443-0ff9-4a71-b6d1-bbddbbe457ab-kube-api-access-xv2nh\") pod \"kuadrant-operator-controller-manager-678d87766f-2hpht\" (UID: \"bd4e5443-0ff9-4a71-b6d1-bbddbbe457ab\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-2hpht" Oct 29 22:45:58.091897 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:45:58.091833 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-2hpht" Oct 29 22:45:58.415487 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:45:58.415464 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-2hpht"] Oct 29 22:45:58.418677 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:45:58.418652 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd4e5443_0ff9_4a71_b6d1_bbddbbe457ab.slice/crio-e9c6a485d0501f521b62c24aade71617deff69b6c6c78b4038f8ab85827e3671 WatchSource:0}: Error finding container e9c6a485d0501f521b62c24aade71617deff69b6c6c78b4038f8ab85827e3671: Status 404 returned error can't find the container with id e9c6a485d0501f521b62c24aade71617deff69b6c6c78b4038f8ab85827e3671 Oct 29 22:45:58.743187 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:45:58.743157 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:45:58.850177 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:45:58.850132 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:45:59.213613 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:45:59.213573 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-2hpht" event={"ID":"bd4e5443-0ff9-4a71-b6d1-bbddbbe457ab","Type":"ContainerStarted","Data":"e9c6a485d0501f521b62c24aade71617deff69b6c6c78b4038f8ab85827e3671"} Oct 29 22:46:01.697580 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:46:01.697546 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:46:01.812972 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:46:01.812931 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:46:02.231196 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:46:02.231163 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-2hpht" event={"ID":"bd4e5443-0ff9-4a71-b6d1-bbddbbe457ab","Type":"ContainerStarted","Data":"c9ebfa8232ab31d7a2977e7e6d6d43497bf212aa70531009c0212e0951737031"} Oct 29 22:46:02.231386 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:46:02.231352 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-2hpht" Oct 29 22:46:02.294963 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:46:02.294904 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-2hpht" podStartSLOduration=1.608186816 podStartE2EDuration="5.294890022s" totalImagesPullingTime="3.686703206s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:45:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:46:02.282556557 +0000 UTC m=+743.065769983" watchObservedRunningTime="2026-10-29 22:46:02.294890022 +0000 UTC m=+743.078103448" Oct 29 22:46:03.634437 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:46:03.634378 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:46:06.502962 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:46:06.502926 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:46:06.616725 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:46:06.616695 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:46:07.207178 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:46:07.207103 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:46:07.322046 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:46:07.322011 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:46:07.897552 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:46:07.897517 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-56c4d546f5-wmm4k"] Oct 29 22:46:07.901355 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:46:07.901320 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-56c4d546f5-wmm4k" Oct 29 22:46:07.919956 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:46:07.919934 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-56c4d546f5-wmm4k"] Oct 29 22:46:07.928406 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:46:07.928383 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"jobset-manager-config\"" Oct 29 22:46:07.928530 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:46:07.928443 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\"" Oct 29 22:46:07.928530 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:46:07.928481 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\"" Oct 29 22:46:07.928530 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:46:07.928519 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-controller-manager-dockercfg-cggb5\"" Oct 29 22:46:07.928717 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:46:07.928575 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"metrics-server-cert\"" Oct 29 22:46:07.928717 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:46:07.928605 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"webhook-server-cert\"" Oct 29 22:46:08.057678 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:46:08.057636 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a5dc510-fbac-402e-8dd2-337e51cfee0d-metrics-certs\") pod \"jobset-controller-manager-56c4d546f5-wmm4k\" (UID: \"9a5dc510-fbac-402e-8dd2-337e51cfee0d\") " pod="openshift-jobset-operator/jobset-controller-manager-56c4d546f5-wmm4k" Oct 29 22:46:08.057678 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:46:08.057689 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/9a5dc510-fbac-402e-8dd2-337e51cfee0d-manager-config\") pod \"jobset-controller-manager-56c4d546f5-wmm4k\" (UID: \"9a5dc510-fbac-402e-8dd2-337e51cfee0d\") " pod="openshift-jobset-operator/jobset-controller-manager-56c4d546f5-wmm4k" Oct 29 22:46:08.057944 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:46:08.057825 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9a5dc510-fbac-402e-8dd2-337e51cfee0d-cert\") pod \"jobset-controller-manager-56c4d546f5-wmm4k\" (UID: \"9a5dc510-fbac-402e-8dd2-337e51cfee0d\") " pod="openshift-jobset-operator/jobset-controller-manager-56c4d546f5-wmm4k" Oct 29 22:46:08.057944 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:46:08.057884 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6h4m\" (UniqueName: \"kubernetes.io/projected/9a5dc510-fbac-402e-8dd2-337e51cfee0d-kube-api-access-w6h4m\") pod \"jobset-controller-manager-56c4d546f5-wmm4k\" (UID: \"9a5dc510-fbac-402e-8dd2-337e51cfee0d\") " pod="openshift-jobset-operator/jobset-controller-manager-56c4d546f5-wmm4k" Oct 29 22:46:08.159239 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:46:08.159151 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/9a5dc510-fbac-402e-8dd2-337e51cfee0d-manager-config\") pod \"jobset-controller-manager-56c4d546f5-wmm4k\" (UID: \"9a5dc510-fbac-402e-8dd2-337e51cfee0d\") " pod="openshift-jobset-operator/jobset-controller-manager-56c4d546f5-wmm4k" Oct 29 22:46:08.159239 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:46:08.159211 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9a5dc510-fbac-402e-8dd2-337e51cfee0d-cert\") pod \"jobset-controller-manager-56c4d546f5-wmm4k\" (UID: \"9a5dc510-fbac-402e-8dd2-337e51cfee0d\") " pod="openshift-jobset-operator/jobset-controller-manager-56c4d546f5-wmm4k" Oct 29 22:46:08.159239 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:46:08.159242 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6h4m\" (UniqueName: \"kubernetes.io/projected/9a5dc510-fbac-402e-8dd2-337e51cfee0d-kube-api-access-w6h4m\") pod \"jobset-controller-manager-56c4d546f5-wmm4k\" (UID: \"9a5dc510-fbac-402e-8dd2-337e51cfee0d\") " pod="openshift-jobset-operator/jobset-controller-manager-56c4d546f5-wmm4k" Oct 29 22:46:08.159567 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:46:08.159290 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a5dc510-fbac-402e-8dd2-337e51cfee0d-metrics-certs\") pod \"jobset-controller-manager-56c4d546f5-wmm4k\" (UID: \"9a5dc510-fbac-402e-8dd2-337e51cfee0d\") " pod="openshift-jobset-operator/jobset-controller-manager-56c4d546f5-wmm4k" Oct 29 22:46:08.159964 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:46:08.159939 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/9a5dc510-fbac-402e-8dd2-337e51cfee0d-manager-config\") pod \"jobset-controller-manager-56c4d546f5-wmm4k\" (UID: \"9a5dc510-fbac-402e-8dd2-337e51cfee0d\") " pod="openshift-jobset-operator/jobset-controller-manager-56c4d546f5-wmm4k" Oct 29 22:46:08.161766 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:46:08.161747 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9a5dc510-fbac-402e-8dd2-337e51cfee0d-cert\") pod \"jobset-controller-manager-56c4d546f5-wmm4k\" (UID: \"9a5dc510-fbac-402e-8dd2-337e51cfee0d\") " pod="openshift-jobset-operator/jobset-controller-manager-56c4d546f5-wmm4k" Oct 29 22:46:08.161937 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:46:08.161918 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a5dc510-fbac-402e-8dd2-337e51cfee0d-metrics-certs\") pod \"jobset-controller-manager-56c4d546f5-wmm4k\" (UID: \"9a5dc510-fbac-402e-8dd2-337e51cfee0d\") " pod="openshift-jobset-operator/jobset-controller-manager-56c4d546f5-wmm4k" Oct 29 22:46:08.182046 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:46:08.182021 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6h4m\" (UniqueName: \"kubernetes.io/projected/9a5dc510-fbac-402e-8dd2-337e51cfee0d-kube-api-access-w6h4m\") pod \"jobset-controller-manager-56c4d546f5-wmm4k\" (UID: \"9a5dc510-fbac-402e-8dd2-337e51cfee0d\") " pod="openshift-jobset-operator/jobset-controller-manager-56c4d546f5-wmm4k" Oct 29 22:46:08.212772 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:46:08.212754 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-56c4d546f5-wmm4k" Oct 29 22:46:08.512642 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:46:08.512608 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-56c4d546f5-wmm4k"] Oct 29 22:46:08.515371 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:46:08.515346 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a5dc510_fbac_402e_8dd2_337e51cfee0d.slice/crio-47ce585786dc72bf6f2a5c770907fe27d16b66b1df8dec26de44aafd9beba8d0 WatchSource:0}: Error finding container 47ce585786dc72bf6f2a5c770907fe27d16b66b1df8dec26de44aafd9beba8d0: Status 404 returned error can't find the container with id 47ce585786dc72bf6f2a5c770907fe27d16b66b1df8dec26de44aafd9beba8d0 Oct 29 22:46:08.955916 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:46:08.955882 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:46:09.063199 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:46:09.063170 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:46:09.266978 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:46:09.266904 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-56c4d546f5-wmm4k" event={"ID":"9a5dc510-fbac-402e-8dd2-337e51cfee0d","Type":"ContainerStarted","Data":"47ce585786dc72bf6f2a5c770907fe27d16b66b1df8dec26de44aafd9beba8d0"} Oct 29 22:46:12.286604 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:46:12.286571 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-56c4d546f5-wmm4k" event={"ID":"9a5dc510-fbac-402e-8dd2-337e51cfee0d","Type":"ContainerStarted","Data":"945c8e479ba34cd6c622e71c596b04d9d5f15ac9e4e03d53551cb03db4658771"} Oct 29 22:46:12.287039 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:46:12.286697 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-jobset-operator/jobset-controller-manager-56c4d546f5-wmm4k" Oct 29 22:46:12.330159 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:46:12.327311 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-controller-manager-56c4d546f5-wmm4k" podStartSLOduration=2.46661429 podStartE2EDuration="5.327295504s" totalImagesPullingTime="2.860681214s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:46:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:46:12.323897494 +0000 UTC m=+753.107110931" watchObservedRunningTime="2026-10-29 22:46:12.327295504 +0000 UTC m=+753.110508931" Oct 29 22:46:13.239076 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:46:13.239041 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-2hpht" Oct 29 22:46:15.713280 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:46:15.713241 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:46:15.829347 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:46:15.829305 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:46:19.026866 ip-10-0-140-74 kubenswrapper[5057]: E1029 22:46:19.026815 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d2dnn" containerName="extractor" Oct 29 22:46:22.713071 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:46:22.712989 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:46:22.848009 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:46:22.847968 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:46:23.296880 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:46:23.296853 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-jobset-operator/jobset-controller-manager-56c4d546f5-wmm4k" Oct 29 22:46:24.095150 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:46:24.095113 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:46:29.009818 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:46:29.009785 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:46:29.136696 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:46:29.136665 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:46:29.927904 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:46:29.927860 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:46:30.054428 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:46:30.054399 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:46:30.277234 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:46:30.277198 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:46:33.491438 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:46:33.491397 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:46:33.701999 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:46:33.701967 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:46:33.815621 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:46:33.815531 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:46:34.505407 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:46:34.505371 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:46:34.624994 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:46:34.624957 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:46:34.964453 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:46:34.964419 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v2tfm"] Oct 29 22:46:35.021398 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:46:35.021369 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v2tfm"] Oct 29 22:46:35.021540 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:46:35.021456 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-v2tfm" Oct 29 22:46:35.034155 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:46:35.034134 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Oct 29 22:46:35.046124 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:46:35.046101 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v2tfm"] Oct 29 22:46:35.079165 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:46:35.079148 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-nrlgf\"" Oct 29 22:46:35.118628 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:46:35.118602 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/5e0f62f7-e32b-45f1-ba71-88bbb389039b-config-file\") pod \"limitador-limitador-7c9b87d574-v2tfm\" (UID: \"5e0f62f7-e32b-45f1-ba71-88bbb389039b\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-v2tfm" Oct 29 22:46:35.118748 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:46:35.118714 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggfjn\" (UniqueName: \"kubernetes.io/projected/5e0f62f7-e32b-45f1-ba71-88bbb389039b-kube-api-access-ggfjn\") pod \"limitador-limitador-7c9b87d574-v2tfm\" (UID: \"5e0f62f7-e32b-45f1-ba71-88bbb389039b\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-v2tfm" Oct 29 22:46:35.220023 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:46:35.219968 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/5e0f62f7-e32b-45f1-ba71-88bbb389039b-config-file\") pod \"limitador-limitador-7c9b87d574-v2tfm\" (UID: \"5e0f62f7-e32b-45f1-ba71-88bbb389039b\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-v2tfm" Oct 29 22:46:35.220119 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:46:35.220110 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggfjn\" (UniqueName: \"kubernetes.io/projected/5e0f62f7-e32b-45f1-ba71-88bbb389039b-kube-api-access-ggfjn\") pod \"limitador-limitador-7c9b87d574-v2tfm\" (UID: \"5e0f62f7-e32b-45f1-ba71-88bbb389039b\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-v2tfm" Oct 29 22:46:35.220622 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:46:35.220605 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/5e0f62f7-e32b-45f1-ba71-88bbb389039b-config-file\") pod \"limitador-limitador-7c9b87d574-v2tfm\" (UID: \"5e0f62f7-e32b-45f1-ba71-88bbb389039b\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-v2tfm" Oct 29 22:46:35.241720 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:46:35.241698 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggfjn\" (UniqueName: \"kubernetes.io/projected/5e0f62f7-e32b-45f1-ba71-88bbb389039b-kube-api-access-ggfjn\") pod \"limitador-limitador-7c9b87d574-v2tfm\" (UID: \"5e0f62f7-e32b-45f1-ba71-88bbb389039b\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-v2tfm" Oct 29 22:46:35.358502 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:46:35.358468 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-v2tfm" Oct 29 22:46:35.689870 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:46:35.689846 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v2tfm"] Oct 29 22:46:35.692240 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:46:35.692215 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e0f62f7_e32b_45f1_ba71_88bbb389039b.slice/crio-2b533e7cba9b596703c0320895775a5f94b41738052b749800f31d9d65831b34 WatchSource:0}: Error finding container 2b533e7cba9b596703c0320895775a5f94b41738052b749800f31d9d65831b34: Status 404 returned error can't find the container with id 2b533e7cba9b596703c0320895775a5f94b41738052b749800f31d9d65831b34 Oct 29 22:46:36.410407 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:46:36.410360 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-v2tfm" event={"ID":"5e0f62f7-e32b-45f1-ba71-88bbb389039b","Type":"ContainerStarted","Data":"2b533e7cba9b596703c0320895775a5f94b41738052b749800f31d9d65831b34"} Oct 29 22:46:36.869747 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:46:36.869713 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:46:36.978708 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:46:36.978677 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:46:37.273831 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:46:37.273803 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:46:37.386115 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:46:37.386081 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:46:37.417238 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:46:37.417189 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-v2tfm" event={"ID":"5e0f62f7-e32b-45f1-ba71-88bbb389039b","Type":"ContainerStarted","Data":"4eaec83496a218563c6a9df85414a0cb7409b332cb83e58b9eca204164529c06"} Oct 29 22:46:37.417416 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:46:37.417262 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-v2tfm" Oct 29 22:46:37.457142 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:46:37.457092 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-7c9b87d574-v2tfm" podStartSLOduration=1.9275947979999999 podStartE2EDuration="3.457075925s" totalImagesPullingTime="1.529481127s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:46:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:46:37.455689873 +0000 UTC m=+778.238903277" watchObservedRunningTime="2026-10-29 22:46:37.457075925 +0000 UTC m=+778.240289353" Oct 29 22:46:37.495093 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:46:37.495020 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:46:45.075235 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:46:45.075201 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:46:45.186088 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:46:45.186045 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:46:48.424244 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:46:48.424214 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-v2tfm" Oct 29 22:46:49.028071 ip-10-0-140-74 kubenswrapper[5057]: E1029 22:46:49.028034 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d2dnn" containerName="extractor" Oct 29 22:46:49.798610 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:46:49.798574 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:46:49.941247 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:46:49.941214 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:46:50.055680 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:46:50.055559 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:46:50.191118 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:46:50.191084 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:46:50.298683 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:46:50.298652 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-769875db56-5k5vp"] Oct 29 22:46:50.301431 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:46:50.301410 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-5k5vp" Oct 29 22:46:50.317719 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:46:50.317698 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Oct 29 22:46:50.327516 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:46:50.327494 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Oct 29 22:46:50.329127 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:46:50.329102 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:46:50.330191 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:46:50.330169 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-5k5vp"] Oct 29 22:46:50.379266 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:46:50.379242 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-metrics-tls\"" Oct 29 22:46:50.379416 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:46:50.379399 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Oct 29 22:46:50.379491 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:46:50.379431 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-z6dbk\"" Oct 29 22:46:50.443603 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:46:50.443563 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:46:50.477674 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:46:50.477643 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bng4\" (UniqueName: \"kubernetes.io/projected/494d8090-9803-4be0-a442-6108c1ebc201-kube-api-access-4bng4\") pod \"odh-model-controller-769875db56-5k5vp\" (UID: \"494d8090-9803-4be0-a442-6108c1ebc201\") " pod="opendatahub/odh-model-controller-769875db56-5k5vp" Oct 29 22:46:50.477674 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:46:50.477677 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/494d8090-9803-4be0-a442-6108c1ebc201-metrics-certs\") pod \"odh-model-controller-769875db56-5k5vp\" (UID: \"494d8090-9803-4be0-a442-6108c1ebc201\") " pod="opendatahub/odh-model-controller-769875db56-5k5vp" Oct 29 22:46:50.477918 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:46:50.477716 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/494d8090-9803-4be0-a442-6108c1ebc201-cert\") pod \"odh-model-controller-769875db56-5k5vp\" (UID: \"494d8090-9803-4be0-a442-6108c1ebc201\") " pod="opendatahub/odh-model-controller-769875db56-5k5vp" Oct 29 22:46:50.579104 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:46:50.579030 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bng4\" (UniqueName: \"kubernetes.io/projected/494d8090-9803-4be0-a442-6108c1ebc201-kube-api-access-4bng4\") pod \"odh-model-controller-769875db56-5k5vp\" (UID: \"494d8090-9803-4be0-a442-6108c1ebc201\") " pod="opendatahub/odh-model-controller-769875db56-5k5vp" Oct 29 22:46:50.579104 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:46:50.579064 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/494d8090-9803-4be0-a442-6108c1ebc201-metrics-certs\") pod \"odh-model-controller-769875db56-5k5vp\" (UID: \"494d8090-9803-4be0-a442-6108c1ebc201\") " pod="opendatahub/odh-model-controller-769875db56-5k5vp" Oct 29 22:46:50.579326 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:46:50.579121 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/494d8090-9803-4be0-a442-6108c1ebc201-cert\") pod \"odh-model-controller-769875db56-5k5vp\" (UID: \"494d8090-9803-4be0-a442-6108c1ebc201\") " pod="opendatahub/odh-model-controller-769875db56-5k5vp" Oct 29 22:46:50.581554 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:46:50.581529 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/494d8090-9803-4be0-a442-6108c1ebc201-cert\") pod \"odh-model-controller-769875db56-5k5vp\" (UID: \"494d8090-9803-4be0-a442-6108c1ebc201\") " pod="opendatahub/odh-model-controller-769875db56-5k5vp" Oct 29 22:46:50.581670 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:46:50.581587 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/494d8090-9803-4be0-a442-6108c1ebc201-metrics-certs\") pod \"odh-model-controller-769875db56-5k5vp\" (UID: \"494d8090-9803-4be0-a442-6108c1ebc201\") " pod="opendatahub/odh-model-controller-769875db56-5k5vp" Oct 29 22:46:50.600035 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:46:50.600013 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bng4\" (UniqueName: \"kubernetes.io/projected/494d8090-9803-4be0-a442-6108c1ebc201-kube-api-access-4bng4\") pod \"odh-model-controller-769875db56-5k5vp\" (UID: \"494d8090-9803-4be0-a442-6108c1ebc201\") " pod="opendatahub/odh-model-controller-769875db56-5k5vp" Oct 29 22:46:50.612882 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:46:50.612859 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-5k5vp" Oct 29 22:46:50.801993 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:46:50.801959 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:46:50.907586 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:46:50.907487 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-5k5vp"] Oct 29 22:46:50.909982 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:46:50.909949 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod494d8090_9803_4be0_a442_6108c1ebc201.slice/crio-9c4ea2fb6483bad904f2c9cf30427acc15880c1f9635e54be92f306587849bea WatchSource:0}: Error finding container 9c4ea2fb6483bad904f2c9cf30427acc15880c1f9635e54be92f306587849bea: Status 404 returned error can't find the container with id 9c4ea2fb6483bad904f2c9cf30427acc15880c1f9635e54be92f306587849bea Oct 29 22:46:51.496906 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:46:51.496839 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-5k5vp" event={"ID":"494d8090-9803-4be0-a442-6108c1ebc201","Type":"ContainerStarted","Data":"9c4ea2fb6483bad904f2c9cf30427acc15880c1f9635e54be92f306587849bea"} Oct 29 22:46:52.072389 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:46:52.072346 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:46:52.196531 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:46:52.196491 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:46:52.589005 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:46:52.588967 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:46:52.706435 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:46:52.706402 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:46:52.819811 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:46:52.819778 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:46:52.938756 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:46:52.938676 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:46:53.050147 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:46:53.050103 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:46:53.511573 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:46:53.511531 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-5k5vp" event={"ID":"494d8090-9803-4be0-a442-6108c1ebc201","Type":"ContainerStarted","Data":"8a6caaf789374e67bfa81b5cfe1c4074c656c49e2f4476e33d0b8ed4f9c63ae5"} Oct 29 22:46:53.512054 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:46:53.511617 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-769875db56-5k5vp" Oct 29 22:46:53.548964 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:46:53.548893 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/odh-model-controller-769875db56-5k5vp" podStartSLOduration=1.131003722 podStartE2EDuration="3.548878314s" totalImagesPullingTime="2.417874592s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:46:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:46:53.54869642 +0000 UTC m=+794.331909858" watchObservedRunningTime="2026-10-29 22:46:53.548878314 +0000 UTC m=+794.332091754" Oct 29 22:46:54.077097 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:46:54.077058 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:46:54.199591 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:46:54.199554 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:46:54.492434 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:46:54.492399 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:46:54.624712 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:46:54.624681 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:46:55.240318 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:46:55.240284 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:47:04.518850 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:47:04.518820 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-769875db56-5k5vp" Oct 29 22:47:14.582363 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:47:14.582312 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:47:14.695471 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:47:14.695434 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:47:15.150008 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:47:15.149972 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:47:15.274438 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:47:15.274409 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:47:15.389788 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:47:15.389756 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:47:15.502593 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:47:15.502556 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:47:18.878381 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:47:18.878318 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:47:18.996527 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:47:18.996492 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:47:19.026476 ip-10-0-140-74 kubenswrapper[5057]: E1029 22:47:19.026445 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d2dnn" containerName="extractor" Oct 29 22:47:20.662313 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:47:20.662283 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:47:20.871884 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:47:20.871853 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:47:30.403446 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:47:30.403410 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:47:49.026309 ip-10-0-140-74 kubenswrapper[5057]: E1029 22:47:49.026274 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d2dnn" containerName="extractor" Oct 29 22:47:49.279738 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:47:49.279658 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:47:49.397908 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:47:49.397760 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:47:50.685905 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:47:50.685851 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:47:50.799471 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:47:50.799387 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:47:50.889459 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:47:50.889429 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6d9676886-4ghds"] Oct 29 22:47:50.893282 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:47:50.893265 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d9676886-4ghds" Oct 29 22:47:50.917302 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:47:50.917281 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Oct 29 22:47:50.917455 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:47:50.917393 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 29 22:47:50.917514 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:47:50.917457 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 29 22:47:50.920020 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:47:50.919998 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d9676886-4ghds"] Oct 29 22:47:50.920132 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:47:50.920120 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 29 22:47:50.920186 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:47:50.920176 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-xrwtg\"" Oct 29 22:47:50.921395 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:47:50.921368 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:47:50.935255 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:47:50.935232 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xb4sh\" (UniqueName: \"kubernetes.io/projected/f617ba24-99f6-4888-b06f-2b911a8d69e3-kube-api-access-xb4sh\") pod \"maas-api-6d9676886-4ghds\" (UID: \"f617ba24-99f6-4888-b06f-2b911a8d69e3\") " pod="odh-ai-gateway-infra/maas-api-6d9676886-4ghds" Oct 29 22:47:50.935379 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:47:50.935304 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f617ba24-99f6-4888-b06f-2b911a8d69e3-metrics-certs\") pod \"maas-api-6d9676886-4ghds\" (UID: \"f617ba24-99f6-4888-b06f-2b911a8d69e3\") " pod="odh-ai-gateway-infra/maas-api-6d9676886-4ghds" Oct 29 22:47:50.935449 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:47:50.935409 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f617ba24-99f6-4888-b06f-2b911a8d69e3-openshift-service-ca\") pod \"maas-api-6d9676886-4ghds\" (UID: \"f617ba24-99f6-4888-b06f-2b911a8d69e3\") " pod="odh-ai-gateway-infra/maas-api-6d9676886-4ghds" Oct 29 22:47:50.935503 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:47:50.935464 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f617ba24-99f6-4888-b06f-2b911a8d69e3-maas-api-tls\") pod \"maas-api-6d9676886-4ghds\" (UID: \"f617ba24-99f6-4888-b06f-2b911a8d69e3\") " pod="odh-ai-gateway-infra/maas-api-6d9676886-4ghds" Oct 29 22:47:51.036730 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:47:51.036706 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xb4sh\" (UniqueName: \"kubernetes.io/projected/f617ba24-99f6-4888-b06f-2b911a8d69e3-kube-api-access-xb4sh\") pod \"maas-api-6d9676886-4ghds\" (UID: \"f617ba24-99f6-4888-b06f-2b911a8d69e3\") " pod="odh-ai-gateway-infra/maas-api-6d9676886-4ghds" Oct 29 22:47:51.036843 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:47:51.036785 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f617ba24-99f6-4888-b06f-2b911a8d69e3-metrics-certs\") pod \"maas-api-6d9676886-4ghds\" (UID: \"f617ba24-99f6-4888-b06f-2b911a8d69e3\") " pod="odh-ai-gateway-infra/maas-api-6d9676886-4ghds" Oct 29 22:47:51.036843 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:47:51.036810 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f617ba24-99f6-4888-b06f-2b911a8d69e3-openshift-service-ca\") pod \"maas-api-6d9676886-4ghds\" (UID: \"f617ba24-99f6-4888-b06f-2b911a8d69e3\") " pod="odh-ai-gateway-infra/maas-api-6d9676886-4ghds" Oct 29 22:47:51.036843 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:47:51.036837 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f617ba24-99f6-4888-b06f-2b911a8d69e3-maas-api-tls\") pod \"maas-api-6d9676886-4ghds\" (UID: \"f617ba24-99f6-4888-b06f-2b911a8d69e3\") " pod="odh-ai-gateway-infra/maas-api-6d9676886-4ghds" Oct 29 22:47:51.037473 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:47:51.037449 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f617ba24-99f6-4888-b06f-2b911a8d69e3-openshift-service-ca\") pod \"maas-api-6d9676886-4ghds\" (UID: \"f617ba24-99f6-4888-b06f-2b911a8d69e3\") " pod="odh-ai-gateway-infra/maas-api-6d9676886-4ghds" Oct 29 22:47:51.039234 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:47:51.039215 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f617ba24-99f6-4888-b06f-2b911a8d69e3-maas-api-tls\") pod \"maas-api-6d9676886-4ghds\" (UID: \"f617ba24-99f6-4888-b06f-2b911a8d69e3\") " pod="odh-ai-gateway-infra/maas-api-6d9676886-4ghds" Oct 29 22:47:51.039309 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:47:51.039251 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f617ba24-99f6-4888-b06f-2b911a8d69e3-metrics-certs\") pod \"maas-api-6d9676886-4ghds\" (UID: \"f617ba24-99f6-4888-b06f-2b911a8d69e3\") " pod="odh-ai-gateway-infra/maas-api-6d9676886-4ghds" Oct 29 22:47:51.059555 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:47:51.059534 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xb4sh\" (UniqueName: \"kubernetes.io/projected/f617ba24-99f6-4888-b06f-2b911a8d69e3-kube-api-access-xb4sh\") pod \"maas-api-6d9676886-4ghds\" (UID: \"f617ba24-99f6-4888-b06f-2b911a8d69e3\") " pod="odh-ai-gateway-infra/maas-api-6d9676886-4ghds" Oct 29 22:47:51.205906 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:47:51.205877 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d9676886-4ghds" Oct 29 22:47:51.503946 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:47:51.503922 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d9676886-4ghds"] Oct 29 22:47:51.506005 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:47:51.505977 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf617ba24_99f6_4888_b06f_2b911a8d69e3.slice/crio-b06e233f49c8a84552f1ed09fb581deb5cb3585cfc1f6dcf6ebde86e394081a6 WatchSource:0}: Error finding container b06e233f49c8a84552f1ed09fb581deb5cb3585cfc1f6dcf6ebde86e394081a6: Status 404 returned error can't find the container with id b06e233f49c8a84552f1ed09fb581deb5cb3585cfc1f6dcf6ebde86e394081a6 Oct 29 22:47:51.812723 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:47:51.812644 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d9676886-4ghds" event={"ID":"f617ba24-99f6-4888-b06f-2b911a8d69e3","Type":"ContainerStarted","Data":"b06e233f49c8a84552f1ed09fb581deb5cb3585cfc1f6dcf6ebde86e394081a6"} Oct 29 22:47:52.703968 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:47:52.703931 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v2tfm"] Oct 29 22:47:54.831162 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:47:54.831120 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d9676886-4ghds" event={"ID":"f617ba24-99f6-4888-b06f-2b911a8d69e3","Type":"ContainerStarted","Data":"6771c8216d7655818ba7aa01e429593f7b7a8634b325e8d8209a9309cd01154b"} Oct 29 22:47:54.831579 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:47:54.831192 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6d9676886-4ghds" Oct 29 22:47:54.869268 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:47:54.869209 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6d9676886-4ghds" podStartSLOduration=2.491944851 podStartE2EDuration="4.869191515s" totalImagesPullingTime="2.377246664s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:47:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:47:54.868642663 +0000 UTC m=+855.651856090" watchObservedRunningTime="2026-10-29 22:47:54.869191515 +0000 UTC m=+855.652404944" Oct 29 22:48:00.841881 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:48:00.841852 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6d9676886-4ghds" Oct 29 22:48:01.301375 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:48:01.301340 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:48:01.473874 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:48:01.473841 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:48:01.858093 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:48:01.858040 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-8457c8746d-4lscp"] Oct 29 22:48:01.864219 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:48:01.864197 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-8457c8746d-4lscp" Oct 29 22:48:01.890246 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:48:01.890221 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-8457c8746d-4lscp"] Oct 29 22:48:01.944393 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:48:01.944317 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f1fa4ed3-77b6-4d9b-b31d-08c26c2bd492-maas-api-tls\") pod \"maas-api-8457c8746d-4lscp\" (UID: \"f1fa4ed3-77b6-4d9b-b31d-08c26c2bd492\") " pod="odh-ai-gateway-infra/maas-api-8457c8746d-4lscp" Oct 29 22:48:01.944584 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:48:01.944418 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1fa4ed3-77b6-4d9b-b31d-08c26c2bd492-metrics-certs\") pod \"maas-api-8457c8746d-4lscp\" (UID: \"f1fa4ed3-77b6-4d9b-b31d-08c26c2bd492\") " pod="odh-ai-gateway-infra/maas-api-8457c8746d-4lscp" Oct 29 22:48:01.944584 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:48:01.944451 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtnlw\" (UniqueName: \"kubernetes.io/projected/f1fa4ed3-77b6-4d9b-b31d-08c26c2bd492-kube-api-access-rtnlw\") pod \"maas-api-8457c8746d-4lscp\" (UID: \"f1fa4ed3-77b6-4d9b-b31d-08c26c2bd492\") " pod="odh-ai-gateway-infra/maas-api-8457c8746d-4lscp" Oct 29 22:48:01.944709 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:48:01.944622 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1fa4ed3-77b6-4d9b-b31d-08c26c2bd492-openshift-service-ca\") pod \"maas-api-8457c8746d-4lscp\" (UID: \"f1fa4ed3-77b6-4d9b-b31d-08c26c2bd492\") " pod="odh-ai-gateway-infra/maas-api-8457c8746d-4lscp" Oct 29 22:48:02.045970 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:48:02.045944 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1fa4ed3-77b6-4d9b-b31d-08c26c2bd492-openshift-service-ca\") pod \"maas-api-8457c8746d-4lscp\" (UID: \"f1fa4ed3-77b6-4d9b-b31d-08c26c2bd492\") " pod="odh-ai-gateway-infra/maas-api-8457c8746d-4lscp" Oct 29 22:48:02.046115 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:48:02.046002 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f1fa4ed3-77b6-4d9b-b31d-08c26c2bd492-maas-api-tls\") pod \"maas-api-8457c8746d-4lscp\" (UID: \"f1fa4ed3-77b6-4d9b-b31d-08c26c2bd492\") " pod="odh-ai-gateway-infra/maas-api-8457c8746d-4lscp" Oct 29 22:48:02.046115 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:48:02.046021 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1fa4ed3-77b6-4d9b-b31d-08c26c2bd492-metrics-certs\") pod \"maas-api-8457c8746d-4lscp\" (UID: \"f1fa4ed3-77b6-4d9b-b31d-08c26c2bd492\") " pod="odh-ai-gateway-infra/maas-api-8457c8746d-4lscp" Oct 29 22:48:02.046115 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:48:02.046040 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtnlw\" (UniqueName: \"kubernetes.io/projected/f1fa4ed3-77b6-4d9b-b31d-08c26c2bd492-kube-api-access-rtnlw\") pod \"maas-api-8457c8746d-4lscp\" (UID: \"f1fa4ed3-77b6-4d9b-b31d-08c26c2bd492\") " pod="odh-ai-gateway-infra/maas-api-8457c8746d-4lscp" Oct 29 22:48:02.046728 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:48:02.046704 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1fa4ed3-77b6-4d9b-b31d-08c26c2bd492-openshift-service-ca\") pod \"maas-api-8457c8746d-4lscp\" (UID: \"f1fa4ed3-77b6-4d9b-b31d-08c26c2bd492\") " pod="odh-ai-gateway-infra/maas-api-8457c8746d-4lscp" Oct 29 22:48:02.048654 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:48:02.048628 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f1fa4ed3-77b6-4d9b-b31d-08c26c2bd492-maas-api-tls\") pod \"maas-api-8457c8746d-4lscp\" (UID: \"f1fa4ed3-77b6-4d9b-b31d-08c26c2bd492\") " pod="odh-ai-gateway-infra/maas-api-8457c8746d-4lscp" Oct 29 22:48:02.048766 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:48:02.048690 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1fa4ed3-77b6-4d9b-b31d-08c26c2bd492-metrics-certs\") pod \"maas-api-8457c8746d-4lscp\" (UID: \"f1fa4ed3-77b6-4d9b-b31d-08c26c2bd492\") " pod="odh-ai-gateway-infra/maas-api-8457c8746d-4lscp" Oct 29 22:48:02.070104 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:48:02.070078 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtnlw\" (UniqueName: \"kubernetes.io/projected/f1fa4ed3-77b6-4d9b-b31d-08c26c2bd492-kube-api-access-rtnlw\") pod \"maas-api-8457c8746d-4lscp\" (UID: \"f1fa4ed3-77b6-4d9b-b31d-08c26c2bd492\") " pod="odh-ai-gateway-infra/maas-api-8457c8746d-4lscp" Oct 29 22:48:02.178605 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:48:02.178519 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-8457c8746d-4lscp" Oct 29 22:48:02.480036 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:48:02.480006 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-8457c8746d-4lscp"] Oct 29 22:48:02.482257 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:48:02.482231 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1fa4ed3_77b6_4d9b_b31d_08c26c2bd492.slice/crio-e9c361b96106acf44eb11b5dbc2f4634ef2b968ec93bfcbd717c4c44c740de66 WatchSource:0}: Error finding container e9c361b96106acf44eb11b5dbc2f4634ef2b968ec93bfcbd717c4c44c740de66: Status 404 returned error can't find the container with id e9c361b96106acf44eb11b5dbc2f4634ef2b968ec93bfcbd717c4c44c740de66 Oct 29 22:48:02.876577 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:48:02.876492 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-8457c8746d-4lscp" event={"ID":"f1fa4ed3-77b6-4d9b-b31d-08c26c2bd492","Type":"ContainerStarted","Data":"e9c361b96106acf44eb11b5dbc2f4634ef2b968ec93bfcbd717c4c44c740de66"} Oct 29 22:48:03.884012 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:48:03.883973 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-8457c8746d-4lscp" event={"ID":"f1fa4ed3-77b6-4d9b-b31d-08c26c2bd492","Type":"ContainerStarted","Data":"1706b84e803744d4150fd2d346fbe567fd14767f8bd569b77e2941371b6e1355"} Oct 29 22:48:03.884517 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:48:03.884041 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-8457c8746d-4lscp" Oct 29 22:48:03.923231 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:48:03.923180 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-8457c8746d-4lscp" podStartSLOduration=2.479216334 podStartE2EDuration="2.923165553s" totalImagesPullingTime="443.949219ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:48:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:48:03.922931667 +0000 UTC m=+864.706145093" watchObservedRunningTime="2026-10-29 22:48:03.923165553 +0000 UTC m=+864.706378980" Oct 29 22:48:09.895001 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:48:09.894971 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-8457c8746d-4lscp" Oct 29 22:48:09.967908 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:48:09.967877 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d9676886-4ghds"] Oct 29 22:48:09.968118 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:48:09.968093 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-6d9676886-4ghds" podUID="f617ba24-99f6-4888-b06f-2b911a8d69e3" containerName="maas-api" containerID="cri-o://6771c8216d7655818ba7aa01e429593f7b7a8634b325e8d8209a9309cd01154b" gracePeriod=30 Oct 29 22:48:10.227736 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:48:10.227705 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d9676886-4ghds" Oct 29 22:48:10.328875 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:48:10.328847 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f617ba24-99f6-4888-b06f-2b911a8d69e3-kube-api-access-xb4sh\" (UniqueName: \"kubernetes.io/projected/f617ba24-99f6-4888-b06f-2b911a8d69e3-kube-api-access-xb4sh\") pod \"f617ba24-99f6-4888-b06f-2b911a8d69e3\" (UID: \"f617ba24-99f6-4888-b06f-2b911a8d69e3\") " Oct 29 22:48:10.329031 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:48:10.328882 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f617ba24-99f6-4888-b06f-2b911a8d69e3-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f617ba24-99f6-4888-b06f-2b911a8d69e3-metrics-certs\") pod \"f617ba24-99f6-4888-b06f-2b911a8d69e3\" (UID: \"f617ba24-99f6-4888-b06f-2b911a8d69e3\") " Oct 29 22:48:10.329031 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:48:10.328922 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f617ba24-99f6-4888-b06f-2b911a8d69e3-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f617ba24-99f6-4888-b06f-2b911a8d69e3-maas-api-tls\") pod \"f617ba24-99f6-4888-b06f-2b911a8d69e3\" (UID: \"f617ba24-99f6-4888-b06f-2b911a8d69e3\") " Oct 29 22:48:10.329031 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:48:10.328971 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f617ba24-99f6-4888-b06f-2b911a8d69e3-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f617ba24-99f6-4888-b06f-2b911a8d69e3-openshift-service-ca\") pod \"f617ba24-99f6-4888-b06f-2b911a8d69e3\" (UID: \"f617ba24-99f6-4888-b06f-2b911a8d69e3\") " Oct 29 22:48:10.329393 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:48:10.329363 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f617ba24-99f6-4888-b06f-2b911a8d69e3-openshift-service-ca" pod "f617ba24-99f6-4888-b06f-2b911a8d69e3" (UID: "f617ba24-99f6-4888-b06f-2b911a8d69e3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:48:10.331167 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:48:10.331145 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f617ba24-99f6-4888-b06f-2b911a8d69e3-metrics-certs" pod "f617ba24-99f6-4888-b06f-2b911a8d69e3" (UID: "f617ba24-99f6-4888-b06f-2b911a8d69e3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:48:10.331260 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:48:10.331169 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f617ba24-99f6-4888-b06f-2b911a8d69e3-maas-api-tls" pod "f617ba24-99f6-4888-b06f-2b911a8d69e3" (UID: "f617ba24-99f6-4888-b06f-2b911a8d69e3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:48:10.331260 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:48:10.331240 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f617ba24-99f6-4888-b06f-2b911a8d69e3-kube-api-access-xb4sh" pod "f617ba24-99f6-4888-b06f-2b911a8d69e3" (UID: "f617ba24-99f6-4888-b06f-2b911a8d69e3"). InnerVolumeSpecName "kube-api-access-xb4sh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:48:10.430707 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:48:10.430680 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xb4sh\" (UniqueName: \"kubernetes.io/projected/f617ba24-99f6-4888-b06f-2b911a8d69e3-kube-api-access-xb4sh\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 22:48:10.430707 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:48:10.430703 5057 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f617ba24-99f6-4888-b06f-2b911a8d69e3-metrics-certs\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 22:48:10.430707 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:48:10.430712 5057 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f617ba24-99f6-4888-b06f-2b911a8d69e3-maas-api-tls\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 22:48:10.430879 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:48:10.430721 5057 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f617ba24-99f6-4888-b06f-2b911a8d69e3-openshift-service-ca\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 22:48:10.919851 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:48:10.919815 5057 generic.go:356] "Generic (PLEG): container finished" podID="f617ba24-99f6-4888-b06f-2b911a8d69e3" containerID="6771c8216d7655818ba7aa01e429593f7b7a8634b325e8d8209a9309cd01154b" exitCode=0 Oct 29 22:48:10.920265 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:48:10.919859 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d9676886-4ghds" event={"ID":"f617ba24-99f6-4888-b06f-2b911a8d69e3","Type":"ContainerDied","Data":"6771c8216d7655818ba7aa01e429593f7b7a8634b325e8d8209a9309cd01154b"} Oct 29 22:48:10.920265 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:48:10.919884 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d9676886-4ghds" Oct 29 22:48:10.920265 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:48:10.919910 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d9676886-4ghds" event={"ID":"f617ba24-99f6-4888-b06f-2b911a8d69e3","Type":"ContainerDied","Data":"b06e233f49c8a84552f1ed09fb581deb5cb3585cfc1f6dcf6ebde86e394081a6"} Oct 29 22:48:10.920265 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:48:10.919939 5057 scope.go:122] "RemoveContainer" containerID="6771c8216d7655818ba7aa01e429593f7b7a8634b325e8d8209a9309cd01154b" Oct 29 22:48:10.930134 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:48:10.930115 5057 scope.go:122] "RemoveContainer" containerID="6771c8216d7655818ba7aa01e429593f7b7a8634b325e8d8209a9309cd01154b" Oct 29 22:48:10.930452 ip-10-0-140-74 kubenswrapper[5057]: E1029 22:48:10.930428 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6771c8216d7655818ba7aa01e429593f7b7a8634b325e8d8209a9309cd01154b\": container with ID starting with 6771c8216d7655818ba7aa01e429593f7b7a8634b325e8d8209a9309cd01154b not found: ID does not exist" containerID="6771c8216d7655818ba7aa01e429593f7b7a8634b325e8d8209a9309cd01154b" Oct 29 22:48:10.930505 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:48:10.930464 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6771c8216d7655818ba7aa01e429593f7b7a8634b325e8d8209a9309cd01154b"} err="failed to get container status \"6771c8216d7655818ba7aa01e429593f7b7a8634b325e8d8209a9309cd01154b\": rpc error: code = NotFound desc = could not find container \"6771c8216d7655818ba7aa01e429593f7b7a8634b325e8d8209a9309cd01154b\": container with ID starting with 6771c8216d7655818ba7aa01e429593f7b7a8634b325e8d8209a9309cd01154b not found: ID does not exist" Oct 29 22:48:10.965574 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:48:10.965550 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d9676886-4ghds"] Oct 29 22:48:10.975665 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:48:10.975633 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d9676886-4ghds"] Oct 29 22:48:11.327733 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:48:11.327702 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f617ba24-99f6-4888-b06f-2b911a8d69e3" path="/var/lib/kubelet/pods/f617ba24-99f6-4888-b06f-2b911a8d69e3/volumes" Oct 29 22:48:19.028890 ip-10-0-140-74 kubenswrapper[5057]: E1029 22:48:19.028853 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d2dnn" containerName="extractor" Oct 29 22:48:30.536808 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:48:30.536771 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:48:39.263729 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:48:39.263687 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:48:39.375180 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:48:39.375145 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:48:39.436994 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:48:39.436965 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-vj6md_aa6a190e-3283-458a-9dfd-2de7cb193fc5/console-operator/2.log" Oct 29 22:48:39.441981 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:48:39.441959 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-vj6md_aa6a190e-3283-458a-9dfd-2de7cb193fc5/console-operator/2.log" Oct 29 22:48:43.455999 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:48:43.455926 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:48:43.576281 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:48:43.576240 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:48:43.632616 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:48:43.632565 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-vsszj"] Oct 29 22:48:43.633078 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:48:43.633061 5057 cpu_manager.go:414] "containerMap: removing container" podUID="f617ba24-99f6-4888-b06f-2b911a8d69e3" containerName="maas-api" Oct 29 22:48:43.633128 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:48:43.633080 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f617ba24-99f6-4888-b06f-2b911a8d69e3" containerName="maas-api" Oct 29 22:48:43.633162 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:48:43.633149 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="f617ba24-99f6-4888-b06f-2b911a8d69e3" containerName="maas-api" Oct 29 22:48:43.637431 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:48:43.637412 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-vsszj" Oct 29 22:48:43.654222 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:48:43.654199 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-vsszj"] Oct 29 22:48:43.661510 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:48:43.661490 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-zplsr\"" Oct 29 22:48:43.678520 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:48:43.678496 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Oct 29 22:48:43.729939 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:48:43.729880 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b4fbaae-ed3b-4a78-b38c-b3a245bf77da-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-vsszj\" (UID: \"6b4fbaae-ed3b-4a78-b38c-b3a245bf77da\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-vsszj" Oct 29 22:48:43.729939 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:48:43.729931 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82q7c\" (UniqueName: \"kubernetes.io/projected/6b4fbaae-ed3b-4a78-b38c-b3a245bf77da-kube-api-access-82q7c\") pod \"payload-pre-processing-7b98bb678c-vsszj\" (UID: \"6b4fbaae-ed3b-4a78-b38c-b3a245bf77da\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-vsszj" Oct 29 22:48:43.830725 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:48:43.830700 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b4fbaae-ed3b-4a78-b38c-b3a245bf77da-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-vsszj\" (UID: \"6b4fbaae-ed3b-4a78-b38c-b3a245bf77da\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-vsszj" Oct 29 22:48:43.830850 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:48:43.830743 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82q7c\" (UniqueName: \"kubernetes.io/projected/6b4fbaae-ed3b-4a78-b38c-b3a245bf77da-kube-api-access-82q7c\") pod \"payload-pre-processing-7b98bb678c-vsszj\" (UID: \"6b4fbaae-ed3b-4a78-b38c-b3a245bf77da\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-vsszj" Oct 29 22:48:43.831462 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:48:43.831446 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b4fbaae-ed3b-4a78-b38c-b3a245bf77da-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-vsszj\" (UID: \"6b4fbaae-ed3b-4a78-b38c-b3a245bf77da\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-vsszj" Oct 29 22:48:43.851787 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:48:43.851761 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-82q7c\" (UniqueName: \"kubernetes.io/projected/6b4fbaae-ed3b-4a78-b38c-b3a245bf77da-kube-api-access-82q7c\") pod \"payload-pre-processing-7b98bb678c-vsszj\" (UID: \"6b4fbaae-ed3b-4a78-b38c-b3a245bf77da\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-vsszj" Oct 29 22:48:43.950303 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:48:43.950277 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-vsszj" Oct 29 22:48:44.246088 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:48:44.246060 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-vsszj"] Oct 29 22:48:44.248438 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:48:44.248410 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b4fbaae_ed3b_4a78_b38c_b3a245bf77da.slice/crio-03ea474585e4617bb2b0afd20806e41cc2b9c8bbc2abac03278034b3dc82e0bf WatchSource:0}: Error finding container 03ea474585e4617bb2b0afd20806e41cc2b9c8bbc2abac03278034b3dc82e0bf: Status 404 returned error can't find the container with id 03ea474585e4617bb2b0afd20806e41cc2b9c8bbc2abac03278034b3dc82e0bf Oct 29 22:48:45.103564 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:48:45.103530 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-vsszj" event={"ID":"6b4fbaae-ed3b-4a78-b38c-b3a245bf77da","Type":"ContainerStarted","Data":"03ea474585e4617bb2b0afd20806e41cc2b9c8bbc2abac03278034b3dc82e0bf"} Oct 29 22:48:47.114569 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:48:47.114531 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-vsszj" event={"ID":"6b4fbaae-ed3b-4a78-b38c-b3a245bf77da","Type":"ContainerStarted","Data":"ed89ac037e845bdd47be0e4045b8a5b1b48f09dbab89bd44d2278036b735ba9d"} Oct 29 22:48:47.115012 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:48:47.114635 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-vsszj" Oct 29 22:48:47.152251 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:48:47.152198 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7b98bb678c-vsszj" podStartSLOduration=2.266171486 podStartE2EDuration="4.152186094s" totalImagesPullingTime="1.886014608s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:48:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:48:47.150702427 +0000 UTC m=+907.933915852" watchObservedRunningTime="2026-10-29 22:48:47.152186094 +0000 UTC m=+907.935399530" Oct 29 22:48:49.026720 ip-10-0-140-74 kubenswrapper[5057]: E1029 22:48:49.026689 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d2dnn" containerName="extractor" Oct 29 22:48:53.012505 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:48:53.012465 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-59d4d9f4d8-v7rk9"] Oct 29 22:48:53.016686 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:48:53.016662 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-59d4d9f4d8-v7rk9" Oct 29 22:48:53.031201 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:48:53.031176 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-59d4d9f4d8-v7rk9"] Oct 29 22:48:53.038660 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:48:53.038636 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\"" Oct 29 22:48:53.038784 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:48:53.038737 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\"" Oct 29 22:48:53.042091 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:48:53.042071 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-mg94d\"" Oct 29 22:48:53.117731 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:48:53.117703 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb8ee125-7b3a-48c1-8a10-0d2bf17a4eb0-metrics-certs\") pod \"ai-gateway-controller-59d4d9f4d8-v7rk9\" (UID: \"bb8ee125-7b3a-48c1-8a10-0d2bf17a4eb0\") " pod="opendatahub/ai-gateway-controller-59d4d9f4d8-v7rk9" Oct 29 22:48:53.117859 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:48:53.117813 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sh2pm\" (UniqueName: \"kubernetes.io/projected/bb8ee125-7b3a-48c1-8a10-0d2bf17a4eb0-kube-api-access-sh2pm\") pod \"ai-gateway-controller-59d4d9f4d8-v7rk9\" (UID: \"bb8ee125-7b3a-48c1-8a10-0d2bf17a4eb0\") " pod="opendatahub/ai-gateway-controller-59d4d9f4d8-v7rk9" Oct 29 22:48:53.218571 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:48:53.218540 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb8ee125-7b3a-48c1-8a10-0d2bf17a4eb0-metrics-certs\") pod \"ai-gateway-controller-59d4d9f4d8-v7rk9\" (UID: \"bb8ee125-7b3a-48c1-8a10-0d2bf17a4eb0\") " pod="opendatahub/ai-gateway-controller-59d4d9f4d8-v7rk9" Oct 29 22:48:53.218726 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:48:53.218694 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sh2pm\" (UniqueName: \"kubernetes.io/projected/bb8ee125-7b3a-48c1-8a10-0d2bf17a4eb0-kube-api-access-sh2pm\") pod \"ai-gateway-controller-59d4d9f4d8-v7rk9\" (UID: \"bb8ee125-7b3a-48c1-8a10-0d2bf17a4eb0\") " pod="opendatahub/ai-gateway-controller-59d4d9f4d8-v7rk9" Oct 29 22:48:53.220923 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:48:53.220901 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb8ee125-7b3a-48c1-8a10-0d2bf17a4eb0-metrics-certs\") pod \"ai-gateway-controller-59d4d9f4d8-v7rk9\" (UID: \"bb8ee125-7b3a-48c1-8a10-0d2bf17a4eb0\") " pod="opendatahub/ai-gateway-controller-59d4d9f4d8-v7rk9" Oct 29 22:48:53.240284 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:48:53.240261 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sh2pm\" (UniqueName: \"kubernetes.io/projected/bb8ee125-7b3a-48c1-8a10-0d2bf17a4eb0-kube-api-access-sh2pm\") pod \"ai-gateway-controller-59d4d9f4d8-v7rk9\" (UID: \"bb8ee125-7b3a-48c1-8a10-0d2bf17a4eb0\") " pod="opendatahub/ai-gateway-controller-59d4d9f4d8-v7rk9" Oct 29 22:48:53.329268 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:48:53.329207 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-59d4d9f4d8-v7rk9" Oct 29 22:48:53.626718 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:48:53.626694 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-59d4d9f4d8-v7rk9"] Oct 29 22:48:53.629050 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:48:53.629022 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb8ee125_7b3a_48c1_8a10_0d2bf17a4eb0.slice/crio-780a72fb596dd25e8cc24c980396be8fabf5c5650ed77368930de4f5fb4eff24 WatchSource:0}: Error finding container 780a72fb596dd25e8cc24c980396be8fabf5c5650ed77368930de4f5fb4eff24: Status 404 returned error can't find the container with id 780a72fb596dd25e8cc24c980396be8fabf5c5650ed77368930de4f5fb4eff24 Oct 29 22:48:54.151324 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:48:54.151288 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-59d4d9f4d8-v7rk9" event={"ID":"bb8ee125-7b3a-48c1-8a10-0d2bf17a4eb0","Type":"ContainerStarted","Data":"780a72fb596dd25e8cc24c980396be8fabf5c5650ed77368930de4f5fb4eff24"} Oct 29 22:48:55.157766 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:48:55.157732 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-59d4d9f4d8-v7rk9" event={"ID":"bb8ee125-7b3a-48c1-8a10-0d2bf17a4eb0","Type":"ContainerStarted","Data":"e90bb136da0422a2a7cd41a08b4bbfa414e7bd9ba685230bc21c6af3d84d6ae4"} Oct 29 22:48:55.158230 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:48:55.157797 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-59d4d9f4d8-v7rk9" Oct 29 22:48:55.196489 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:48:55.196397 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-59d4d9f4d8-v7rk9" podStartSLOduration=1.9085712030000002 podStartE2EDuration="3.196383843s" totalImagesPullingTime="1.28781264s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:48:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:48:55.194455506 +0000 UTC m=+915.977668935" watchObservedRunningTime="2026-10-29 22:48:55.196383843 +0000 UTC m=+915.979597269" Oct 29 22:48:58.121965 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:48:58.121936 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-vsszj" Oct 29 22:49:06.166179 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:06.166148 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-59d4d9f4d8-v7rk9" Oct 29 22:49:07.924760 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:07.924717 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:49:08.429170 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:08.429136 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:49:09.696633 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:09.696589 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:49:09.807476 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:09.807442 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:49:14.409723 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:14.409690 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-dbdjq"] Oct 29 22:49:14.418900 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:14.418877 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-dbdjq" Oct 29 22:49:14.433374 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:14.433319 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-dbdjq"] Oct 29 22:49:14.440645 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:14.440621 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Oct 29 22:49:14.443215 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:14.443193 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Oct 29 22:49:14.443315 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:14.443199 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-8dzpq\"" Oct 29 22:49:14.443315 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:14.443222 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Oct 29 22:49:14.476665 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:14.476637 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:49:14.521581 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:14.521554 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ee5747c-e4e9-48f6-ad11-ab3673116724-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-dbdjq\" (UID: \"3ee5747c-e4e9-48f6-ad11-ab3673116724\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-dbdjq" Oct 29 22:49:14.521686 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:14.521598 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ee5747c-e4e9-48f6-ad11-ab3673116724-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-dbdjq\" (UID: \"3ee5747c-e4e9-48f6-ad11-ab3673116724\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-dbdjq" Oct 29 22:49:14.521686 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:14.521665 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ee5747c-e4e9-48f6-ad11-ab3673116724-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-dbdjq\" (UID: \"3ee5747c-e4e9-48f6-ad11-ab3673116724\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-dbdjq" Oct 29 22:49:14.521803 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:14.521693 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ee5747c-e4e9-48f6-ad11-ab3673116724-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-dbdjq\" (UID: \"3ee5747c-e4e9-48f6-ad11-ab3673116724\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-dbdjq" Oct 29 22:49:14.521803 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:14.521735 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbr7z\" (UniqueName: \"kubernetes.io/projected/3ee5747c-e4e9-48f6-ad11-ab3673116724-kube-api-access-wbr7z\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-dbdjq\" (UID: \"3ee5747c-e4e9-48f6-ad11-ab3673116724\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-dbdjq" Oct 29 22:49:14.521896 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:14.521812 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ee5747c-e4e9-48f6-ad11-ab3673116724-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-dbdjq\" (UID: \"3ee5747c-e4e9-48f6-ad11-ab3673116724\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-dbdjq" Oct 29 22:49:14.623186 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:14.623158 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbr7z\" (UniqueName: \"kubernetes.io/projected/3ee5747c-e4e9-48f6-ad11-ab3673116724-kube-api-access-wbr7z\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-dbdjq\" (UID: \"3ee5747c-e4e9-48f6-ad11-ab3673116724\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-dbdjq" Oct 29 22:49:14.623321 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:14.623189 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ee5747c-e4e9-48f6-ad11-ab3673116724-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-dbdjq\" (UID: \"3ee5747c-e4e9-48f6-ad11-ab3673116724\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-dbdjq" Oct 29 22:49:14.623321 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:14.623258 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ee5747c-e4e9-48f6-ad11-ab3673116724-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-dbdjq\" (UID: \"3ee5747c-e4e9-48f6-ad11-ab3673116724\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-dbdjq" Oct 29 22:49:14.623321 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:14.623283 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ee5747c-e4e9-48f6-ad11-ab3673116724-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-dbdjq\" (UID: \"3ee5747c-e4e9-48f6-ad11-ab3673116724\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-dbdjq" Oct 29 22:49:14.623321 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:14.623310 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ee5747c-e4e9-48f6-ad11-ab3673116724-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-dbdjq\" (UID: \"3ee5747c-e4e9-48f6-ad11-ab3673116724\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-dbdjq" Oct 29 22:49:14.623568 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:14.623415 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ee5747c-e4e9-48f6-ad11-ab3673116724-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-dbdjq\" (UID: \"3ee5747c-e4e9-48f6-ad11-ab3673116724\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-dbdjq" Oct 29 22:49:14.623759 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:14.623733 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ee5747c-e4e9-48f6-ad11-ab3673116724-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-dbdjq\" (UID: \"3ee5747c-e4e9-48f6-ad11-ab3673116724\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-dbdjq" Oct 29 22:49:14.623842 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:14.623788 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ee5747c-e4e9-48f6-ad11-ab3673116724-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-dbdjq\" (UID: \"3ee5747c-e4e9-48f6-ad11-ab3673116724\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-dbdjq" Oct 29 22:49:14.623842 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:14.623825 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ee5747c-e4e9-48f6-ad11-ab3673116724-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-dbdjq\" (UID: \"3ee5747c-e4e9-48f6-ad11-ab3673116724\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-dbdjq" Oct 29 22:49:14.625519 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:14.625498 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ee5747c-e4e9-48f6-ad11-ab3673116724-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-dbdjq\" (UID: \"3ee5747c-e4e9-48f6-ad11-ab3673116724\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-dbdjq" Oct 29 22:49:14.625823 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:14.625806 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ee5747c-e4e9-48f6-ad11-ab3673116724-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-dbdjq\" (UID: \"3ee5747c-e4e9-48f6-ad11-ab3673116724\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-dbdjq" Oct 29 22:49:14.647138 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:14.647116 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbr7z\" (UniqueName: \"kubernetes.io/projected/3ee5747c-e4e9-48f6-ad11-ab3673116724-kube-api-access-wbr7z\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-dbdjq\" (UID: \"3ee5747c-e4e9-48f6-ad11-ab3673116724\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-dbdjq" Oct 29 22:49:14.734176 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:14.734148 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-dbdjq" Oct 29 22:49:15.034117 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:15.034042 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-dbdjq"] Oct 29 22:49:15.036571 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:49:15.036539 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ee5747c_e4e9_48f6_ad11_ab3673116724.slice/crio-511055c49a7430f4d5dba2433568514bfce1868a77863d7f8ad0da249ef4f02e WatchSource:0}: Error finding container 511055c49a7430f4d5dba2433568514bfce1868a77863d7f8ad0da249ef4f02e: Status 404 returned error can't find the container with id 511055c49a7430f4d5dba2433568514bfce1868a77863d7f8ad0da249ef4f02e Oct 29 22:49:15.162850 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:15.162816 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v2tfm"] Oct 29 22:49:15.263931 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:15.263898 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-dbdjq" event={"ID":"3ee5747c-e4e9-48f6-ad11-ab3673116724","Type":"ContainerStarted","Data":"511055c49a7430f4d5dba2433568514bfce1868a77863d7f8ad0da249ef4f02e"} Oct 29 22:49:17.277155 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:17.277115 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-dbdjq" event={"ID":"3ee5747c-e4e9-48f6-ad11-ab3673116724","Type":"ContainerStarted","Data":"65953a86f02ed250423290a1a3fd5261025e937fefc82f607324e5fe319e226e"} Oct 29 22:49:17.357584 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:17.357525 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-dbdjq" podStartSLOduration=2.142697118 podStartE2EDuration="3.357509501s" totalImagesPullingTime="1.214812383s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:49:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:49:17.353170244 +0000 UTC m=+938.136383669" watchObservedRunningTime="2026-10-29 22:49:17.357509501 +0000 UTC m=+938.140722927" Oct 29 22:49:17.672735 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:17.672649 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:49:17.734700 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:17.734659 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-dbdjq" Oct 29 22:49:18.467327 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:18.467288 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v2tfm"] Oct 29 22:49:19.026908 ip-10-0-140-74 kubenswrapper[5057]: E1029 22:49:19.026874 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d2dnn" containerName="extractor" Oct 29 22:49:19.734457 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:19.734421 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-vp6zb"] Oct 29 22:49:19.739010 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:19.738978 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-vp6zb" Oct 29 22:49:19.756986 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:19.756959 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-vp6zb"] Oct 29 22:49:19.764009 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:19.763979 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\"" Oct 29 22:49:19.803810 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:19.803754 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:49:19.883139 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:19.883106 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3254fb15-5180-4c0e-afdb-69062103f3e9-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-vp6zb\" (UID: \"3254fb15-5180-4c0e-afdb-69062103f3e9\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-vp6zb" Oct 29 22:49:19.883291 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:19.883150 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3254fb15-5180-4c0e-afdb-69062103f3e9-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-vp6zb\" (UID: \"3254fb15-5180-4c0e-afdb-69062103f3e9\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-vp6zb" Oct 29 22:49:19.883291 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:19.883186 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3254fb15-5180-4c0e-afdb-69062103f3e9-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-vp6zb\" (UID: \"3254fb15-5180-4c0e-afdb-69062103f3e9\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-vp6zb" Oct 29 22:49:19.883397 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:19.883316 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3254fb15-5180-4c0e-afdb-69062103f3e9-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-vp6zb\" (UID: \"3254fb15-5180-4c0e-afdb-69062103f3e9\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-vp6zb" Oct 29 22:49:19.883397 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:19.883383 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-526kp\" (UniqueName: \"kubernetes.io/projected/3254fb15-5180-4c0e-afdb-69062103f3e9-kube-api-access-526kp\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-vp6zb\" (UID: \"3254fb15-5180-4c0e-afdb-69062103f3e9\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-vp6zb" Oct 29 22:49:19.883464 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:19.883421 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3254fb15-5180-4c0e-afdb-69062103f3e9-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-vp6zb\" (UID: \"3254fb15-5180-4c0e-afdb-69062103f3e9\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-vp6zb" Oct 29 22:49:19.984449 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:19.984419 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3254fb15-5180-4c0e-afdb-69062103f3e9-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-vp6zb\" (UID: \"3254fb15-5180-4c0e-afdb-69062103f3e9\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-vp6zb" Oct 29 22:49:19.984643 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:19.984464 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3254fb15-5180-4c0e-afdb-69062103f3e9-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-vp6zb\" (UID: \"3254fb15-5180-4c0e-afdb-69062103f3e9\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-vp6zb" Oct 29 22:49:19.984643 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:19.984506 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-526kp\" (UniqueName: \"kubernetes.io/projected/3254fb15-5180-4c0e-afdb-69062103f3e9-kube-api-access-526kp\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-vp6zb\" (UID: \"3254fb15-5180-4c0e-afdb-69062103f3e9\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-vp6zb" Oct 29 22:49:19.984643 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:19.984522 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3254fb15-5180-4c0e-afdb-69062103f3e9-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-vp6zb\" (UID: \"3254fb15-5180-4c0e-afdb-69062103f3e9\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-vp6zb" Oct 29 22:49:19.984643 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:19.984546 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3254fb15-5180-4c0e-afdb-69062103f3e9-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-vp6zb\" (UID: \"3254fb15-5180-4c0e-afdb-69062103f3e9\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-vp6zb" Oct 29 22:49:19.984643 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:19.984572 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3254fb15-5180-4c0e-afdb-69062103f3e9-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-vp6zb\" (UID: \"3254fb15-5180-4c0e-afdb-69062103f3e9\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-vp6zb" Oct 29 22:49:19.984935 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:19.984903 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3254fb15-5180-4c0e-afdb-69062103f3e9-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-vp6zb\" (UID: \"3254fb15-5180-4c0e-afdb-69062103f3e9\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-vp6zb" Oct 29 22:49:19.984981 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:19.984953 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3254fb15-5180-4c0e-afdb-69062103f3e9-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-vp6zb\" (UID: \"3254fb15-5180-4c0e-afdb-69062103f3e9\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-vp6zb" Oct 29 22:49:19.985022 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:19.985004 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3254fb15-5180-4c0e-afdb-69062103f3e9-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-vp6zb\" (UID: \"3254fb15-5180-4c0e-afdb-69062103f3e9\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-vp6zb" Oct 29 22:49:19.986660 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:19.986639 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3254fb15-5180-4c0e-afdb-69062103f3e9-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-vp6zb\" (UID: \"3254fb15-5180-4c0e-afdb-69062103f3e9\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-vp6zb" Oct 29 22:49:19.986961 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:19.986944 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3254fb15-5180-4c0e-afdb-69062103f3e9-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-vp6zb\" (UID: \"3254fb15-5180-4c0e-afdb-69062103f3e9\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-vp6zb" Oct 29 22:49:20.005802 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:20.005771 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-526kp\" (UniqueName: \"kubernetes.io/projected/3254fb15-5180-4c0e-afdb-69062103f3e9-kube-api-access-526kp\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-vp6zb\" (UID: \"3254fb15-5180-4c0e-afdb-69062103f3e9\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-vp6zb" Oct 29 22:49:20.053941 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:20.053880 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-vp6zb" Oct 29 22:49:20.354270 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:20.354186 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-vp6zb"] Oct 29 22:49:20.358571 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:49:20.358545 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3254fb15_5180_4c0e_afdb_69062103f3e9.slice/crio-43dcc733e4d49cb5890e2bbd1a69dcf5ad8c2636ef6bf98ac34e18d4f1d4b3bb WatchSource:0}: Error finding container 43dcc733e4d49cb5890e2bbd1a69dcf5ad8c2636ef6bf98ac34e18d4f1d4b3bb: Status 404 returned error can't find the container with id 43dcc733e4d49cb5890e2bbd1a69dcf5ad8c2636ef6bf98ac34e18d4f1d4b3bb Oct 29 22:49:21.300292 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:21.300257 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-vp6zb" event={"ID":"3254fb15-5180-4c0e-afdb-69062103f3e9","Type":"ContainerStarted","Data":"14b3249012b4a316bc195de193d99ff54763679c40ca805edab45ea870da1ee8"} Oct 29 22:49:21.300670 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:21.300307 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-vp6zb" event={"ID":"3254fb15-5180-4c0e-afdb-69062103f3e9","Type":"ContainerStarted","Data":"43dcc733e4d49cb5890e2bbd1a69dcf5ad8c2636ef6bf98ac34e18d4f1d4b3bb"} Oct 29 22:49:21.340580 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:21.340530 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-vp6zb" podStartSLOduration=2.150648107 podStartE2EDuration="2.340515352s" totalImagesPullingTime="189.867245ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:49:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:49:21.337982511 +0000 UTC m=+942.121195937" watchObservedRunningTime="2026-10-29 22:49:21.340515352 +0000 UTC m=+942.123728779" Oct 29 22:49:21.487650 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:21.487620 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:49:22.054713 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:22.054682 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-vp6zb" Oct 29 22:49:22.885946 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:22.885910 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v2tfm"] Oct 29 22:49:23.230147 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:23.230105 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-jqfzk"] Oct 29 22:49:23.234169 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:23.234148 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-jqfzk" Oct 29 22:49:23.253240 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:23.253216 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-jqfzk"] Oct 29 22:49:23.257097 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:23.257073 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Oct 29 22:49:23.298641 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:23.298609 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:49:23.317220 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:23.317194 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cee3c70c-dc5a-4fae-9d28-820fc4dea236-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-jqfzk\" (UID: \"cee3c70c-dc5a-4fae-9d28-820fc4dea236\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-jqfzk" Oct 29 22:49:23.317392 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:23.317310 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvvhp\" (UniqueName: \"kubernetes.io/projected/cee3c70c-dc5a-4fae-9d28-820fc4dea236-kube-api-access-nvvhp\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-jqfzk\" (UID: \"cee3c70c-dc5a-4fae-9d28-820fc4dea236\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-jqfzk" Oct 29 22:49:23.317392 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:23.317383 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cee3c70c-dc5a-4fae-9d28-820fc4dea236-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-jqfzk\" (UID: \"cee3c70c-dc5a-4fae-9d28-820fc4dea236\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-jqfzk" Oct 29 22:49:23.317506 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:23.317417 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cee3c70c-dc5a-4fae-9d28-820fc4dea236-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-jqfzk\" (UID: \"cee3c70c-dc5a-4fae-9d28-820fc4dea236\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-jqfzk" Oct 29 22:49:23.317506 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:23.317488 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cee3c70c-dc5a-4fae-9d28-820fc4dea236-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-jqfzk\" (UID: \"cee3c70c-dc5a-4fae-9d28-820fc4dea236\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-jqfzk" Oct 29 22:49:23.317634 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:23.317611 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cee3c70c-dc5a-4fae-9d28-820fc4dea236-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-jqfzk\" (UID: \"cee3c70c-dc5a-4fae-9d28-820fc4dea236\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-jqfzk" Oct 29 22:49:23.418154 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:23.418109 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cee3c70c-dc5a-4fae-9d28-820fc4dea236-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-jqfzk\" (UID: \"cee3c70c-dc5a-4fae-9d28-820fc4dea236\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-jqfzk" Oct 29 22:49:23.418445 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:23.418406 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cee3c70c-dc5a-4fae-9d28-820fc4dea236-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-jqfzk\" (UID: \"cee3c70c-dc5a-4fae-9d28-820fc4dea236\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-jqfzk" Oct 29 22:49:23.418605 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:23.418580 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cee3c70c-dc5a-4fae-9d28-820fc4dea236-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-jqfzk\" (UID: \"cee3c70c-dc5a-4fae-9d28-820fc4dea236\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-jqfzk" Oct 29 22:49:23.418605 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:23.418594 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cee3c70c-dc5a-4fae-9d28-820fc4dea236-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-jqfzk\" (UID: \"cee3c70c-dc5a-4fae-9d28-820fc4dea236\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-jqfzk" Oct 29 22:49:23.418778 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:23.418725 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvvhp\" (UniqueName: \"kubernetes.io/projected/cee3c70c-dc5a-4fae-9d28-820fc4dea236-kube-api-access-nvvhp\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-jqfzk\" (UID: \"cee3c70c-dc5a-4fae-9d28-820fc4dea236\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-jqfzk" Oct 29 22:49:23.418778 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:23.418763 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cee3c70c-dc5a-4fae-9d28-820fc4dea236-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-jqfzk\" (UID: \"cee3c70c-dc5a-4fae-9d28-820fc4dea236\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-jqfzk" Oct 29 22:49:23.418944 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:23.418798 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cee3c70c-dc5a-4fae-9d28-820fc4dea236-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-jqfzk\" (UID: \"cee3c70c-dc5a-4fae-9d28-820fc4dea236\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-jqfzk" Oct 29 22:49:23.419124 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:23.419094 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cee3c70c-dc5a-4fae-9d28-820fc4dea236-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-jqfzk\" (UID: \"cee3c70c-dc5a-4fae-9d28-820fc4dea236\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-jqfzk" Oct 29 22:49:23.419533 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:23.419512 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cee3c70c-dc5a-4fae-9d28-820fc4dea236-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-jqfzk\" (UID: \"cee3c70c-dc5a-4fae-9d28-820fc4dea236\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-jqfzk" Oct 29 22:49:23.421146 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:23.421116 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cee3c70c-dc5a-4fae-9d28-820fc4dea236-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-jqfzk\" (UID: \"cee3c70c-dc5a-4fae-9d28-820fc4dea236\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-jqfzk" Oct 29 22:49:23.421248 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:23.421233 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cee3c70c-dc5a-4fae-9d28-820fc4dea236-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-jqfzk\" (UID: \"cee3c70c-dc5a-4fae-9d28-820fc4dea236\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-jqfzk" Oct 29 22:49:23.439943 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:23.439916 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvvhp\" (UniqueName: \"kubernetes.io/projected/cee3c70c-dc5a-4fae-9d28-820fc4dea236-kube-api-access-nvvhp\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-jqfzk\" (UID: \"cee3c70c-dc5a-4fae-9d28-820fc4dea236\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-jqfzk" Oct 29 22:49:23.553015 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:23.552930 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-jqfzk" Oct 29 22:49:23.857819 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:23.857753 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-jqfzk"] Oct 29 22:49:23.861007 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:49:23.860976 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcee3c70c_dc5a_4fae_9d28_820fc4dea236.slice/crio-472ea356c3bfa309fe42cecbaab7e3bdf76218592f6cf8c46bcd620c6f5fc72d WatchSource:0}: Error finding container 472ea356c3bfa309fe42cecbaab7e3bdf76218592f6cf8c46bcd620c6f5fc72d: Status 404 returned error can't find the container with id 472ea356c3bfa309fe42cecbaab7e3bdf76218592f6cf8c46bcd620c6f5fc72d Oct 29 22:49:24.318713 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:24.318678 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-jqfzk" event={"ID":"cee3c70c-dc5a-4fae-9d28-820fc4dea236","Type":"ContainerStarted","Data":"824745ab2913008648973da4458b3d97106d2ccdfd79a842fa9ac57c1c2a9e1b"} Oct 29 22:49:24.319066 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:24.318733 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-jqfzk" event={"ID":"cee3c70c-dc5a-4fae-9d28-820fc4dea236","Type":"ContainerStarted","Data":"472ea356c3bfa309fe42cecbaab7e3bdf76218592f6cf8c46bcd620c6f5fc72d"} Oct 29 22:49:24.357109 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:24.357049 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-jqfzk" podStartSLOduration=1.176811287 podStartE2EDuration="1.357034132s" totalImagesPullingTime="180.222845ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:49:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:49:24.356259972 +0000 UTC m=+945.139473398" watchObservedRunningTime="2026-10-29 22:49:24.357034132 +0000 UTC m=+945.140247559" Oct 29 22:49:24.554073 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:24.554040 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-jqfzk" Oct 29 22:49:24.734625 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:24.734598 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-dbdjq" Oct 29 22:49:24.747161 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:24.747137 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-dbdjq" Oct 29 22:49:25.336203 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:25.336175 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-dbdjq" Oct 29 22:49:25.822130 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:25.822083 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:49:27.332992 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:27.332958 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-w67ch"] Oct 29 22:49:27.337285 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:27.337260 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-w67ch" Oct 29 22:49:27.359378 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:27.359323 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-w67ch"] Oct 29 22:49:27.363373 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:27.363353 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"echo-simulator-kserve-self-signed-certs\"" Oct 29 22:49:27.408624 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:27.408587 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:49:27.463727 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:27.463696 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/35c5b0e2-86a7-4408-83e3-1799bf6d66c8-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-w67ch\" (UID: \"35c5b0e2-86a7-4408-83e3-1799bf6d66c8\") " pod="llm/echo-simulator-kserve-64b6f8f96c-w67ch" Oct 29 22:49:27.463882 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:27.463757 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/35c5b0e2-86a7-4408-83e3-1799bf6d66c8-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-w67ch\" (UID: \"35c5b0e2-86a7-4408-83e3-1799bf6d66c8\") " pod="llm/echo-simulator-kserve-64b6f8f96c-w67ch" Oct 29 22:49:27.463932 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:27.463892 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35c5b0e2-86a7-4408-83e3-1799bf6d66c8-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-w67ch\" (UID: \"35c5b0e2-86a7-4408-83e3-1799bf6d66c8\") " pod="llm/echo-simulator-kserve-64b6f8f96c-w67ch" Oct 29 22:49:27.463976 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:27.463937 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/35c5b0e2-86a7-4408-83e3-1799bf6d66c8-home\") pod \"echo-simulator-kserve-64b6f8f96c-w67ch\" (UID: \"35c5b0e2-86a7-4408-83e3-1799bf6d66c8\") " pod="llm/echo-simulator-kserve-64b6f8f96c-w67ch" Oct 29 22:49:27.464012 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:27.463994 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/35c5b0e2-86a7-4408-83e3-1799bf6d66c8-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-w67ch\" (UID: \"35c5b0e2-86a7-4408-83e3-1799bf6d66c8\") " pod="llm/echo-simulator-kserve-64b6f8f96c-w67ch" Oct 29 22:49:27.464056 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:27.464024 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hd75w\" (UniqueName: \"kubernetes.io/projected/35c5b0e2-86a7-4408-83e3-1799bf6d66c8-kube-api-access-hd75w\") pod \"echo-simulator-kserve-64b6f8f96c-w67ch\" (UID: \"35c5b0e2-86a7-4408-83e3-1799bf6d66c8\") " pod="llm/echo-simulator-kserve-64b6f8f96c-w67ch" Oct 29 22:49:27.565400 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:27.565366 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/35c5b0e2-86a7-4408-83e3-1799bf6d66c8-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-w67ch\" (UID: \"35c5b0e2-86a7-4408-83e3-1799bf6d66c8\") " pod="llm/echo-simulator-kserve-64b6f8f96c-w67ch" Oct 29 22:49:27.565400 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:27.565402 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hd75w\" (UniqueName: \"kubernetes.io/projected/35c5b0e2-86a7-4408-83e3-1799bf6d66c8-kube-api-access-hd75w\") pod \"echo-simulator-kserve-64b6f8f96c-w67ch\" (UID: \"35c5b0e2-86a7-4408-83e3-1799bf6d66c8\") " pod="llm/echo-simulator-kserve-64b6f8f96c-w67ch" Oct 29 22:49:27.565635 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:27.565491 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/35c5b0e2-86a7-4408-83e3-1799bf6d66c8-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-w67ch\" (UID: \"35c5b0e2-86a7-4408-83e3-1799bf6d66c8\") " pod="llm/echo-simulator-kserve-64b6f8f96c-w67ch" Oct 29 22:49:27.565635 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:27.565542 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/35c5b0e2-86a7-4408-83e3-1799bf6d66c8-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-w67ch\" (UID: \"35c5b0e2-86a7-4408-83e3-1799bf6d66c8\") " pod="llm/echo-simulator-kserve-64b6f8f96c-w67ch" Oct 29 22:49:27.565635 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:27.565594 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35c5b0e2-86a7-4408-83e3-1799bf6d66c8-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-w67ch\" (UID: \"35c5b0e2-86a7-4408-83e3-1799bf6d66c8\") " pod="llm/echo-simulator-kserve-64b6f8f96c-w67ch" Oct 29 22:49:27.565795 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:27.565635 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/35c5b0e2-86a7-4408-83e3-1799bf6d66c8-home\") pod \"echo-simulator-kserve-64b6f8f96c-w67ch\" (UID: \"35c5b0e2-86a7-4408-83e3-1799bf6d66c8\") " pod="llm/echo-simulator-kserve-64b6f8f96c-w67ch" Oct 29 22:49:27.565996 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:27.565968 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/35c5b0e2-86a7-4408-83e3-1799bf6d66c8-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-w67ch\" (UID: \"35c5b0e2-86a7-4408-83e3-1799bf6d66c8\") " pod="llm/echo-simulator-kserve-64b6f8f96c-w67ch" Oct 29 22:49:27.565996 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:27.565985 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35c5b0e2-86a7-4408-83e3-1799bf6d66c8-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-w67ch\" (UID: \"35c5b0e2-86a7-4408-83e3-1799bf6d66c8\") " pod="llm/echo-simulator-kserve-64b6f8f96c-w67ch" Oct 29 22:49:27.566173 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:27.566062 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/35c5b0e2-86a7-4408-83e3-1799bf6d66c8-home\") pod \"echo-simulator-kserve-64b6f8f96c-w67ch\" (UID: \"35c5b0e2-86a7-4408-83e3-1799bf6d66c8\") " pod="llm/echo-simulator-kserve-64b6f8f96c-w67ch" Oct 29 22:49:27.567821 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:27.567801 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/35c5b0e2-86a7-4408-83e3-1799bf6d66c8-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-w67ch\" (UID: \"35c5b0e2-86a7-4408-83e3-1799bf6d66c8\") " pod="llm/echo-simulator-kserve-64b6f8f96c-w67ch" Oct 29 22:49:27.567935 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:27.567919 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/35c5b0e2-86a7-4408-83e3-1799bf6d66c8-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-w67ch\" (UID: \"35c5b0e2-86a7-4408-83e3-1799bf6d66c8\") " pod="llm/echo-simulator-kserve-64b6f8f96c-w67ch" Oct 29 22:49:27.589130 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:27.589082 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hd75w\" (UniqueName: \"kubernetes.io/projected/35c5b0e2-86a7-4408-83e3-1799bf6d66c8-kube-api-access-hd75w\") pod \"echo-simulator-kserve-64b6f8f96c-w67ch\" (UID: \"35c5b0e2-86a7-4408-83e3-1799bf6d66c8\") " pod="llm/echo-simulator-kserve-64b6f8f96c-w67ch" Oct 29 22:49:27.660195 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:27.660173 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-w67ch" Oct 29 22:49:27.959705 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:27.959679 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-w67ch"] Oct 29 22:49:27.961773 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:49:27.961745 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35c5b0e2_86a7_4408_83e3_1799bf6d66c8.slice/crio-bbb20baeb44b1d93b863dbde4f5d2c4a1b836f257b04ca70d39a4ea9ee0a378e WatchSource:0}: Error finding container bbb20baeb44b1d93b863dbde4f5d2c4a1b836f257b04ca70d39a4ea9ee0a378e: Status 404 returned error can't find the container with id bbb20baeb44b1d93b863dbde4f5d2c4a1b836f257b04ca70d39a4ea9ee0a378e Oct 29 22:49:28.342896 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:28.342824 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-w67ch" event={"ID":"35c5b0e2-86a7-4408-83e3-1799bf6d66c8","Type":"ContainerStarted","Data":"bbb20baeb44b1d93b863dbde4f5d2c4a1b836f257b04ca70d39a4ea9ee0a378e"} Oct 29 22:49:28.540933 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:28.540886 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v2tfm"] Oct 29 22:49:30.011911 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:30.011878 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-2wlhw"] Oct 29 22:49:30.016915 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:30.016888 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-2wlhw" Oct 29 22:49:30.040006 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:30.039977 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Oct 29 22:49:30.043773 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:30.043748 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-2wlhw"] Oct 29 22:49:30.054291 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:30.054267 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-vp6zb" Oct 29 22:49:30.066745 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:30.066707 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-vp6zb" Oct 29 22:49:30.093019 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:30.092986 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8c5a5036-5eab-4356-833d-ffde96608b46-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-2wlhw\" (UID: \"8c5a5036-5eab-4356-833d-ffde96608b46\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-2wlhw" Oct 29 22:49:30.093127 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:30.093048 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pj7gw\" (UniqueName: \"kubernetes.io/projected/8c5a5036-5eab-4356-833d-ffde96608b46-kube-api-access-pj7gw\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-2wlhw\" (UID: \"8c5a5036-5eab-4356-833d-ffde96608b46\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-2wlhw" Oct 29 22:49:30.093172 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:30.093144 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8c5a5036-5eab-4356-833d-ffde96608b46-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-2wlhw\" (UID: \"8c5a5036-5eab-4356-833d-ffde96608b46\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-2wlhw" Oct 29 22:49:30.093212 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:30.093188 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8c5a5036-5eab-4356-833d-ffde96608b46-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-2wlhw\" (UID: \"8c5a5036-5eab-4356-833d-ffde96608b46\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-2wlhw" Oct 29 22:49:30.093255 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:30.093234 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c5a5036-5eab-4356-833d-ffde96608b46-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-2wlhw\" (UID: \"8c5a5036-5eab-4356-833d-ffde96608b46\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-2wlhw" Oct 29 22:49:30.093359 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:30.093322 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c5a5036-5eab-4356-833d-ffde96608b46-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-2wlhw\" (UID: \"8c5a5036-5eab-4356-833d-ffde96608b46\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-2wlhw" Oct 29 22:49:30.095505 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:30.095481 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:49:30.193982 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:30.193956 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8c5a5036-5eab-4356-833d-ffde96608b46-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-2wlhw\" (UID: \"8c5a5036-5eab-4356-833d-ffde96608b46\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-2wlhw" Oct 29 22:49:30.194121 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:30.194009 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pj7gw\" (UniqueName: \"kubernetes.io/projected/8c5a5036-5eab-4356-833d-ffde96608b46-kube-api-access-pj7gw\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-2wlhw\" (UID: \"8c5a5036-5eab-4356-833d-ffde96608b46\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-2wlhw" Oct 29 22:49:30.194194 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:30.194122 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8c5a5036-5eab-4356-833d-ffde96608b46-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-2wlhw\" (UID: \"8c5a5036-5eab-4356-833d-ffde96608b46\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-2wlhw" Oct 29 22:49:30.194194 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:30.194162 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8c5a5036-5eab-4356-833d-ffde96608b46-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-2wlhw\" (UID: \"8c5a5036-5eab-4356-833d-ffde96608b46\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-2wlhw" Oct 29 22:49:30.194316 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:30.194210 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c5a5036-5eab-4356-833d-ffde96608b46-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-2wlhw\" (UID: \"8c5a5036-5eab-4356-833d-ffde96608b46\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-2wlhw" Oct 29 22:49:30.194316 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:30.194310 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c5a5036-5eab-4356-833d-ffde96608b46-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-2wlhw\" (UID: \"8c5a5036-5eab-4356-833d-ffde96608b46\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-2wlhw" Oct 29 22:49:30.194504 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:30.194401 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8c5a5036-5eab-4356-833d-ffde96608b46-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-2wlhw\" (UID: \"8c5a5036-5eab-4356-833d-ffde96608b46\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-2wlhw" Oct 29 22:49:30.194543 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:30.194509 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8c5a5036-5eab-4356-833d-ffde96608b46-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-2wlhw\" (UID: \"8c5a5036-5eab-4356-833d-ffde96608b46\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-2wlhw" Oct 29 22:49:30.194619 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:30.194600 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c5a5036-5eab-4356-833d-ffde96608b46-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-2wlhw\" (UID: \"8c5a5036-5eab-4356-833d-ffde96608b46\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-2wlhw" Oct 29 22:49:30.196444 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:30.196426 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8c5a5036-5eab-4356-833d-ffde96608b46-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-2wlhw\" (UID: \"8c5a5036-5eab-4356-833d-ffde96608b46\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-2wlhw" Oct 29 22:49:30.196792 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:30.196773 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c5a5036-5eab-4356-833d-ffde96608b46-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-2wlhw\" (UID: \"8c5a5036-5eab-4356-833d-ffde96608b46\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-2wlhw" Oct 29 22:49:30.216851 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:30.216833 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pj7gw\" (UniqueName: \"kubernetes.io/projected/8c5a5036-5eab-4356-833d-ffde96608b46-kube-api-access-pj7gw\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-2wlhw\" (UID: \"8c5a5036-5eab-4356-833d-ffde96608b46\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-2wlhw" Oct 29 22:49:30.331511 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:30.331436 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-2wlhw" Oct 29 22:49:30.358727 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:30.358690 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-w67ch" event={"ID":"35c5b0e2-86a7-4408-83e3-1799bf6d66c8","Type":"ContainerStarted","Data":"528d7e06d9a33c87afc4beeb5d4c0d29bdb7ff1466ada856c65035599abeacd4"} Oct 29 22:49:30.370299 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:30.370273 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-vp6zb" Oct 29 22:49:30.398876 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:30.397743 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/echo-simulator-kserve-64b6f8f96c-w67ch" podStartSLOduration=1.943728371 podStartE2EDuration="3.397727641s" totalImagesPullingTime="1.45399927s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:49:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:49:30.396741711 +0000 UTC m=+951.179955164" watchObservedRunningTime="2026-10-29 22:49:30.397727641 +0000 UTC m=+951.180941069" Oct 29 22:49:30.631525 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:30.631502 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-2wlhw"] Oct 29 22:49:30.633189 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:49:30.633163 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c5a5036_5eab_4356_833d_ffde96608b46.slice/crio-900cb641fd8bc49035ed67e2dbc15fd37c181af2fae1556a58645258a47f1e1b WatchSource:0}: Error finding container 900cb641fd8bc49035ed67e2dbc15fd37c181af2fae1556a58645258a47f1e1b: Status 404 returned error can't find the container with id 900cb641fd8bc49035ed67e2dbc15fd37c181af2fae1556a58645258a47f1e1b Oct 29 22:49:30.660774 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:30.660752 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/echo-simulator-kserve-64b6f8f96c-w67ch" Oct 29 22:49:30.702809 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:30.702780 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:49:31.277403 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:31.277359 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:49:31.366100 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:31.366066 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-2wlhw" event={"ID":"8c5a5036-5eab-4356-833d-ffde96608b46","Type":"ContainerStarted","Data":"ecd5bd7c37dbe17cd543d5e22adb429334ab2058e98af3ac110b52d91fe38c20"} Oct 29 22:49:31.366269 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:31.366124 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-2wlhw" event={"ID":"8c5a5036-5eab-4356-833d-ffde96608b46","Type":"ContainerStarted","Data":"900cb641fd8bc49035ed67e2dbc15fd37c181af2fae1556a58645258a47f1e1b"} Oct 29 22:49:31.404630 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:31.404581 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-2wlhw" podStartSLOduration=2.22756276 podStartE2EDuration="2.404567689s" totalImagesPullingTime="177.004929ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:49:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:49:31.403611218 +0000 UTC m=+952.186824644" watchObservedRunningTime="2026-10-29 22:49:31.404567689 +0000 UTC m=+952.187781132" Oct 29 22:49:32.332452 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:32.332419 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-2wlhw" Oct 29 22:49:33.553367 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:33.553312 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-jqfzk" Oct 29 22:49:33.566009 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:33.565985 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-jqfzk" Oct 29 22:49:34.395618 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:34.395591 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-jqfzk" Oct 29 22:49:37.443970 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:37.443938 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v2tfm"] Oct 29 22:49:37.661041 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:37.661003 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/echo-simulator-kserve-64b6f8f96c-w67ch" Oct 29 22:49:37.673693 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:37.673666 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/echo-simulator-kserve-64b6f8f96c-w67ch" Oct 29 22:49:38.417778 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:38.417750 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/echo-simulator-kserve-64b6f8f96c-w67ch" Oct 29 22:49:40.332095 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:40.332061 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-2wlhw" Oct 29 22:49:40.344881 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:40.344856 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-2wlhw" Oct 29 22:49:40.428028 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:40.428000 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-2wlhw" Oct 29 22:49:40.925555 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:40.925518 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-987xq"] Oct 29 22:49:40.930672 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:40.930650 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-987xq" Oct 29 22:49:40.943416 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:40.943379 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Oct 29 22:49:40.951401 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:40.951376 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-987xq"] Oct 29 22:49:41.007813 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:41.007785 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21dce295-63ce-4d61-8cfa-86669417b525-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-987xq\" (UID: \"21dce295-63ce-4d61-8cfa-86669417b525\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-987xq" Oct 29 22:49:41.007983 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:41.007815 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21dce295-63ce-4d61-8cfa-86669417b525-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-987xq\" (UID: \"21dce295-63ce-4d61-8cfa-86669417b525\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-987xq" Oct 29 22:49:41.007983 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:41.007839 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21dce295-63ce-4d61-8cfa-86669417b525-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-987xq\" (UID: \"21dce295-63ce-4d61-8cfa-86669417b525\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-987xq" Oct 29 22:49:41.007983 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:41.007856 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21dce295-63ce-4d61-8cfa-86669417b525-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-987xq\" (UID: \"21dce295-63ce-4d61-8cfa-86669417b525\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-987xq" Oct 29 22:49:41.007983 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:41.007952 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21dce295-63ce-4d61-8cfa-86669417b525-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-987xq\" (UID: \"21dce295-63ce-4d61-8cfa-86669417b525\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-987xq" Oct 29 22:49:41.008202 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:41.008115 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9pkxv\" (UniqueName: \"kubernetes.io/projected/21dce295-63ce-4d61-8cfa-86669417b525-kube-api-access-9pkxv\") pod \"maas-default-gateway-openshift-default-6db586dd6c-987xq\" (UID: \"21dce295-63ce-4d61-8cfa-86669417b525\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-987xq" Oct 29 22:49:41.008202 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:41.008160 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21dce295-63ce-4d61-8cfa-86669417b525-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-987xq\" (UID: \"21dce295-63ce-4d61-8cfa-86669417b525\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-987xq" Oct 29 22:49:41.008286 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:41.008256 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21dce295-63ce-4d61-8cfa-86669417b525-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-987xq\" (UID: \"21dce295-63ce-4d61-8cfa-86669417b525\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-987xq" Oct 29 22:49:41.008323 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:41.008285 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21dce295-63ce-4d61-8cfa-86669417b525-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-987xq\" (UID: \"21dce295-63ce-4d61-8cfa-86669417b525\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-987xq" Oct 29 22:49:41.026883 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:41.026862 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-2tp5f\"" Oct 29 22:49:41.109360 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:41.109316 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21dce295-63ce-4d61-8cfa-86669417b525-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-987xq\" (UID: \"21dce295-63ce-4d61-8cfa-86669417b525\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-987xq" Oct 29 22:49:41.109506 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:41.109365 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21dce295-63ce-4d61-8cfa-86669417b525-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-987xq\" (UID: \"21dce295-63ce-4d61-8cfa-86669417b525\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-987xq" Oct 29 22:49:41.109506 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:41.109393 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21dce295-63ce-4d61-8cfa-86669417b525-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-987xq\" (UID: \"21dce295-63ce-4d61-8cfa-86669417b525\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-987xq" Oct 29 22:49:41.109506 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:41.109421 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21dce295-63ce-4d61-8cfa-86669417b525-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-987xq\" (UID: \"21dce295-63ce-4d61-8cfa-86669417b525\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-987xq" Oct 29 22:49:41.109506 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:41.109449 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21dce295-63ce-4d61-8cfa-86669417b525-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-987xq\" (UID: \"21dce295-63ce-4d61-8cfa-86669417b525\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-987xq" Oct 29 22:49:41.109704 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:41.109565 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pkxv\" (UniqueName: \"kubernetes.io/projected/21dce295-63ce-4d61-8cfa-86669417b525-kube-api-access-9pkxv\") pod \"maas-default-gateway-openshift-default-6db586dd6c-987xq\" (UID: \"21dce295-63ce-4d61-8cfa-86669417b525\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-987xq" Oct 29 22:49:41.109704 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:41.109606 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21dce295-63ce-4d61-8cfa-86669417b525-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-987xq\" (UID: \"21dce295-63ce-4d61-8cfa-86669417b525\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-987xq" Oct 29 22:49:41.109704 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:41.109678 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21dce295-63ce-4d61-8cfa-86669417b525-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-987xq\" (UID: \"21dce295-63ce-4d61-8cfa-86669417b525\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-987xq" Oct 29 22:49:41.109850 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:41.109706 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21dce295-63ce-4d61-8cfa-86669417b525-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-987xq\" (UID: \"21dce295-63ce-4d61-8cfa-86669417b525\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-987xq" Oct 29 22:49:41.109850 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:41.109786 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21dce295-63ce-4d61-8cfa-86669417b525-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-987xq\" (UID: \"21dce295-63ce-4d61-8cfa-86669417b525\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-987xq" Oct 29 22:49:41.109996 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:41.109973 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21dce295-63ce-4d61-8cfa-86669417b525-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-987xq\" (UID: \"21dce295-63ce-4d61-8cfa-86669417b525\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-987xq" Oct 29 22:49:41.110094 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:41.110079 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21dce295-63ce-4d61-8cfa-86669417b525-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-987xq\" (UID: \"21dce295-63ce-4d61-8cfa-86669417b525\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-987xq" Oct 29 22:49:41.110208 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:41.110183 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21dce295-63ce-4d61-8cfa-86669417b525-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-987xq\" (UID: \"21dce295-63ce-4d61-8cfa-86669417b525\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-987xq" Oct 29 22:49:41.110300 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:41.110276 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21dce295-63ce-4d61-8cfa-86669417b525-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-987xq\" (UID: \"21dce295-63ce-4d61-8cfa-86669417b525\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-987xq" Oct 29 22:49:41.112017 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:41.111989 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21dce295-63ce-4d61-8cfa-86669417b525-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-987xq\" (UID: \"21dce295-63ce-4d61-8cfa-86669417b525\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-987xq" Oct 29 22:49:41.112319 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:41.112302 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21dce295-63ce-4d61-8cfa-86669417b525-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-987xq\" (UID: \"21dce295-63ce-4d61-8cfa-86669417b525\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-987xq" Oct 29 22:49:41.130482 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:41.130451 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21dce295-63ce-4d61-8cfa-86669417b525-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-987xq\" (UID: \"21dce295-63ce-4d61-8cfa-86669417b525\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-987xq" Oct 29 22:49:41.130581 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:41.130551 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pkxv\" (UniqueName: \"kubernetes.io/projected/21dce295-63ce-4d61-8cfa-86669417b525-kube-api-access-9pkxv\") pod \"maas-default-gateway-openshift-default-6db586dd6c-987xq\" (UID: \"21dce295-63ce-4d61-8cfa-86669417b525\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-987xq" Oct 29 22:49:41.246637 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:41.246606 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-987xq" Oct 29 22:49:41.568618 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:41.568593 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-987xq"] Oct 29 22:49:41.570578 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:49:41.570554 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21dce295_63ce_4d61_8cfa_86669417b525.slice/crio-4fdefaa28aaf8a1cf1e4257794518cb070ca79cf6526c9f95dade2a987e9f7b4 WatchSource:0}: Error finding container 4fdefaa28aaf8a1cf1e4257794518cb070ca79cf6526c9f95dade2a987e9f7b4: Status 404 returned error can't find the container with id 4fdefaa28aaf8a1cf1e4257794518cb070ca79cf6526c9f95dade2a987e9f7b4 Oct 29 22:49:42.434172 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:42.434133 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-987xq" event={"ID":"21dce295-63ce-4d61-8cfa-86669417b525","Type":"ContainerStarted","Data":"4fdefaa28aaf8a1cf1e4257794518cb070ca79cf6526c9f95dade2a987e9f7b4"} Oct 29 22:49:44.068015 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:44.067984 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:49:44.068279 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:44.068049 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:49:44.068279 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:44.068079 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:49:44.449113 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:44.449058 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-987xq" event={"ID":"21dce295-63ce-4d61-8cfa-86669417b525","Type":"ContainerStarted","Data":"e58c63cf8d89bd25e95ba2b9afb495e0840adae3c8ef59a7d9dfce95d9fadf59"} Oct 29 22:49:44.514354 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:44.512393 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-987xq" podStartSLOduration=2.018058644 podStartE2EDuration="4.512374108s" totalImagesPullingTime="2.494315464s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:49:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:49:44.500606422 +0000 UTC m=+965.283819880" watchObservedRunningTime="2026-10-29 22:49:44.512374108 +0000 UTC m=+965.295587535" Oct 29 22:49:45.247397 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:45.247361 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-987xq" Oct 29 22:49:45.249005 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:45.248979 5057 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-987xq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Oct 29 22:49:45.249117 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:45.249035 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-987xq" podUID="21dce295-63ce-4d61-8cfa-86669417b525" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Oct 29 22:49:46.247429 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:46.247395 5057 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-987xq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Oct 29 22:49:46.247796 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:46.247447 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-987xq" podUID="21dce295-63ce-4d61-8cfa-86669417b525" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Oct 29 22:49:47.247533 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:47.247494 5057 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-987xq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Oct 29 22:49:47.247921 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:47.247549 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-987xq" podUID="21dce295-63ce-4d61-8cfa-86669417b525" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Oct 29 22:49:48.247073 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:48.247032 5057 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-987xq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Oct 29 22:49:48.247269 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:48.247091 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-987xq" podUID="21dce295-63ce-4d61-8cfa-86669417b525" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Oct 29 22:49:49.029249 ip-10-0-140-74 kubenswrapper[5057]: E1029 22:49:49.029214 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d2dnn" containerName="extractor" Oct 29 22:49:49.246998 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:49.246963 5057 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-987xq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Oct 29 22:49:49.247167 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:49.247027 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-987xq" podUID="21dce295-63ce-4d61-8cfa-86669417b525" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Oct 29 22:49:50.247099 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:50.247064 5057 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-987xq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Oct 29 22:49:50.247504 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:50.247126 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-987xq" podUID="21dce295-63ce-4d61-8cfa-86669417b525" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Oct 29 22:49:51.247786 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:51.247753 5057 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-987xq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Oct 29 22:49:51.248179 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:51.247804 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-987xq" podUID="21dce295-63ce-4d61-8cfa-86669417b525" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Oct 29 22:49:51.570490 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:51.570402 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:49:52.679523 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:52.679474 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-987xq" Oct 29 22:49:52.680317 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:52.680290 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-987xq" Oct 29 22:49:52.682001 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:52.681978 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-987xq" Oct 29 22:49:53.106374 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:53.106344 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:49:53.228259 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:53.228228 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:49:55.991081 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:55.991048 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-cm4wj"] Oct 29 22:49:55.995208 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:55.995186 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-cm4wj" Oct 29 22:49:56.027780 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:56.027751 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-cm4wj"] Oct 29 22:49:56.093957 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:56.093921 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e861691-eac4-4f06-a6aa-b6553efdd358-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-cm4wj\" (UID: \"4e861691-eac4-4f06-a6aa-b6553efdd358\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-cm4wj" Oct 29 22:49:56.093957 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:56.093956 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e861691-eac4-4f06-a6aa-b6553efdd358-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-cm4wj\" (UID: \"4e861691-eac4-4f06-a6aa-b6553efdd358\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-cm4wj" Oct 29 22:49:56.094142 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:56.093974 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e861691-eac4-4f06-a6aa-b6553efdd358-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-cm4wj\" (UID: \"4e861691-eac4-4f06-a6aa-b6553efdd358\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-cm4wj" Oct 29 22:49:56.094142 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:56.093993 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e861691-eac4-4f06-a6aa-b6553efdd358-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-cm4wj\" (UID: \"4e861691-eac4-4f06-a6aa-b6553efdd358\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-cm4wj" Oct 29 22:49:56.094142 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:56.094110 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhlvp\" (UniqueName: \"kubernetes.io/projected/4e861691-eac4-4f06-a6aa-b6553efdd358-kube-api-access-qhlvp\") pod \"maas-default-gateway-openshift-default-6db586dd6c-cm4wj\" (UID: \"4e861691-eac4-4f06-a6aa-b6553efdd358\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-cm4wj" Oct 29 22:49:56.094376 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:56.094160 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e861691-eac4-4f06-a6aa-b6553efdd358-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-cm4wj\" (UID: \"4e861691-eac4-4f06-a6aa-b6553efdd358\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-cm4wj" Oct 29 22:49:56.094376 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:56.094286 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e861691-eac4-4f06-a6aa-b6553efdd358-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-cm4wj\" (UID: \"4e861691-eac4-4f06-a6aa-b6553efdd358\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-cm4wj" Oct 29 22:49:56.094492 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:56.094448 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e861691-eac4-4f06-a6aa-b6553efdd358-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-cm4wj\" (UID: \"4e861691-eac4-4f06-a6aa-b6553efdd358\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-cm4wj" Oct 29 22:49:56.094492 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:56.094481 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e861691-eac4-4f06-a6aa-b6553efdd358-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-cm4wj\" (UID: \"4e861691-eac4-4f06-a6aa-b6553efdd358\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-cm4wj" Oct 29 22:49:56.195615 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:56.195584 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e861691-eac4-4f06-a6aa-b6553efdd358-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-cm4wj\" (UID: \"4e861691-eac4-4f06-a6aa-b6553efdd358\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-cm4wj" Oct 29 22:49:56.195814 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:56.195639 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhlvp\" (UniqueName: \"kubernetes.io/projected/4e861691-eac4-4f06-a6aa-b6553efdd358-kube-api-access-qhlvp\") pod \"maas-default-gateway-openshift-default-6db586dd6c-cm4wj\" (UID: \"4e861691-eac4-4f06-a6aa-b6553efdd358\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-cm4wj" Oct 29 22:49:56.195814 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:56.195686 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e861691-eac4-4f06-a6aa-b6553efdd358-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-cm4wj\" (UID: \"4e861691-eac4-4f06-a6aa-b6553efdd358\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-cm4wj" Oct 29 22:49:56.195814 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:56.195726 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e861691-eac4-4f06-a6aa-b6553efdd358-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-cm4wj\" (UID: \"4e861691-eac4-4f06-a6aa-b6553efdd358\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-cm4wj" Oct 29 22:49:56.195814 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:56.195788 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e861691-eac4-4f06-a6aa-b6553efdd358-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-cm4wj\" (UID: \"4e861691-eac4-4f06-a6aa-b6553efdd358\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-cm4wj" Oct 29 22:49:56.195814 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:56.195804 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e861691-eac4-4f06-a6aa-b6553efdd358-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-cm4wj\" (UID: \"4e861691-eac4-4f06-a6aa-b6553efdd358\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-cm4wj" Oct 29 22:49:56.196074 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:56.195840 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e861691-eac4-4f06-a6aa-b6553efdd358-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-cm4wj\" (UID: \"4e861691-eac4-4f06-a6aa-b6553efdd358\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-cm4wj" Oct 29 22:49:56.196074 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:56.196022 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e861691-eac4-4f06-a6aa-b6553efdd358-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-cm4wj\" (UID: \"4e861691-eac4-4f06-a6aa-b6553efdd358\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-cm4wj" Oct 29 22:49:56.196178 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:56.196077 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e861691-eac4-4f06-a6aa-b6553efdd358-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-cm4wj\" (UID: \"4e861691-eac4-4f06-a6aa-b6553efdd358\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-cm4wj" Oct 29 22:49:56.196291 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:56.196236 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e861691-eac4-4f06-a6aa-b6553efdd358-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-cm4wj\" (UID: \"4e861691-eac4-4f06-a6aa-b6553efdd358\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-cm4wj" Oct 29 22:49:56.196667 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:56.196569 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e861691-eac4-4f06-a6aa-b6553efdd358-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-cm4wj\" (UID: \"4e861691-eac4-4f06-a6aa-b6553efdd358\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-cm4wj" Oct 29 22:49:56.196667 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:56.196621 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e861691-eac4-4f06-a6aa-b6553efdd358-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-cm4wj\" (UID: \"4e861691-eac4-4f06-a6aa-b6553efdd358\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-cm4wj" Oct 29 22:49:56.196667 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:56.196622 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e861691-eac4-4f06-a6aa-b6553efdd358-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-cm4wj\" (UID: \"4e861691-eac4-4f06-a6aa-b6553efdd358\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-cm4wj" Oct 29 22:49:56.196853 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:56.196832 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e861691-eac4-4f06-a6aa-b6553efdd358-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-cm4wj\" (UID: \"4e861691-eac4-4f06-a6aa-b6553efdd358\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-cm4wj" Oct 29 22:49:56.197977 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:56.197953 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e861691-eac4-4f06-a6aa-b6553efdd358-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-cm4wj\" (UID: \"4e861691-eac4-4f06-a6aa-b6553efdd358\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-cm4wj" Oct 29 22:49:56.198311 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:56.198293 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e861691-eac4-4f06-a6aa-b6553efdd358-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-cm4wj\" (UID: \"4e861691-eac4-4f06-a6aa-b6553efdd358\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-cm4wj" Oct 29 22:49:56.219439 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:56.219408 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e861691-eac4-4f06-a6aa-b6553efdd358-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-cm4wj\" (UID: \"4e861691-eac4-4f06-a6aa-b6553efdd358\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-cm4wj" Oct 29 22:49:56.219570 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:56.219543 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhlvp\" (UniqueName: \"kubernetes.io/projected/4e861691-eac4-4f06-a6aa-b6553efdd358-kube-api-access-qhlvp\") pod \"maas-default-gateway-openshift-default-6db586dd6c-cm4wj\" (UID: \"4e861691-eac4-4f06-a6aa-b6553efdd358\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-cm4wj" Oct 29 22:49:56.311148 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:56.311066 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-cm4wj" Oct 29 22:49:56.635162 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:56.635135 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-cm4wj"] Oct 29 22:49:56.637240 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:49:56.637214 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e861691_eac4_4f06_a6aa_b6553efdd358.slice/crio-0f2b89cce68b4a2dfc5e0ad3d5e3925ed0f065498a7b62d8093ea522e2c45f09 WatchSource:0}: Error finding container 0f2b89cce68b4a2dfc5e0ad3d5e3925ed0f065498a7b62d8093ea522e2c45f09: Status 404 returned error can't find the container with id 0f2b89cce68b4a2dfc5e0ad3d5e3925ed0f065498a7b62d8093ea522e2c45f09 Oct 29 22:49:56.640511 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:56.640474 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:49:56.640601 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:56.640582 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:49:56.640659 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:56.640631 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:49:57.532589 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:57.532544 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-cm4wj" event={"ID":"4e861691-eac4-4f06-a6aa-b6553efdd358","Type":"ContainerStarted","Data":"a7018331ff913a04921035ea6e43293b1040114978909d2dc906a22bd9e40423"} Oct 29 22:49:57.532991 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:57.532636 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-cm4wj" event={"ID":"4e861691-eac4-4f06-a6aa-b6553efdd358","Type":"ContainerStarted","Data":"0f2b89cce68b4a2dfc5e0ad3d5e3925ed0f065498a7b62d8093ea522e2c45f09"} Oct 29 22:49:57.598352 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:57.593253 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-cm4wj" podStartSLOduration=2.593231867 podStartE2EDuration="2.593231867s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:49:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:49:57.583324604 +0000 UTC m=+978.366538030" watchObservedRunningTime="2026-10-29 22:49:57.593231867 +0000 UTC m=+978.376445295" Oct 29 22:49:58.311653 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:58.311615 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-cm4wj" Oct 29 22:49:58.313428 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:58.313400 5057 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-cm4wj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Oct 29 22:49:58.313559 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:58.313455 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-cm4wj" podUID="4e861691-eac4-4f06-a6aa-b6553efdd358" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Oct 29 22:49:59.311981 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:59.311937 5057 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-cm4wj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Oct 29 22:49:59.312496 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:49:59.312003 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-cm4wj" podUID="4e861691-eac4-4f06-a6aa-b6553efdd358" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Oct 29 22:50:00.311783 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:00.311751 5057 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-cm4wj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Oct 29 22:50:00.311954 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:00.311803 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-cm4wj" podUID="4e861691-eac4-4f06-a6aa-b6553efdd358" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Oct 29 22:50:01.311811 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:01.311777 5057 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-cm4wj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Oct 29 22:50:01.312202 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:01.311843 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-cm4wj" podUID="4e861691-eac4-4f06-a6aa-b6553efdd358" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Oct 29 22:50:02.311853 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:02.311815 5057 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-cm4wj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Oct 29 22:50:02.312252 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:02.311895 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-cm4wj" podUID="4e861691-eac4-4f06-a6aa-b6553efdd358" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Oct 29 22:50:03.312178 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:03.312143 5057 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-cm4wj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Oct 29 22:50:03.312561 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:03.312197 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-cm4wj" podUID="4e861691-eac4-4f06-a6aa-b6553efdd358" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Oct 29 22:50:05.143057 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:05.143018 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-cm4wj" Oct 29 22:50:05.143569 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:05.143314 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-cm4wj" Oct 29 22:50:05.144224 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:05.144190 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-cm4wj" Oct 29 22:50:19.027104 ip-10-0-140-74 kubenswrapper[5057]: E1029 22:50:19.027067 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d2dnn" containerName="extractor" Oct 29 22:50:19.660215 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:19.660175 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:50:19.794556 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:19.794520 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:50:19.905429 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:19.905392 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:50:20.039685 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:20.039643 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:50:20.156383 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:20.155878 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:50:20.283401 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:20.283365 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:50:20.419449 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:20.419351 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:50:20.528631 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:20.528598 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:50:20.691742 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:20.691646 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:50:20.888667 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:20.888626 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:50:20.998165 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:20.998132 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:50:21.108850 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:21.108811 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:50:21.289847 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:21.289759 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:50:21.642359 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:21.642245 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:50:22.005580 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:22.005544 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:50:22.494517 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:22.494486 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:50:22.621658 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:22.621608 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:50:23.634202 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:23.634162 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:50:23.816308 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:23.816271 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:50:24.106706 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:24.106666 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:50:24.224117 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:24.224074 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:50:24.342771 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:24.342734 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:50:24.472438 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:24.472402 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:50:24.616867 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:24.616822 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:50:24.781424 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:24.781321 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:50:24.894572 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:24.894533 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:50:25.083827 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:25.083754 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-639067f9-74c94b54c8-qjsft"] Oct 29 22:50:25.092767 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:25.092743 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-639067f9-74c94b54c8-qjsft" Oct 29 22:50:25.119177 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:25.119144 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-639067f9-74c94b54c8-qjsft"] Oct 29 22:50:25.124898 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:25.124867 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-639067f9-serving-cert\"" Oct 29 22:50:25.171169 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:25.171132 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:50:25.223407 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:25.223373 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ee18842c-3a3f-4148-807f-09838ef6ccf9-maas-api-tls\") pod \"maas-api-e2e-migrate-639067f9-74c94b54c8-qjsft\" (UID: \"ee18842c-3a3f-4148-807f-09838ef6ccf9\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-639067f9-74c94b54c8-qjsft" Oct 29 22:50:25.223574 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:25.223453 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee18842c-3a3f-4148-807f-09838ef6ccf9-openshift-service-ca\") pod \"maas-api-e2e-migrate-639067f9-74c94b54c8-qjsft\" (UID: \"ee18842c-3a3f-4148-807f-09838ef6ccf9\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-639067f9-74c94b54c8-qjsft" Oct 29 22:50:25.223623 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:25.223575 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ee18842c-3a3f-4148-807f-09838ef6ccf9-metrics-certs\") pod \"maas-api-e2e-migrate-639067f9-74c94b54c8-qjsft\" (UID: \"ee18842c-3a3f-4148-807f-09838ef6ccf9\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-639067f9-74c94b54c8-qjsft" Oct 29 22:50:25.223745 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:25.223727 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5g8k\" (UniqueName: \"kubernetes.io/projected/ee18842c-3a3f-4148-807f-09838ef6ccf9-kube-api-access-v5g8k\") pod \"maas-api-e2e-migrate-639067f9-74c94b54c8-qjsft\" (UID: \"ee18842c-3a3f-4148-807f-09838ef6ccf9\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-639067f9-74c94b54c8-qjsft" Oct 29 22:50:25.284968 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:25.284929 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:50:25.324727 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:25.324695 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5g8k\" (UniqueName: \"kubernetes.io/projected/ee18842c-3a3f-4148-807f-09838ef6ccf9-kube-api-access-v5g8k\") pod \"maas-api-e2e-migrate-639067f9-74c94b54c8-qjsft\" (UID: \"ee18842c-3a3f-4148-807f-09838ef6ccf9\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-639067f9-74c94b54c8-qjsft" Oct 29 22:50:25.324882 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:25.324777 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ee18842c-3a3f-4148-807f-09838ef6ccf9-maas-api-tls\") pod \"maas-api-e2e-migrate-639067f9-74c94b54c8-qjsft\" (UID: \"ee18842c-3a3f-4148-807f-09838ef6ccf9\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-639067f9-74c94b54c8-qjsft" Oct 29 22:50:25.324882 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:25.324854 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee18842c-3a3f-4148-807f-09838ef6ccf9-openshift-service-ca\") pod \"maas-api-e2e-migrate-639067f9-74c94b54c8-qjsft\" (UID: \"ee18842c-3a3f-4148-807f-09838ef6ccf9\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-639067f9-74c94b54c8-qjsft" Oct 29 22:50:25.325006 ip-10-0-140-74 kubenswrapper[5057]: E1029 22:50:25.324924 5057 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-639067f9-serving-cert: secret "maas-api-e2e-migrate-639067f9-serving-cert" not found Oct 29 22:50:25.325006 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:25.324935 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ee18842c-3a3f-4148-807f-09838ef6ccf9-metrics-certs\") pod \"maas-api-e2e-migrate-639067f9-74c94b54c8-qjsft\" (UID: \"ee18842c-3a3f-4148-807f-09838ef6ccf9\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-639067f9-74c94b54c8-qjsft" Oct 29 22:50:25.325006 ip-10-0-140-74 kubenswrapper[5057]: E1029 22:50:25.324991 5057 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ee18842c-3a3f-4148-807f-09838ef6ccf9-maas-api-tls podName:ee18842c-3a3f-4148-807f-09838ef6ccf9 nodeName:}" failed. No retries permitted until 2026-10-29 22:50:25.824971508 +0000 UTC m=+1006.608184931 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ee18842c-3a3f-4148-807f-09838ef6ccf9-maas-api-tls") pod "maas-api-e2e-migrate-639067f9-74c94b54c8-qjsft" (UID: "ee18842c-3a3f-4148-807f-09838ef6ccf9") : secret "maas-api-e2e-migrate-639067f9-serving-cert" not found Oct 29 22:50:25.325730 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:25.325709 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee18842c-3a3f-4148-807f-09838ef6ccf9-openshift-service-ca\") pod \"maas-api-e2e-migrate-639067f9-74c94b54c8-qjsft\" (UID: \"ee18842c-3a3f-4148-807f-09838ef6ccf9\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-639067f9-74c94b54c8-qjsft" Oct 29 22:50:25.327599 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:25.327581 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ee18842c-3a3f-4148-807f-09838ef6ccf9-metrics-certs\") pod \"maas-api-e2e-migrate-639067f9-74c94b54c8-qjsft\" (UID: \"ee18842c-3a3f-4148-807f-09838ef6ccf9\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-639067f9-74c94b54c8-qjsft" Oct 29 22:50:25.349920 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:25.349857 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5g8k\" (UniqueName: \"kubernetes.io/projected/ee18842c-3a3f-4148-807f-09838ef6ccf9-kube-api-access-v5g8k\") pod \"maas-api-e2e-migrate-639067f9-74c94b54c8-qjsft\" (UID: \"ee18842c-3a3f-4148-807f-09838ef6ccf9\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-639067f9-74c94b54c8-qjsft" Oct 29 22:50:25.480047 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:25.480012 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:50:25.823045 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:25.822992 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:50:25.830943 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:25.830912 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ee18842c-3a3f-4148-807f-09838ef6ccf9-maas-api-tls\") pod \"maas-api-e2e-migrate-639067f9-74c94b54c8-qjsft\" (UID: \"ee18842c-3a3f-4148-807f-09838ef6ccf9\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-639067f9-74c94b54c8-qjsft" Oct 29 22:50:25.833681 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:25.833653 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ee18842c-3a3f-4148-807f-09838ef6ccf9-maas-api-tls\") pod \"maas-api-e2e-migrate-639067f9-74c94b54c8-qjsft\" (UID: \"ee18842c-3a3f-4148-807f-09838ef6ccf9\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-639067f9-74c94b54c8-qjsft" Oct 29 22:50:25.933192 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:25.933161 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:50:25.982044 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:25.982012 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-639067f9-588f8ccf99-89pzl"] Oct 29 22:50:25.985929 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:25.985910 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-639067f9-588f8ccf99-89pzl" Oct 29 22:50:26.003753 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:26.003715 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-639067f9-dockercfg-fq8d4\"" Oct 29 22:50:26.007027 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:26.006998 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-639067f9-588f8ccf99-89pzl"] Oct 29 22:50:26.019346 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:26.019308 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-639067f9\"" Oct 29 22:50:26.020540 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:26.020518 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-639067f9-74c94b54c8-qjsft" Oct 29 22:50:26.070871 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:26.070836 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-639067f9-6b77796fcb-plz6x"] Oct 29 22:50:26.078368 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:26.078288 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-639067f9-6b77796fcb-plz6x" Oct 29 22:50:26.096176 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:26.096146 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-639067f9-6b77796fcb-plz6x"] Oct 29 22:50:26.136230 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:26.136186 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31493bbc-acc0-423b-9968-3be0bca796b5-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-639067f9-588f8ccf99-89pzl\" (UID: \"31493bbc-acc0-423b-9968-3be0bca796b5\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-639067f9-588f8ccf99-89pzl" Oct 29 22:50:26.136413 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:26.136258 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-blphc\" (UniqueName: \"kubernetes.io/projected/31493bbc-acc0-423b-9968-3be0bca796b5-kube-api-access-blphc\") pod \"payload-pre-processing-e2e-migrate-639067f9-588f8ccf99-89pzl\" (UID: \"31493bbc-acc0-423b-9968-3be0bca796b5\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-639067f9-588f8ccf99-89pzl" Oct 29 22:50:26.237685 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:26.237656 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31493bbc-acc0-423b-9968-3be0bca796b5-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-639067f9-588f8ccf99-89pzl\" (UID: \"31493bbc-acc0-423b-9968-3be0bca796b5\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-639067f9-588f8ccf99-89pzl" Oct 29 22:50:26.237842 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:26.237704 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pp5g\" (UniqueName: \"kubernetes.io/projected/185a7421-26cf-46ae-a65d-ed283704904e-kube-api-access-7pp5g\") pod \"payload-processing-e2e-migrate-639067f9-6b77796fcb-plz6x\" (UID: \"185a7421-26cf-46ae-a65d-ed283704904e\") " pod="openshift-ingress/payload-processing-e2e-migrate-639067f9-6b77796fcb-plz6x" Oct 29 22:50:26.237842 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:26.237724 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blphc\" (UniqueName: \"kubernetes.io/projected/31493bbc-acc0-423b-9968-3be0bca796b5-kube-api-access-blphc\") pod \"payload-pre-processing-e2e-migrate-639067f9-588f8ccf99-89pzl\" (UID: \"31493bbc-acc0-423b-9968-3be0bca796b5\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-639067f9-588f8ccf99-89pzl" Oct 29 22:50:26.237937 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:26.237910 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/185a7421-26cf-46ae-a65d-ed283704904e-plugins-config-volume\") pod \"payload-processing-e2e-migrate-639067f9-6b77796fcb-plz6x\" (UID: \"185a7421-26cf-46ae-a65d-ed283704904e\") " pod="openshift-ingress/payload-processing-e2e-migrate-639067f9-6b77796fcb-plz6x" Oct 29 22:50:26.238590 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:26.238564 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31493bbc-acc0-423b-9968-3be0bca796b5-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-639067f9-588f8ccf99-89pzl\" (UID: \"31493bbc-acc0-423b-9968-3be0bca796b5\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-639067f9-588f8ccf99-89pzl" Oct 29 22:50:26.258826 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:26.258801 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-blphc\" (UniqueName: \"kubernetes.io/projected/31493bbc-acc0-423b-9968-3be0bca796b5-kube-api-access-blphc\") pod \"payload-pre-processing-e2e-migrate-639067f9-588f8ccf99-89pzl\" (UID: \"31493bbc-acc0-423b-9968-3be0bca796b5\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-639067f9-588f8ccf99-89pzl" Oct 29 22:50:26.304713 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:26.304683 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-639067f9-588f8ccf99-89pzl" Oct 29 22:50:26.339676 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:26.339588 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pp5g\" (UniqueName: \"kubernetes.io/projected/185a7421-26cf-46ae-a65d-ed283704904e-kube-api-access-7pp5g\") pod \"payload-processing-e2e-migrate-639067f9-6b77796fcb-plz6x\" (UID: \"185a7421-26cf-46ae-a65d-ed283704904e\") " pod="openshift-ingress/payload-processing-e2e-migrate-639067f9-6b77796fcb-plz6x" Oct 29 22:50:26.339762 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:26.339669 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/185a7421-26cf-46ae-a65d-ed283704904e-plugins-config-volume\") pod \"payload-processing-e2e-migrate-639067f9-6b77796fcb-plz6x\" (UID: \"185a7421-26cf-46ae-a65d-ed283704904e\") " pod="openshift-ingress/payload-processing-e2e-migrate-639067f9-6b77796fcb-plz6x" Oct 29 22:50:26.340034 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:26.339930 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-639067f9-74c94b54c8-qjsft"] Oct 29 22:50:26.340628 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:26.340601 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/185a7421-26cf-46ae-a65d-ed283704904e-plugins-config-volume\") pod \"payload-processing-e2e-migrate-639067f9-6b77796fcb-plz6x\" (UID: \"185a7421-26cf-46ae-a65d-ed283704904e\") " pod="openshift-ingress/payload-processing-e2e-migrate-639067f9-6b77796fcb-plz6x" Oct 29 22:50:26.343226 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:50:26.343198 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee18842c_3a3f_4148_807f_09838ef6ccf9.slice/crio-de4b8d9ec40116eb05a28706f2ef6b5374b6e7bc7f8411b9d4b622cd9bb2e2f9 WatchSource:0}: Error finding container de4b8d9ec40116eb05a28706f2ef6b5374b6e7bc7f8411b9d4b622cd9bb2e2f9: Status 404 returned error can't find the container with id de4b8d9ec40116eb05a28706f2ef6b5374b6e7bc7f8411b9d4b622cd9bb2e2f9 Oct 29 22:50:26.345162 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:26.345144 5057 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 29 22:50:26.361987 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:26.361886 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pp5g\" (UniqueName: \"kubernetes.io/projected/185a7421-26cf-46ae-a65d-ed283704904e-kube-api-access-7pp5g\") pod \"payload-processing-e2e-migrate-639067f9-6b77796fcb-plz6x\" (UID: \"185a7421-26cf-46ae-a65d-ed283704904e\") " pod="openshift-ingress/payload-processing-e2e-migrate-639067f9-6b77796fcb-plz6x" Oct 29 22:50:26.418686 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:26.418357 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-639067f9-6b77796fcb-plz6x" Oct 29 22:50:26.613450 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:26.613423 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-639067f9-588f8ccf99-89pzl"] Oct 29 22:50:26.615666 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:50:26.615633 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31493bbc_acc0_423b_9968_3be0bca796b5.slice/crio-aa97cb144251574f478405f6220e8c2a75f2225c4fe4ef1de074c2a1a769460f WatchSource:0}: Error finding container aa97cb144251574f478405f6220e8c2a75f2225c4fe4ef1de074c2a1a769460f: Status 404 returned error can't find the container with id aa97cb144251574f478405f6220e8c2a75f2225c4fe4ef1de074c2a1a769460f Oct 29 22:50:26.699193 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:26.699167 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-639067f9-588f8ccf99-89pzl" event={"ID":"31493bbc-acc0-423b-9968-3be0bca796b5","Type":"ContainerStarted","Data":"87c3a235f393e5b2eec80a4badf1b7023c9c05f17eb13b772474177ef4e331f1"} Oct 29 22:50:26.699322 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:26.699224 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-639067f9-588f8ccf99-89pzl" event={"ID":"31493bbc-acc0-423b-9968-3be0bca796b5","Type":"ContainerStarted","Data":"aa97cb144251574f478405f6220e8c2a75f2225c4fe4ef1de074c2a1a769460f"} Oct 29 22:50:26.699322 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:26.699284 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-639067f9-588f8ccf99-89pzl" Oct 29 22:50:26.700933 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:26.700916 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-639067f9-74c94b54c8-qjsft" event={"ID":"ee18842c-3a3f-4148-807f-09838ef6ccf9","Type":"ContainerStarted","Data":"de4b8d9ec40116eb05a28706f2ef6b5374b6e7bc7f8411b9d4b622cd9bb2e2f9"} Oct 29 22:50:26.725967 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:26.725943 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-639067f9-6b77796fcb-plz6x"] Oct 29 22:50:26.737875 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:26.737816 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-639067f9-588f8ccf99-89pzl" podStartSLOduration=1.737799182 podStartE2EDuration="1.737799182s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:50:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:50:26.734506737 +0000 UTC m=+1007.517720162" watchObservedRunningTime="2026-10-29 22:50:26.737799182 +0000 UTC m=+1007.521012609" Oct 29 22:50:26.896162 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:26.896128 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-31226c-6c695fb468-fkr4w"] Oct 29 22:50:26.900405 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:26.900385 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-31226c-6c695fb468-fkr4w" Oct 29 22:50:26.919531 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:26.919508 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-31226c-6c695fb468-fkr4w"] Oct 29 22:50:26.983210 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:26.983176 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:50:27.003267 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:27.003243 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w6-31226c-serving-cert\"" Oct 29 22:50:27.048066 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:27.048036 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3d4a1e8-5e7a-4894-811f-5bb731d83b0d-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-31226c-6c695fb468-fkr4w\" (UID: \"d3d4a1e8-5e7a-4894-811f-5bb731d83b0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-31226c-6c695fb468-fkr4w" Oct 29 22:50:27.048226 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:27.048095 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3d4a1e8-5e7a-4894-811f-5bb731d83b0d-maas-api-tls\") pod \"maas-api-e2e-worker-w6-31226c-6c695fb468-fkr4w\" (UID: \"d3d4a1e8-5e7a-4894-811f-5bb731d83b0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-31226c-6c695fb468-fkr4w" Oct 29 22:50:27.048284 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:27.048217 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d3d4a1e8-5e7a-4894-811f-5bb731d83b0d-metrics-certs\") pod \"maas-api-e2e-worker-w6-31226c-6c695fb468-fkr4w\" (UID: \"d3d4a1e8-5e7a-4894-811f-5bb731d83b0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-31226c-6c695fb468-fkr4w" Oct 29 22:50:27.048476 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:27.048452 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdrcq\" (UniqueName: \"kubernetes.io/projected/d3d4a1e8-5e7a-4894-811f-5bb731d83b0d-kube-api-access-cdrcq\") pod \"maas-api-e2e-worker-w6-31226c-6c695fb468-fkr4w\" (UID: \"d3d4a1e8-5e7a-4894-811f-5bb731d83b0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-31226c-6c695fb468-fkr4w" Oct 29 22:50:27.093448 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:27.093400 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:50:27.149845 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:27.149770 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdrcq\" (UniqueName: \"kubernetes.io/projected/d3d4a1e8-5e7a-4894-811f-5bb731d83b0d-kube-api-access-cdrcq\") pod \"maas-api-e2e-worker-w6-31226c-6c695fb468-fkr4w\" (UID: \"d3d4a1e8-5e7a-4894-811f-5bb731d83b0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-31226c-6c695fb468-fkr4w" Oct 29 22:50:27.149990 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:27.149844 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3d4a1e8-5e7a-4894-811f-5bb731d83b0d-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-31226c-6c695fb468-fkr4w\" (UID: \"d3d4a1e8-5e7a-4894-811f-5bb731d83b0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-31226c-6c695fb468-fkr4w" Oct 29 22:50:27.149990 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:27.149877 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3d4a1e8-5e7a-4894-811f-5bb731d83b0d-maas-api-tls\") pod \"maas-api-e2e-worker-w6-31226c-6c695fb468-fkr4w\" (UID: \"d3d4a1e8-5e7a-4894-811f-5bb731d83b0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-31226c-6c695fb468-fkr4w" Oct 29 22:50:27.149990 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:27.149905 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d3d4a1e8-5e7a-4894-811f-5bb731d83b0d-metrics-certs\") pod \"maas-api-e2e-worker-w6-31226c-6c695fb468-fkr4w\" (UID: \"d3d4a1e8-5e7a-4894-811f-5bb731d83b0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-31226c-6c695fb468-fkr4w" Oct 29 22:50:27.150580 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:27.150552 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3d4a1e8-5e7a-4894-811f-5bb731d83b0d-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-31226c-6c695fb468-fkr4w\" (UID: \"d3d4a1e8-5e7a-4894-811f-5bb731d83b0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-31226c-6c695fb468-fkr4w" Oct 29 22:50:27.152520 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:27.152499 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d3d4a1e8-5e7a-4894-811f-5bb731d83b0d-metrics-certs\") pod \"maas-api-e2e-worker-w6-31226c-6c695fb468-fkr4w\" (UID: \"d3d4a1e8-5e7a-4894-811f-5bb731d83b0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-31226c-6c695fb468-fkr4w" Oct 29 22:50:27.152610 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:27.152532 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3d4a1e8-5e7a-4894-811f-5bb731d83b0d-maas-api-tls\") pod \"maas-api-e2e-worker-w6-31226c-6c695fb468-fkr4w\" (UID: \"d3d4a1e8-5e7a-4894-811f-5bb731d83b0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-31226c-6c695fb468-fkr4w" Oct 29 22:50:27.171116 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:27.171088 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdrcq\" (UniqueName: \"kubernetes.io/projected/d3d4a1e8-5e7a-4894-811f-5bb731d83b0d-kube-api-access-cdrcq\") pod \"maas-api-e2e-worker-w6-31226c-6c695fb468-fkr4w\" (UID: \"d3d4a1e8-5e7a-4894-811f-5bb731d83b0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-31226c-6c695fb468-fkr4w" Oct 29 22:50:27.207589 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:27.207558 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:50:27.214505 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:27.214481 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-31226c-6c695fb468-fkr4w" Oct 29 22:50:27.515980 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:27.515946 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-31226c-6c695fb468-fkr4w"] Oct 29 22:50:27.518305 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:50:27.518267 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3d4a1e8_5e7a_4894_811f_5bb731d83b0d.slice/crio-b89407f91ac437375b417532c954cfcf74202208c51170efa846841416afbed8 WatchSource:0}: Error finding container b89407f91ac437375b417532c954cfcf74202208c51170efa846841416afbed8: Status 404 returned error can't find the container with id b89407f91ac437375b417532c954cfcf74202208c51170efa846841416afbed8 Oct 29 22:50:27.610540 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:27.610506 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:50:27.708143 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:27.708104 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-639067f9-6b77796fcb-plz6x" event={"ID":"185a7421-26cf-46ae-a65d-ed283704904e","Type":"ContainerStarted","Data":"b1b3daa8cd2ceaa05692f72b604e53b909e26770031a6e54af10e230600426bf"} Oct 29 22:50:27.708371 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:27.708170 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-639067f9-6b77796fcb-plz6x" Oct 29 22:50:27.708371 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:27.708189 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-639067f9-6b77796fcb-plz6x" event={"ID":"185a7421-26cf-46ae-a65d-ed283704904e","Type":"ContainerStarted","Data":"bdbf276ce2c03ab3997feb5add02316f8c828ac62a19765ddc5765a7c04579d1"} Oct 29 22:50:27.709400 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:27.709369 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-31226c-6c695fb468-fkr4w" event={"ID":"d3d4a1e8-5e7a-4894-811f-5bb731d83b0d","Type":"ContainerStarted","Data":"b89407f91ac437375b417532c954cfcf74202208c51170efa846841416afbed8"} Oct 29 22:50:27.711029 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:27.711003 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-639067f9-74c94b54c8-qjsft" event={"ID":"ee18842c-3a3f-4148-807f-09838ef6ccf9","Type":"ContainerStarted","Data":"5bbcf00c9a151a66497559fbf947831fe2e17b0a23b2a035092cd09046c65f79"} Oct 29 22:50:27.711148 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:27.711058 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-639067f9-74c94b54c8-qjsft" Oct 29 22:50:27.722251 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:27.722221 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:50:27.728537 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:27.728514 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v2tfm"] Oct 29 22:50:27.747690 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:27.747635 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-639067f9-6b77796fcb-plz6x" podStartSLOduration=1.7476219830000002 podStartE2EDuration="1.747621983s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:50:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:50:27.744966402 +0000 UTC m=+1008.528179828" watchObservedRunningTime="2026-10-29 22:50:27.747621983 +0000 UTC m=+1008.530835444" Oct 29 22:50:27.790408 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:27.790302 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-639067f9-74c94b54c8-qjsft" podStartSLOduration=2.333060883 podStartE2EDuration="2.790288217s" totalImagesPullingTime="457.227334ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:50:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:50:27.788801401 +0000 UTC m=+1008.572014838" watchObservedRunningTime="2026-10-29 22:50:27.790288217 +0000 UTC m=+1008.573501642" Oct 29 22:50:27.795032 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:27.795006 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-31226c-8475b558c9-2v6n6"] Oct 29 22:50:27.801015 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:27.800990 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-31226c-8475b558c9-2v6n6" Oct 29 22:50:27.814605 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:27.814579 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w6-31226c\"" Oct 29 22:50:27.814728 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:27.814704 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w6-31226c-dockercfg-4gqmp\"" Oct 29 22:50:27.818370 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:27.817981 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-31226c-8475b558c9-2v6n6"] Oct 29 22:50:27.888612 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:27.888583 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-31226c-84fb67d984-mpgbh"] Oct 29 22:50:27.892706 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:27.892686 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-31226c-84fb67d984-mpgbh" Oct 29 22:50:27.912576 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:27.912555 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-31226c-84fb67d984-mpgbh"] Oct 29 22:50:27.959044 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:27.959018 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d4b783f-0071-4524-804e-396c53f069eb-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-31226c-8475b558c9-2v6n6\" (UID: \"8d4b783f-0071-4524-804e-396c53f069eb\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-31226c-8475b558c9-2v6n6" Oct 29 22:50:27.959179 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:27.959159 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pkvj\" (UniqueName: \"kubernetes.io/projected/8d4b783f-0071-4524-804e-396c53f069eb-kube-api-access-7pkvj\") pod \"payload-pre-processing-e2e-worker-w6-31226c-8475b558c9-2v6n6\" (UID: \"8d4b783f-0071-4524-804e-396c53f069eb\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-31226c-8475b558c9-2v6n6" Oct 29 22:50:28.060486 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:28.060398 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d4b783f-0071-4524-804e-396c53f069eb-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-31226c-8475b558c9-2v6n6\" (UID: \"8d4b783f-0071-4524-804e-396c53f069eb\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-31226c-8475b558c9-2v6n6" Oct 29 22:50:28.060660 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:28.060490 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3481d9c3-f557-4b9d-b0f2-b0c0bdb81bd9-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-31226c-84fb67d984-mpgbh\" (UID: \"3481d9c3-f557-4b9d-b0f2-b0c0bdb81bd9\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-31226c-84fb67d984-mpgbh" Oct 29 22:50:28.060660 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:28.060534 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jwjl\" (UniqueName: \"kubernetes.io/projected/3481d9c3-f557-4b9d-b0f2-b0c0bdb81bd9-kube-api-access-4jwjl\") pod \"payload-processing-e2e-worker-w6-31226c-84fb67d984-mpgbh\" (UID: \"3481d9c3-f557-4b9d-b0f2-b0c0bdb81bd9\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-31226c-84fb67d984-mpgbh" Oct 29 22:50:28.060788 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:28.060698 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pkvj\" (UniqueName: \"kubernetes.io/projected/8d4b783f-0071-4524-804e-396c53f069eb-kube-api-access-7pkvj\") pod \"payload-pre-processing-e2e-worker-w6-31226c-8475b558c9-2v6n6\" (UID: \"8d4b783f-0071-4524-804e-396c53f069eb\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-31226c-8475b558c9-2v6n6" Oct 29 22:50:28.061243 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:28.061217 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:50:28.061360 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:28.061312 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d4b783f-0071-4524-804e-396c53f069eb-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-31226c-8475b558c9-2v6n6\" (UID: \"8d4b783f-0071-4524-804e-396c53f069eb\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-31226c-8475b558c9-2v6n6" Oct 29 22:50:28.082924 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:28.082903 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pkvj\" (UniqueName: \"kubernetes.io/projected/8d4b783f-0071-4524-804e-396c53f069eb-kube-api-access-7pkvj\") pod \"payload-pre-processing-e2e-worker-w6-31226c-8475b558c9-2v6n6\" (UID: \"8d4b783f-0071-4524-804e-396c53f069eb\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-31226c-8475b558c9-2v6n6" Oct 29 22:50:28.116814 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:28.116778 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-31226c-8475b558c9-2v6n6" Oct 29 22:50:28.162217 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:28.162180 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3481d9c3-f557-4b9d-b0f2-b0c0bdb81bd9-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-31226c-84fb67d984-mpgbh\" (UID: \"3481d9c3-f557-4b9d-b0f2-b0c0bdb81bd9\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-31226c-84fb67d984-mpgbh" Oct 29 22:50:28.162379 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:28.162240 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jwjl\" (UniqueName: \"kubernetes.io/projected/3481d9c3-f557-4b9d-b0f2-b0c0bdb81bd9-kube-api-access-4jwjl\") pod \"payload-processing-e2e-worker-w6-31226c-84fb67d984-mpgbh\" (UID: \"3481d9c3-f557-4b9d-b0f2-b0c0bdb81bd9\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-31226c-84fb67d984-mpgbh" Oct 29 22:50:28.163080 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:28.163051 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3481d9c3-f557-4b9d-b0f2-b0c0bdb81bd9-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-31226c-84fb67d984-mpgbh\" (UID: \"3481d9c3-f557-4b9d-b0f2-b0c0bdb81bd9\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-31226c-84fb67d984-mpgbh" Oct 29 22:50:28.174693 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:28.174661 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:50:28.192026 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:28.191998 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jwjl\" (UniqueName: \"kubernetes.io/projected/3481d9c3-f557-4b9d-b0f2-b0c0bdb81bd9-kube-api-access-4jwjl\") pod \"payload-processing-e2e-worker-w6-31226c-84fb67d984-mpgbh\" (UID: \"3481d9c3-f557-4b9d-b0f2-b0c0bdb81bd9\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-31226c-84fb67d984-mpgbh" Oct 29 22:50:28.207467 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:28.207441 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-31226c-84fb67d984-mpgbh" Oct 29 22:50:28.440370 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:28.431924 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-31226c-8475b558c9-2v6n6"] Oct 29 22:50:28.524473 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:28.524248 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-31226c-84fb67d984-mpgbh"] Oct 29 22:50:28.528059 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:50:28.528022 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3481d9c3_f557_4b9d_b0f2_b0c0bdb81bd9.slice/crio-0ec98b129f183eb1cb94e00beac2a9f6186480e0633af9bdc7ddb032904f4a59 WatchSource:0}: Error finding container 0ec98b129f183eb1cb94e00beac2a9f6186480e0633af9bdc7ddb032904f4a59: Status 404 returned error can't find the container with id 0ec98b129f183eb1cb94e00beac2a9f6186480e0633af9bdc7ddb032904f4a59 Oct 29 22:50:28.565614 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:28.565588 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:50:28.719694 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:28.719658 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-31226c-6c695fb468-fkr4w" event={"ID":"d3d4a1e8-5e7a-4894-811f-5bb731d83b0d","Type":"ContainerStarted","Data":"9e2f1e464f82fed716d204ec0226fb417ca8ff769d8081a6a16e43421deb38db"} Oct 29 22:50:28.719875 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:28.719724 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-31226c-6c695fb468-fkr4w" Oct 29 22:50:28.721477 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:28.721450 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-31226c-8475b558c9-2v6n6" event={"ID":"8d4b783f-0071-4524-804e-396c53f069eb","Type":"ContainerStarted","Data":"ec85848b77e7759edeb94a5140c9aca054dfe8421fd1f8d0c1be351cae3a83b1"} Oct 29 22:50:28.721605 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:28.721498 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-31226c-8475b558c9-2v6n6" event={"ID":"8d4b783f-0071-4524-804e-396c53f069eb","Type":"ContainerStarted","Data":"15451b41b864963ec6930664d81d7c40a21edc62aa77711cf60fee7ded5b8839"} Oct 29 22:50:28.721605 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:28.721523 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-31226c-8475b558c9-2v6n6" Oct 29 22:50:28.723113 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:28.723090 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-31226c-84fb67d984-mpgbh" event={"ID":"3481d9c3-f557-4b9d-b0f2-b0c0bdb81bd9","Type":"ContainerStarted","Data":"59beb4040f5ec60491fecb6584e0ae3b32213333c9f4f1ff0c302a0ecfdd7280"} Oct 29 22:50:28.723233 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:28.723136 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-31226c-84fb67d984-mpgbh" event={"ID":"3481d9c3-f557-4b9d-b0f2-b0c0bdb81bd9","Type":"ContainerStarted","Data":"0ec98b129f183eb1cb94e00beac2a9f6186480e0633af9bdc7ddb032904f4a59"} Oct 29 22:50:28.723419 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:28.723394 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-31226c-84fb67d984-mpgbh" Oct 29 22:50:28.781153 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:28.781100 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-31226c-6c695fb468-fkr4w" podStartSLOduration=2.406913004 podStartE2EDuration="2.781083504s" totalImagesPullingTime="374.1705ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:50:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:50:28.778529188 +0000 UTC m=+1009.561742613" watchObservedRunningTime="2026-10-29 22:50:28.781083504 +0000 UTC m=+1009.564296931" Oct 29 22:50:28.805020 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:28.804990 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-72818393-648db7d5c9-55zdf"] Oct 29 22:50:28.809562 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:28.809537 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-72818393-648db7d5c9-55zdf" Oct 29 22:50:28.822467 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:28.822417 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-31226c-8475b558c9-2v6n6" podStartSLOduration=1.8224007260000001 podStartE2EDuration="1.822400726s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:50:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:50:28.821495862 +0000 UTC m=+1009.604709299" watchObservedRunningTime="2026-10-29 22:50:28.822400726 +0000 UTC m=+1009.605614149" Oct 29 22:50:28.825487 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:28.825460 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-72818393\"" Oct 29 22:50:28.825592 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:28.825473 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-72818393-dockercfg-dzwwq\"" Oct 29 22:50:28.836543 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:28.836512 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-72818393-648db7d5c9-55zdf"] Oct 29 22:50:28.850927 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:28.850884 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:50:28.866222 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:28.866174 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w6-31226c-84fb67d984-mpgbh" podStartSLOduration=1.866160168 podStartE2EDuration="1.866160168s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:50:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:50:28.863276648 +0000 UTC m=+1009.646490074" watchObservedRunningTime="2026-10-29 22:50:28.866160168 +0000 UTC m=+1009.649373637" Oct 29 22:50:28.870234 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:28.870212 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-72818393-68f6b9db5b-2xnkt"] Oct 29 22:50:28.874559 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:28.874541 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-72818393-68f6b9db5b-2xnkt" Oct 29 22:50:28.892396 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:28.892372 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-72818393-68f6b9db5b-2xnkt"] Oct 29 22:50:28.963989 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:28.963958 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:50:28.971186 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:28.971129 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfmwl\" (UniqueName: \"kubernetes.io/projected/9e333bec-919e-49c4-b9c0-a732247ce509-kube-api-access-qfmwl\") pod \"payload-processing-e2e-ait-72818393-68f6b9db5b-2xnkt\" (UID: \"9e333bec-919e-49c4-b9c0-a732247ce509\") " pod="openshift-ingress/payload-processing-e2e-ait-72818393-68f6b9db5b-2xnkt" Oct 29 22:50:28.971308 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:28.971282 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89a6412b-1be7-4a77-bb4d-c8f81c7d6f43-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-72818393-648db7d5c9-55zdf\" (UID: \"89a6412b-1be7-4a77-bb4d-c8f81c7d6f43\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-72818393-648db7d5c9-55zdf" Oct 29 22:50:28.971456 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:28.971434 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzfdj\" (UniqueName: \"kubernetes.io/projected/89a6412b-1be7-4a77-bb4d-c8f81c7d6f43-kube-api-access-nzfdj\") pod \"payload-pre-processing-e2e-ait-72818393-648db7d5c9-55zdf\" (UID: \"89a6412b-1be7-4a77-bb4d-c8f81c7d6f43\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-72818393-648db7d5c9-55zdf" Oct 29 22:50:28.971575 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:28.971554 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e333bec-919e-49c4-b9c0-a732247ce509-plugins-config-volume\") pod \"payload-processing-e2e-ait-72818393-68f6b9db5b-2xnkt\" (UID: \"9e333bec-919e-49c4-b9c0-a732247ce509\") " pod="openshift-ingress/payload-processing-e2e-ait-72818393-68f6b9db5b-2xnkt" Oct 29 22:50:29.072233 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:29.072192 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89a6412b-1be7-4a77-bb4d-c8f81c7d6f43-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-72818393-648db7d5c9-55zdf\" (UID: \"89a6412b-1be7-4a77-bb4d-c8f81c7d6f43\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-72818393-648db7d5c9-55zdf" Oct 29 22:50:29.072434 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:29.072281 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzfdj\" (UniqueName: \"kubernetes.io/projected/89a6412b-1be7-4a77-bb4d-c8f81c7d6f43-kube-api-access-nzfdj\") pod \"payload-pre-processing-e2e-ait-72818393-648db7d5c9-55zdf\" (UID: \"89a6412b-1be7-4a77-bb4d-c8f81c7d6f43\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-72818393-648db7d5c9-55zdf" Oct 29 22:50:29.072434 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:29.072367 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e333bec-919e-49c4-b9c0-a732247ce509-plugins-config-volume\") pod \"payload-processing-e2e-ait-72818393-68f6b9db5b-2xnkt\" (UID: \"9e333bec-919e-49c4-b9c0-a732247ce509\") " pod="openshift-ingress/payload-processing-e2e-ait-72818393-68f6b9db5b-2xnkt" Oct 29 22:50:29.072434 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:29.072415 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfmwl\" (UniqueName: \"kubernetes.io/projected/9e333bec-919e-49c4-b9c0-a732247ce509-kube-api-access-qfmwl\") pod \"payload-processing-e2e-ait-72818393-68f6b9db5b-2xnkt\" (UID: \"9e333bec-919e-49c4-b9c0-a732247ce509\") " pod="openshift-ingress/payload-processing-e2e-ait-72818393-68f6b9db5b-2xnkt" Oct 29 22:50:29.073133 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:29.073107 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e333bec-919e-49c4-b9c0-a732247ce509-plugins-config-volume\") pod \"payload-processing-e2e-ait-72818393-68f6b9db5b-2xnkt\" (UID: \"9e333bec-919e-49c4-b9c0-a732247ce509\") " pod="openshift-ingress/payload-processing-e2e-ait-72818393-68f6b9db5b-2xnkt" Oct 29 22:50:29.077175 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:29.077147 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89a6412b-1be7-4a77-bb4d-c8f81c7d6f43-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-72818393-648db7d5c9-55zdf\" (UID: \"89a6412b-1be7-4a77-bb4d-c8f81c7d6f43\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-72818393-648db7d5c9-55zdf" Oct 29 22:50:29.093877 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:29.093857 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfmwl\" (UniqueName: \"kubernetes.io/projected/9e333bec-919e-49c4-b9c0-a732247ce509-kube-api-access-qfmwl\") pod \"payload-processing-e2e-ait-72818393-68f6b9db5b-2xnkt\" (UID: \"9e333bec-919e-49c4-b9c0-a732247ce509\") " pod="openshift-ingress/payload-processing-e2e-ait-72818393-68f6b9db5b-2xnkt" Oct 29 22:50:29.094104 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:29.094080 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzfdj\" (UniqueName: \"kubernetes.io/projected/89a6412b-1be7-4a77-bb4d-c8f81c7d6f43-kube-api-access-nzfdj\") pod \"payload-pre-processing-e2e-ait-72818393-648db7d5c9-55zdf\" (UID: \"89a6412b-1be7-4a77-bb4d-c8f81c7d6f43\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-72818393-648db7d5c9-55zdf" Oct 29 22:50:29.124790 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:29.124765 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-72818393-648db7d5c9-55zdf" Oct 29 22:50:29.145293 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:29.145261 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:50:29.188984 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:29.188946 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-72818393-68f6b9db5b-2xnkt" Oct 29 22:50:29.332046 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:29.331936 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:50:29.430814 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:29.430790 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-72818393-648db7d5c9-55zdf"] Oct 29 22:50:29.432963 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:50:29.432929 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89a6412b_1be7_4a77_bb4d_c8f81c7d6f43.slice/crio-8ae2a7b9f7b75538f858d10dd971749bbd90515bf67b5c7c9983dd2ce1b1f1bd WatchSource:0}: Error finding container 8ae2a7b9f7b75538f858d10dd971749bbd90515bf67b5c7c9983dd2ce1b1f1bd: Status 404 returned error can't find the container with id 8ae2a7b9f7b75538f858d10dd971749bbd90515bf67b5c7c9983dd2ce1b1f1bd Oct 29 22:50:29.514529 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:29.514501 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-72818393-68f6b9db5b-2xnkt"] Oct 29 22:50:29.516174 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:50:29.516108 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e333bec_919e_49c4_b9c0_a732247ce509.slice/crio-65c01662423592a486a854c518dcf784e7371a0babea4f1fcaf5ffbd94c869ab WatchSource:0}: Error finding container 65c01662423592a486a854c518dcf784e7371a0babea4f1fcaf5ffbd94c869ab: Status 404 returned error can't find the container with id 65c01662423592a486a854c518dcf784e7371a0babea4f1fcaf5ffbd94c869ab Oct 29 22:50:29.675895 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:29.675854 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:50:29.730444 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:29.730410 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-72818393-68f6b9db5b-2xnkt" event={"ID":"9e333bec-919e-49c4-b9c0-a732247ce509","Type":"ContainerStarted","Data":"7b49f6e3b69f95038d639b02f9bd69bcd2147641f8cbaca6f422baa84a02c8dc"} Oct 29 22:50:29.730627 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:29.730474 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-72818393-68f6b9db5b-2xnkt" event={"ID":"9e333bec-919e-49c4-b9c0-a732247ce509","Type":"ContainerStarted","Data":"65c01662423592a486a854c518dcf784e7371a0babea4f1fcaf5ffbd94c869ab"} Oct 29 22:50:29.730763 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:29.730736 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-72818393-68f6b9db5b-2xnkt" Oct 29 22:50:29.732523 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:29.732476 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-72818393-648db7d5c9-55zdf" event={"ID":"89a6412b-1be7-4a77-bb4d-c8f81c7d6f43","Type":"ContainerStarted","Data":"741eab13ee0928fd6409b13ec80805bf759a43055e70c6fb0b4e30532665a4f0"} Oct 29 22:50:29.732665 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:29.732552 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-72818393-648db7d5c9-55zdf" event={"ID":"89a6412b-1be7-4a77-bb4d-c8f81c7d6f43","Type":"ContainerStarted","Data":"8ae2a7b9f7b75538f858d10dd971749bbd90515bf67b5c7c9983dd2ce1b1f1bd"} Oct 29 22:50:29.732827 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:29.732803 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-72818393-648db7d5c9-55zdf" Oct 29 22:50:29.767715 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:29.767621 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-72818393-68f6b9db5b-2xnkt" podStartSLOduration=1.76760439 podStartE2EDuration="1.76760439s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:50:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:50:29.76626882 +0000 UTC m=+1010.549482444" watchObservedRunningTime="2026-10-29 22:50:29.76760439 +0000 UTC m=+1010.550817815" Oct 29 22:50:29.806860 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:29.806807 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-72818393-648db7d5c9-55zdf" podStartSLOduration=1.806791858 podStartE2EDuration="1.806791858s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:50:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:50:29.80285768 +0000 UTC m=+1010.586071109" watchObservedRunningTime="2026-10-29 22:50:29.806791858 +0000 UTC m=+1010.590005286" Oct 29 22:50:29.810545 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:29.810519 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:50:29.851981 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:29.851952 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1db2136c-5588cff679-gkt2z"] Oct 29 22:50:29.856302 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:29.856284 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-1db2136c-5588cff679-gkt2z" Oct 29 22:50:29.874364 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:29.873161 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-1db2136c\"" Oct 29 22:50:29.874364 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:29.873407 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-1db2136c-dockercfg-xpgts\"" Oct 29 22:50:29.874364 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:29.873921 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1db2136c-5588cff679-gkt2z"] Oct 29 22:50:29.983886 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:29.983851 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b2af96a-99d4-46fc-9a2a-e773d96f5898-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-1db2136c-5588cff679-gkt2z\" (UID: \"3b2af96a-99d4-46fc-9a2a-e773d96f5898\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1db2136c-5588cff679-gkt2z" Oct 29 22:50:29.983886 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:29.983890 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dq5g8\" (UniqueName: \"kubernetes.io/projected/3b2af96a-99d4-46fc-9a2a-e773d96f5898-kube-api-access-dq5g8\") pod \"payload-pre-processing-e2e-ait-1db2136c-5588cff679-gkt2z\" (UID: \"3b2af96a-99d4-46fc-9a2a-e773d96f5898\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1db2136c-5588cff679-gkt2z" Oct 29 22:50:30.085603 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:30.085524 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b2af96a-99d4-46fc-9a2a-e773d96f5898-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-1db2136c-5588cff679-gkt2z\" (UID: \"3b2af96a-99d4-46fc-9a2a-e773d96f5898\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1db2136c-5588cff679-gkt2z" Oct 29 22:50:30.085603 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:30.085564 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dq5g8\" (UniqueName: \"kubernetes.io/projected/3b2af96a-99d4-46fc-9a2a-e773d96f5898-kube-api-access-dq5g8\") pod \"payload-pre-processing-e2e-ait-1db2136c-5588cff679-gkt2z\" (UID: \"3b2af96a-99d4-46fc-9a2a-e773d96f5898\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1db2136c-5588cff679-gkt2z" Oct 29 22:50:30.086268 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:30.086245 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b2af96a-99d4-46fc-9a2a-e773d96f5898-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-1db2136c-5588cff679-gkt2z\" (UID: \"3b2af96a-99d4-46fc-9a2a-e773d96f5898\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1db2136c-5588cff679-gkt2z" Oct 29 22:50:30.108642 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:30.108613 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dq5g8\" (UniqueName: \"kubernetes.io/projected/3b2af96a-99d4-46fc-9a2a-e773d96f5898-kube-api-access-dq5g8\") pod \"payload-pre-processing-e2e-ait-1db2136c-5588cff679-gkt2z\" (UID: \"3b2af96a-99d4-46fc-9a2a-e773d96f5898\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1db2136c-5588cff679-gkt2z" Oct 29 22:50:30.170407 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:30.170380 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-1db2136c-5588cff679-gkt2z" Oct 29 22:50:30.469201 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:30.469176 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1db2136c-5588cff679-gkt2z"] Oct 29 22:50:30.471036 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:50:30.471010 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b2af96a_99d4_46fc_9a2a_e773d96f5898.slice/crio-0744881ef1e4096bfac82431ea0c4a6ff13e3c66bc3aceb7021353ab04540e42 WatchSource:0}: Error finding container 0744881ef1e4096bfac82431ea0c4a6ff13e3c66bc3aceb7021353ab04540e42: Status 404 returned error can't find the container with id 0744881ef1e4096bfac82431ea0c4a6ff13e3c66bc3aceb7021353ab04540e42 Oct 29 22:50:30.568777 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:30.568728 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:50:30.616648 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:30.616616 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-72818393-5c8b48595d-24b86"] Oct 29 22:50:30.621194 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:30.621168 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-72818393-5c8b48595d-24b86" Oct 29 22:50:30.651095 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:30.649874 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-72818393-serving-cert\"" Oct 29 22:50:30.654985 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:30.654822 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-72818393-5c8b48595d-24b86"] Oct 29 22:50:30.691860 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:30.691825 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q4kgg\" (UniqueName: \"kubernetes.io/projected/3871fc17-a6ab-407c-944d-8f862b323e2d-kube-api-access-q4kgg\") pod \"maas-api-e2e-ait-72818393-5c8b48595d-24b86\" (UID: \"3871fc17-a6ab-407c-944d-8f862b323e2d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-72818393-5c8b48595d-24b86" Oct 29 22:50:30.692009 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:30.691870 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3871fc17-a6ab-407c-944d-8f862b323e2d-maas-api-tls\") pod \"maas-api-e2e-ait-72818393-5c8b48595d-24b86\" (UID: \"3871fc17-a6ab-407c-944d-8f862b323e2d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-72818393-5c8b48595d-24b86" Oct 29 22:50:30.692080 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:30.692048 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3871fc17-a6ab-407c-944d-8f862b323e2d-metrics-certs\") pod \"maas-api-e2e-ait-72818393-5c8b48595d-24b86\" (UID: \"3871fc17-a6ab-407c-944d-8f862b323e2d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-72818393-5c8b48595d-24b86" Oct 29 22:50:30.692134 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:30.692084 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3871fc17-a6ab-407c-944d-8f862b323e2d-openshift-service-ca\") pod \"maas-api-e2e-ait-72818393-5c8b48595d-24b86\" (UID: \"3871fc17-a6ab-407c-944d-8f862b323e2d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-72818393-5c8b48595d-24b86" Oct 29 22:50:30.727439 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:30.727350 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:50:30.741981 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:30.741950 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-1db2136c-5588cff679-gkt2z" event={"ID":"3b2af96a-99d4-46fc-9a2a-e773d96f5898","Type":"ContainerStarted","Data":"ec742484b685dad3ece4771eb72a9f5b79ff91398def48dbd3767b60aad81484"} Oct 29 22:50:30.742137 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:30.742014 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-1db2136c-5588cff679-gkt2z" event={"ID":"3b2af96a-99d4-46fc-9a2a-e773d96f5898","Type":"ContainerStarted","Data":"0744881ef1e4096bfac82431ea0c4a6ff13e3c66bc3aceb7021353ab04540e42"} Oct 29 22:50:30.742379 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:30.742354 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-1db2136c-5588cff679-gkt2z" Oct 29 22:50:30.761949 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:30.761919 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-3c314108-6769db8fbc-75ccb"] Oct 29 22:50:30.766437 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:30.766418 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-3c314108-6769db8fbc-75ccb" Oct 29 22:50:30.779562 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:30.779537 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-3c314108\"" Oct 29 22:50:30.792753 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:30.792700 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-1db2136c-5588cff679-gkt2z" podStartSLOduration=1.7926836229999998 podStartE2EDuration="1.792683623s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:50:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:50:30.78818782 +0000 UTC m=+1011.571401245" watchObservedRunningTime="2026-10-29 22:50:30.792683623 +0000 UTC m=+1011.575897051" Oct 29 22:50:30.793070 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:30.793046 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3871fc17-a6ab-407c-944d-8f862b323e2d-maas-api-tls\") pod \"maas-api-e2e-ait-72818393-5c8b48595d-24b86\" (UID: \"3871fc17-a6ab-407c-944d-8f862b323e2d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-72818393-5c8b48595d-24b86" Oct 29 22:50:30.793936 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:30.793901 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3871fc17-a6ab-407c-944d-8f862b323e2d-metrics-certs\") pod \"maas-api-e2e-ait-72818393-5c8b48595d-24b86\" (UID: \"3871fc17-a6ab-407c-944d-8f862b323e2d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-72818393-5c8b48595d-24b86" Oct 29 22:50:30.794047 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:30.793958 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3871fc17-a6ab-407c-944d-8f862b323e2d-openshift-service-ca\") pod \"maas-api-e2e-ait-72818393-5c8b48595d-24b86\" (UID: \"3871fc17-a6ab-407c-944d-8f862b323e2d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-72818393-5c8b48595d-24b86" Oct 29 22:50:30.794119 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:30.794061 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4kgg\" (UniqueName: \"kubernetes.io/projected/3871fc17-a6ab-407c-944d-8f862b323e2d-kube-api-access-q4kgg\") pod \"maas-api-e2e-ait-72818393-5c8b48595d-24b86\" (UID: \"3871fc17-a6ab-407c-944d-8f862b323e2d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-72818393-5c8b48595d-24b86" Oct 29 22:50:30.795101 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:30.795052 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3871fc17-a6ab-407c-944d-8f862b323e2d-openshift-service-ca\") pod \"maas-api-e2e-ait-72818393-5c8b48595d-24b86\" (UID: \"3871fc17-a6ab-407c-944d-8f862b323e2d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-72818393-5c8b48595d-24b86" Oct 29 22:50:30.796364 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:30.795749 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3871fc17-a6ab-407c-944d-8f862b323e2d-maas-api-tls\") pod \"maas-api-e2e-ait-72818393-5c8b48595d-24b86\" (UID: \"3871fc17-a6ab-407c-944d-8f862b323e2d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-72818393-5c8b48595d-24b86" Oct 29 22:50:30.796939 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:30.796922 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3871fc17-a6ab-407c-944d-8f862b323e2d-metrics-certs\") pod \"maas-api-e2e-ait-72818393-5c8b48595d-24b86\" (UID: \"3871fc17-a6ab-407c-944d-8f862b323e2d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-72818393-5c8b48595d-24b86" Oct 29 22:50:30.798777 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:30.798754 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-3c314108-6769db8fbc-75ccb"] Oct 29 22:50:30.810972 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:30.810949 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-3c314108-dockercfg-6jqtq\"" Oct 29 22:50:30.821426 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:30.821407 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4kgg\" (UniqueName: \"kubernetes.io/projected/3871fc17-a6ab-407c-944d-8f862b323e2d-kube-api-access-q4kgg\") pod \"maas-api-e2e-ait-72818393-5c8b48595d-24b86\" (UID: \"3871fc17-a6ab-407c-944d-8f862b323e2d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-72818393-5c8b48595d-24b86" Oct 29 22:50:30.840058 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:30.840033 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-3c314108-67c4d8d458-hbl94"] Oct 29 22:50:30.844323 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:30.844307 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-3c314108-67c4d8d458-hbl94" Oct 29 22:50:30.866615 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:30.866589 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-3c314108-67c4d8d458-hbl94"] Oct 29 22:50:30.869427 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:30.869405 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:50:30.894825 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:30.894798 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47qnj\" (UniqueName: \"kubernetes.io/projected/846589eb-a0a2-49c1-a9e0-e793b456633b-kube-api-access-47qnj\") pod \"payload-pre-processing-e2e-derive-3c314108-6769db8fbc-75ccb\" (UID: \"846589eb-a0a2-49c1-a9e0-e793b456633b\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-3c314108-6769db8fbc-75ccb" Oct 29 22:50:30.894979 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:30.894953 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/846589eb-a0a2-49c1-a9e0-e793b456633b-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-3c314108-6769db8fbc-75ccb\" (UID: \"846589eb-a0a2-49c1-a9e0-e793b456633b\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-3c314108-6769db8fbc-75ccb" Oct 29 22:50:30.895107 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:30.895089 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1c2e436-c3c1-4cc8-a362-f77f162d6508-plugins-config-volume\") pod \"payload-processing-e2e-derive-3c314108-67c4d8d458-hbl94\" (UID: \"a1c2e436-c3c1-4cc8-a362-f77f162d6508\") " pod="openshift-ingress/payload-processing-e2e-derive-3c314108-67c4d8d458-hbl94" Oct 29 22:50:30.895183 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:30.895121 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6m5f\" (UniqueName: \"kubernetes.io/projected/a1c2e436-c3c1-4cc8-a362-f77f162d6508-kube-api-access-j6m5f\") pod \"payload-processing-e2e-derive-3c314108-67c4d8d458-hbl94\" (UID: \"a1c2e436-c3c1-4cc8-a362-f77f162d6508\") " pod="openshift-ingress/payload-processing-e2e-derive-3c314108-67c4d8d458-hbl94" Oct 29 22:50:30.937558 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:30.937527 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-72818393-5c8b48595d-24b86" Oct 29 22:50:30.998390 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:30.997041 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1c2e436-c3c1-4cc8-a362-f77f162d6508-plugins-config-volume\") pod \"payload-processing-e2e-derive-3c314108-67c4d8d458-hbl94\" (UID: \"a1c2e436-c3c1-4cc8-a362-f77f162d6508\") " pod="openshift-ingress/payload-processing-e2e-derive-3c314108-67c4d8d458-hbl94" Oct 29 22:50:30.998390 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:30.997091 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6m5f\" (UniqueName: \"kubernetes.io/projected/a1c2e436-c3c1-4cc8-a362-f77f162d6508-kube-api-access-j6m5f\") pod \"payload-processing-e2e-derive-3c314108-67c4d8d458-hbl94\" (UID: \"a1c2e436-c3c1-4cc8-a362-f77f162d6508\") " pod="openshift-ingress/payload-processing-e2e-derive-3c314108-67c4d8d458-hbl94" Oct 29 22:50:30.998390 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:30.997205 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47qnj\" (UniqueName: \"kubernetes.io/projected/846589eb-a0a2-49c1-a9e0-e793b456633b-kube-api-access-47qnj\") pod \"payload-pre-processing-e2e-derive-3c314108-6769db8fbc-75ccb\" (UID: \"846589eb-a0a2-49c1-a9e0-e793b456633b\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-3c314108-6769db8fbc-75ccb" Oct 29 22:50:30.998390 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:30.997321 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/846589eb-a0a2-49c1-a9e0-e793b456633b-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-3c314108-6769db8fbc-75ccb\" (UID: \"846589eb-a0a2-49c1-a9e0-e793b456633b\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-3c314108-6769db8fbc-75ccb" Oct 29 22:50:30.998390 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:30.998205 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/846589eb-a0a2-49c1-a9e0-e793b456633b-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-3c314108-6769db8fbc-75ccb\" (UID: \"846589eb-a0a2-49c1-a9e0-e793b456633b\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-3c314108-6769db8fbc-75ccb" Oct 29 22:50:30.999168 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:30.998477 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1c2e436-c3c1-4cc8-a362-f77f162d6508-plugins-config-volume\") pod \"payload-processing-e2e-derive-3c314108-67c4d8d458-hbl94\" (UID: \"a1c2e436-c3c1-4cc8-a362-f77f162d6508\") " pod="openshift-ingress/payload-processing-e2e-derive-3c314108-67c4d8d458-hbl94" Oct 29 22:50:31.000611 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:31.000578 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:50:31.020988 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:31.020959 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-47qnj\" (UniqueName: \"kubernetes.io/projected/846589eb-a0a2-49c1-a9e0-e793b456633b-kube-api-access-47qnj\") pod \"payload-pre-processing-e2e-derive-3c314108-6769db8fbc-75ccb\" (UID: \"846589eb-a0a2-49c1-a9e0-e793b456633b\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-3c314108-6769db8fbc-75ccb" Oct 29 22:50:31.021096 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:31.021005 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6m5f\" (UniqueName: \"kubernetes.io/projected/a1c2e436-c3c1-4cc8-a362-f77f162d6508-kube-api-access-j6m5f\") pod \"payload-processing-e2e-derive-3c314108-67c4d8d458-hbl94\" (UID: \"a1c2e436-c3c1-4cc8-a362-f77f162d6508\") " pod="openshift-ingress/payload-processing-e2e-derive-3c314108-67c4d8d458-hbl94" Oct 29 22:50:31.080097 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:31.080064 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-3c314108-6769db8fbc-75ccb" Oct 29 22:50:31.159757 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:31.159715 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-3c314108-67c4d8d458-hbl94" Oct 29 22:50:31.258870 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:31.258839 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-72818393-5c8b48595d-24b86"] Oct 29 22:50:31.259442 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:50:31.259403 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3871fc17_a6ab_407c_944d_8f862b323e2d.slice/crio-6800319df2381c35c6e9f9f9d5c54f6a9401f9963c615dc2b51ea181b1bc05da WatchSource:0}: Error finding container 6800319df2381c35c6e9f9f9d5c54f6a9401f9963c615dc2b51ea181b1bc05da: Status 404 returned error can't find the container with id 6800319df2381c35c6e9f9f9d5c54f6a9401f9963c615dc2b51ea181b1bc05da Oct 29 22:50:31.385575 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:31.385547 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-3c314108-6769db8fbc-75ccb"] Oct 29 22:50:31.388714 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:50:31.388674 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod846589eb_a0a2_49c1_a9e0_e793b456633b.slice/crio-61025276b39889732ee6201bb70ae5bae1bed12eddbbe5314cb3f497d8062f45 WatchSource:0}: Error finding container 61025276b39889732ee6201bb70ae5bae1bed12eddbbe5314cb3f497d8062f45: Status 404 returned error can't find the container with id 61025276b39889732ee6201bb70ae5bae1bed12eddbbe5314cb3f497d8062f45 Oct 29 22:50:31.469080 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:31.469047 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-3c314108-67c4d8d458-hbl94"] Oct 29 22:50:31.476902 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:50:31.476853 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1c2e436_c3c1_4cc8_a362_f77f162d6508.slice/crio-8aa879d946825e44093a16b22638a241375d7dc7e578b33b056fc36202513185 WatchSource:0}: Error finding container 8aa879d946825e44093a16b22638a241375d7dc7e578b33b056fc36202513185: Status 404 returned error can't find the container with id 8aa879d946825e44093a16b22638a241375d7dc7e578b33b056fc36202513185 Oct 29 22:50:31.751573 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:31.751530 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-3c314108-67c4d8d458-hbl94" event={"ID":"a1c2e436-c3c1-4cc8-a362-f77f162d6508","Type":"ContainerStarted","Data":"efcf2e9b80c3efdf2f7ed4f6709d590371b415e8eacf1b9602cb35c60521a41e"} Oct 29 22:50:31.751757 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:31.751599 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-3c314108-67c4d8d458-hbl94" event={"ID":"a1c2e436-c3c1-4cc8-a362-f77f162d6508","Type":"ContainerStarted","Data":"8aa879d946825e44093a16b22638a241375d7dc7e578b33b056fc36202513185"} Oct 29 22:50:31.751757 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:31.751631 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-3c314108-67c4d8d458-hbl94" Oct 29 22:50:31.753152 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:31.753120 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-3c314108-6769db8fbc-75ccb" event={"ID":"846589eb-a0a2-49c1-a9e0-e793b456633b","Type":"ContainerStarted","Data":"5503e3d7d3366655f6141c398f673deb7a5a2a2781a20d56f65628835965c172"} Oct 29 22:50:31.753309 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:31.753172 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-3c314108-6769db8fbc-75ccb" event={"ID":"846589eb-a0a2-49c1-a9e0-e793b456633b","Type":"ContainerStarted","Data":"61025276b39889732ee6201bb70ae5bae1bed12eddbbe5314cb3f497d8062f45"} Oct 29 22:50:31.753309 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:31.753199 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-3c314108-6769db8fbc-75ccb" Oct 29 22:50:31.754762 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:31.754742 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-72818393-5c8b48595d-24b86" event={"ID":"3871fc17-a6ab-407c-944d-8f862b323e2d","Type":"ContainerStarted","Data":"43f95a561100305373e64427fa7f323bcef1903a18e3d9ed893bbf22483820ea"} Oct 29 22:50:31.754869 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:31.754777 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-72818393-5c8b48595d-24b86" event={"ID":"3871fc17-a6ab-407c-944d-8f862b323e2d","Type":"ContainerStarted","Data":"6800319df2381c35c6e9f9f9d5c54f6a9401f9963c615dc2b51ea181b1bc05da"} Oct 29 22:50:31.754940 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:31.754875 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-72818393-5c8b48595d-24b86" Oct 29 22:50:31.828222 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:31.828115 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-72818393-5c8b48595d-24b86" podStartSLOduration=1.464503936 podStartE2EDuration="1.828102911s" totalImagesPullingTime="363.598975ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:50:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:50:31.826980996 +0000 UTC m=+1012.610194432" watchObservedRunningTime="2026-10-29 22:50:31.828102911 +0000 UTC m=+1012.611316338" Oct 29 22:50:31.829210 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:31.829167 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-3c314108-67c4d8d458-hbl94" podStartSLOduration=1.829158773 podStartE2EDuration="1.829158773s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:50:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:50:31.788995211 +0000 UTC m=+1012.572208638" watchObservedRunningTime="2026-10-29 22:50:31.829158773 +0000 UTC m=+1012.612372200" Oct 29 22:50:31.866826 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:31.866760 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-3c314108-6769db8fbc-75ccb" podStartSLOduration=1.866742441 podStartE2EDuration="1.866742441s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:50:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:50:31.863587396 +0000 UTC m=+1012.646800846" watchObservedRunningTime="2026-10-29 22:50:31.866742441 +0000 UTC m=+1012.649955868" Oct 29 22:50:32.480865 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:32.480834 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:50:32.594326 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:32.594269 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:50:32.645635 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:32.645603 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-08e15d-6bd798d867-mnq4t"] Oct 29 22:50:32.650086 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:32.650066 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-08e15d-6bd798d867-mnq4t" Oct 29 22:50:32.673324 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:32.673300 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-08e15d-dockercfg-g6mzq\"" Oct 29 22:50:32.673466 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:32.673350 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-08e15d\"" Oct 29 22:50:32.677828 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:32.677801 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-08e15d-6bd798d867-mnq4t"] Oct 29 22:50:32.720498 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:32.720471 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/684cec70-0bb2-4eff-917e-6a04c8734d74-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-08e15d-6bd798d867-mnq4t\" (UID: \"684cec70-0bb2-4eff-917e-6a04c8734d74\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-08e15d-6bd798d867-mnq4t" Oct 29 22:50:32.720660 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:32.720533 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wpvjw\" (UniqueName: \"kubernetes.io/projected/684cec70-0bb2-4eff-917e-6a04c8734d74-kube-api-access-wpvjw\") pod \"payload-pre-processing-e2e-worker-w0-08e15d-6bd798d867-mnq4t\" (UID: \"684cec70-0bb2-4eff-917e-6a04c8734d74\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-08e15d-6bd798d867-mnq4t" Oct 29 22:50:32.722038 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:32.722012 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:50:32.822532 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:32.822450 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/684cec70-0bb2-4eff-917e-6a04c8734d74-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-08e15d-6bd798d867-mnq4t\" (UID: \"684cec70-0bb2-4eff-917e-6a04c8734d74\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-08e15d-6bd798d867-mnq4t" Oct 29 22:50:32.822838 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:32.822807 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wpvjw\" (UniqueName: \"kubernetes.io/projected/684cec70-0bb2-4eff-917e-6a04c8734d74-kube-api-access-wpvjw\") pod \"payload-pre-processing-e2e-worker-w0-08e15d-6bd798d867-mnq4t\" (UID: \"684cec70-0bb2-4eff-917e-6a04c8734d74\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-08e15d-6bd798d867-mnq4t" Oct 29 22:50:32.823721 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:32.823692 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/684cec70-0bb2-4eff-917e-6a04c8734d74-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-08e15d-6bd798d867-mnq4t\" (UID: \"684cec70-0bb2-4eff-917e-6a04c8734d74\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-08e15d-6bd798d867-mnq4t" Oct 29 22:50:32.844541 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:32.844516 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wpvjw\" (UniqueName: \"kubernetes.io/projected/684cec70-0bb2-4eff-917e-6a04c8734d74-kube-api-access-wpvjw\") pod \"payload-pre-processing-e2e-worker-w0-08e15d-6bd798d867-mnq4t\" (UID: \"684cec70-0bb2-4eff-917e-6a04c8734d74\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-08e15d-6bd798d867-mnq4t" Oct 29 22:50:32.847114 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:32.847083 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:50:32.965382 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:32.965349 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-08e15d-6bd798d867-mnq4t" Oct 29 22:50:32.987418 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:32.987382 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:50:33.113013 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:33.112922 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:50:33.280843 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:33.280692 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-08e15d-6bd798d867-mnq4t"] Oct 29 22:50:33.285752 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:50:33.285725 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod684cec70_0bb2_4eff_917e_6a04c8734d74.slice/crio-4b6783b1565491a31c0c50932649a4bf0a8bdaf5d20d818ee295c0d2842b06c5 WatchSource:0}: Error finding container 4b6783b1565491a31c0c50932649a4bf0a8bdaf5d20d818ee295c0d2842b06c5: Status 404 returned error can't find the container with id 4b6783b1565491a31c0c50932649a4bf0a8bdaf5d20d818ee295c0d2842b06c5 Oct 29 22:50:33.729261 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:33.729232 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-639067f9-74c94b54c8-qjsft" Oct 29 22:50:33.774484 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:33.774451 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-08e15d-6bd798d867-mnq4t" event={"ID":"684cec70-0bb2-4eff-917e-6a04c8734d74","Type":"ContainerStarted","Data":"ba185bb1c59c6d13d01637228fd6f3dc866f5ce40dff3d4253fe06e110a9e2c9"} Oct 29 22:50:33.774634 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:33.774506 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-08e15d-6bd798d867-mnq4t" event={"ID":"684cec70-0bb2-4eff-917e-6a04c8734d74","Type":"ContainerStarted","Data":"4b6783b1565491a31c0c50932649a4bf0a8bdaf5d20d818ee295c0d2842b06c5"} Oct 29 22:50:33.774685 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:33.774649 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-08e15d-6bd798d867-mnq4t" Oct 29 22:50:33.820484 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:33.820428 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-08e15d-6bd798d867-mnq4t" podStartSLOduration=1.8203240379999999 podStartE2EDuration="1.820324038s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:50:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:50:33.818928527 +0000 UTC m=+1014.602141953" watchObservedRunningTime="2026-10-29 22:50:33.820324038 +0000 UTC m=+1014.603537466" Oct 29 22:50:34.739319 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:34.739279 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-31226c-6c695fb468-fkr4w" Oct 29 22:50:34.795905 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:34.795867 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:50:36.291227 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:36.291178 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:50:36.417064 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:36.417030 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:50:36.525776 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:36.525745 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:50:37.569678 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:37.569642 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v2tfm"] Oct 29 22:50:37.648793 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:37.648754 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:50:37.714634 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:37.714603 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-639067f9-588f8ccf99-89pzl" Oct 29 22:50:37.772439 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:37.772409 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-72818393-5c8b48595d-24b86" Oct 29 22:50:37.806257 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:37.806228 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:50:37.980716 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:37.980683 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-3c314108-7d6d7db8c7-s2s47"] Oct 29 22:50:37.985550 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:37.985526 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3c314108-7d6d7db8c7-s2s47" Oct 29 22:50:38.006953 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:38.006902 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-3c314108-7d6d7db8c7-s2s47"] Oct 29 22:50:38.021477 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:38.021443 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-3c314108-serving-cert\"" Oct 29 22:50:38.065394 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:38.065319 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:50:38.082419 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:38.082389 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/029233d7-bcbd-41e8-af12-f3276061c833-maas-api-tls\") pod \"maas-api-e2e-derive-3c314108-7d6d7db8c7-s2s47\" (UID: \"029233d7-bcbd-41e8-af12-f3276061c833\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3c314108-7d6d7db8c7-s2s47" Oct 29 22:50:38.082552 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:38.082524 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/029233d7-bcbd-41e8-af12-f3276061c833-metrics-certs\") pod \"maas-api-e2e-derive-3c314108-7d6d7db8c7-s2s47\" (UID: \"029233d7-bcbd-41e8-af12-f3276061c833\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3c314108-7d6d7db8c7-s2s47" Oct 29 22:50:38.082682 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:38.082663 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/029233d7-bcbd-41e8-af12-f3276061c833-openshift-service-ca\") pod \"maas-api-e2e-derive-3c314108-7d6d7db8c7-s2s47\" (UID: \"029233d7-bcbd-41e8-af12-f3276061c833\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3c314108-7d6d7db8c7-s2s47" Oct 29 22:50:38.082739 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:38.082700 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vn8fq\" (UniqueName: \"kubernetes.io/projected/029233d7-bcbd-41e8-af12-f3276061c833-kube-api-access-vn8fq\") pod \"maas-api-e2e-derive-3c314108-7d6d7db8c7-s2s47\" (UID: \"029233d7-bcbd-41e8-af12-f3276061c833\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3c314108-7d6d7db8c7-s2s47" Oct 29 22:50:38.177890 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:38.177852 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:50:38.184012 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:38.183983 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/029233d7-bcbd-41e8-af12-f3276061c833-maas-api-tls\") pod \"maas-api-e2e-derive-3c314108-7d6d7db8c7-s2s47\" (UID: \"029233d7-bcbd-41e8-af12-f3276061c833\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3c314108-7d6d7db8c7-s2s47" Oct 29 22:50:38.184151 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:38.184037 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/029233d7-bcbd-41e8-af12-f3276061c833-metrics-certs\") pod \"maas-api-e2e-derive-3c314108-7d6d7db8c7-s2s47\" (UID: \"029233d7-bcbd-41e8-af12-f3276061c833\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3c314108-7d6d7db8c7-s2s47" Oct 29 22:50:38.184151 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:38.184113 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/029233d7-bcbd-41e8-af12-f3276061c833-openshift-service-ca\") pod \"maas-api-e2e-derive-3c314108-7d6d7db8c7-s2s47\" (UID: \"029233d7-bcbd-41e8-af12-f3276061c833\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3c314108-7d6d7db8c7-s2s47" Oct 29 22:50:38.184151 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:38.184139 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vn8fq\" (UniqueName: \"kubernetes.io/projected/029233d7-bcbd-41e8-af12-f3276061c833-kube-api-access-vn8fq\") pod \"maas-api-e2e-derive-3c314108-7d6d7db8c7-s2s47\" (UID: \"029233d7-bcbd-41e8-af12-f3276061c833\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3c314108-7d6d7db8c7-s2s47" Oct 29 22:50:38.184882 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:38.184858 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/029233d7-bcbd-41e8-af12-f3276061c833-openshift-service-ca\") pod \"maas-api-e2e-derive-3c314108-7d6d7db8c7-s2s47\" (UID: \"029233d7-bcbd-41e8-af12-f3276061c833\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3c314108-7d6d7db8c7-s2s47" Oct 29 22:50:38.187230 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:38.187205 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/029233d7-bcbd-41e8-af12-f3276061c833-metrics-certs\") pod \"maas-api-e2e-derive-3c314108-7d6d7db8c7-s2s47\" (UID: \"029233d7-bcbd-41e8-af12-f3276061c833\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3c314108-7d6d7db8c7-s2s47" Oct 29 22:50:38.187318 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:38.187241 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/029233d7-bcbd-41e8-af12-f3276061c833-maas-api-tls\") pod \"maas-api-e2e-derive-3c314108-7d6d7db8c7-s2s47\" (UID: \"029233d7-bcbd-41e8-af12-f3276061c833\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3c314108-7d6d7db8c7-s2s47" Oct 29 22:50:38.205985 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:38.205956 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vn8fq\" (UniqueName: \"kubernetes.io/projected/029233d7-bcbd-41e8-af12-f3276061c833-kube-api-access-vn8fq\") pod \"maas-api-e2e-derive-3c314108-7d6d7db8c7-s2s47\" (UID: \"029233d7-bcbd-41e8-af12-f3276061c833\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3c314108-7d6d7db8c7-s2s47" Oct 29 22:50:38.291202 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:38.291129 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:50:38.305042 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:38.305012 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3c314108-7d6d7db8c7-s2s47" Oct 29 22:50:38.613519 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:38.613489 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-3c314108-7d6d7db8c7-s2s47"] Oct 29 22:50:38.615475 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:50:38.615451 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod029233d7_bcbd_41e8_af12_f3276061c833.slice/crio-a54fd8a947ae5c5683351fe415e12d0b735e94be7add2910ad2d58d3a002bd33 WatchSource:0}: Error finding container a54fd8a947ae5c5683351fe415e12d0b735e94be7add2910ad2d58d3a002bd33: Status 404 returned error can't find the container with id a54fd8a947ae5c5683351fe415e12d0b735e94be7add2910ad2d58d3a002bd33 Oct 29 22:50:38.725746 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:38.725721 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-639067f9-6b77796fcb-plz6x" Oct 29 22:50:38.813726 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:38.813695 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3c314108-7d6d7db8c7-s2s47" event={"ID":"029233d7-bcbd-41e8-af12-f3276061c833","Type":"ContainerStarted","Data":"a54fd8a947ae5c5683351fe415e12d0b735e94be7add2910ad2d58d3a002bd33"} Oct 29 22:50:39.735719 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:39.735689 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-31226c-84fb67d984-mpgbh" Oct 29 22:50:39.736712 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:39.736631 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-31226c-8475b558c9-2v6n6" Oct 29 22:50:39.822460 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:39.822425 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3c314108-7d6d7db8c7-s2s47" event={"ID":"029233d7-bcbd-41e8-af12-f3276061c833","Type":"ContainerStarted","Data":"ee67ee37bab54f052b904641ec1aaa82b9b9a46fd30ef03958e588faa9efd7f7"} Oct 29 22:50:39.822629 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:39.822490 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3c314108-7d6d7db8c7-s2s47" Oct 29 22:50:39.865310 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:39.865251 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3c314108-7d6d7db8c7-s2s47" podStartSLOduration=2.483150983 podStartE2EDuration="2.865235957s" totalImagesPullingTime="382.084974ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:50:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:50:39.862592005 +0000 UTC m=+1020.645805430" watchObservedRunningTime="2026-10-29 22:50:39.865235957 +0000 UTC m=+1020.648449382" Oct 29 22:50:40.201569 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:40.201535 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:50:40.745901 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:40.745868 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-72818393-68f6b9db5b-2xnkt" Oct 29 22:50:40.746267 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:40.745948 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-72818393-648db7d5c9-55zdf" Oct 29 22:50:41.757913 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:41.757884 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-1db2136c-5588cff679-gkt2z" Oct 29 22:50:42.112952 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:42.112863 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:50:42.767925 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:42.767898 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-3c314108-67c4d8d458-hbl94" Oct 29 22:50:42.768426 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:42.767969 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-3c314108-6769db8fbc-75ccb" Oct 29 22:50:43.620057 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:43.620008 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:50:44.155964 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:44.155919 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:50:44.388481 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:44.388451 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:50:44.501206 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:44.501167 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:50:44.785092 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:44.785027 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-08e15d-6bd798d867-mnq4t" Oct 29 22:50:45.790424 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:45.790384 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:50:45.837398 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:45.837371 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3c314108-7d6d7db8c7-s2s47" Oct 29 22:50:45.895291 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:45.895261 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-e131af-58475b998f-bx6vp"] Oct 29 22:50:45.901155 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:45.901131 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e131af-58475b998f-bx6vp" Oct 29 22:50:45.921121 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:45.921091 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w1-e131af-serving-cert\"" Oct 29 22:50:45.926176 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:45.926144 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-e131af-58475b998f-bx6vp"] Oct 29 22:50:45.979670 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:45.979630 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c334eeb-957b-47ba-9944-c363e0f6b35e-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-e131af-58475b998f-bx6vp\" (UID: \"3c334eeb-957b-47ba-9944-c363e0f6b35e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e131af-58475b998f-bx6vp" Oct 29 22:50:45.979851 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:45.979719 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7d2j6\" (UniqueName: \"kubernetes.io/projected/3c334eeb-957b-47ba-9944-c363e0f6b35e-kube-api-access-7d2j6\") pod \"maas-api-e2e-worker-w1-e131af-58475b998f-bx6vp\" (UID: \"3c334eeb-957b-47ba-9944-c363e0f6b35e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e131af-58475b998f-bx6vp" Oct 29 22:50:45.979851 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:45.979757 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c334eeb-957b-47ba-9944-c363e0f6b35e-metrics-certs\") pod \"maas-api-e2e-worker-w1-e131af-58475b998f-bx6vp\" (UID: \"3c334eeb-957b-47ba-9944-c363e0f6b35e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e131af-58475b998f-bx6vp" Oct 29 22:50:45.979851 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:45.979841 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3c334eeb-957b-47ba-9944-c363e0f6b35e-maas-api-tls\") pod \"maas-api-e2e-worker-w1-e131af-58475b998f-bx6vp\" (UID: \"3c334eeb-957b-47ba-9944-c363e0f6b35e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e131af-58475b998f-bx6vp" Oct 29 22:50:45.998388 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:45.998357 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:50:46.081393 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:46.081310 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7d2j6\" (UniqueName: \"kubernetes.io/projected/3c334eeb-957b-47ba-9944-c363e0f6b35e-kube-api-access-7d2j6\") pod \"maas-api-e2e-worker-w1-e131af-58475b998f-bx6vp\" (UID: \"3c334eeb-957b-47ba-9944-c363e0f6b35e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e131af-58475b998f-bx6vp" Oct 29 22:50:46.081393 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:46.081373 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c334eeb-957b-47ba-9944-c363e0f6b35e-metrics-certs\") pod \"maas-api-e2e-worker-w1-e131af-58475b998f-bx6vp\" (UID: \"3c334eeb-957b-47ba-9944-c363e0f6b35e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e131af-58475b998f-bx6vp" Oct 29 22:50:46.081591 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:46.081463 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3c334eeb-957b-47ba-9944-c363e0f6b35e-maas-api-tls\") pod \"maas-api-e2e-worker-w1-e131af-58475b998f-bx6vp\" (UID: \"3c334eeb-957b-47ba-9944-c363e0f6b35e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e131af-58475b998f-bx6vp" Oct 29 22:50:46.081708 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:46.081584 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c334eeb-957b-47ba-9944-c363e0f6b35e-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-e131af-58475b998f-bx6vp\" (UID: \"3c334eeb-957b-47ba-9944-c363e0f6b35e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e131af-58475b998f-bx6vp" Oct 29 22:50:46.082273 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:46.082248 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c334eeb-957b-47ba-9944-c363e0f6b35e-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-e131af-58475b998f-bx6vp\" (UID: \"3c334eeb-957b-47ba-9944-c363e0f6b35e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e131af-58475b998f-bx6vp" Oct 29 22:50:46.083829 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:46.083809 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3c334eeb-957b-47ba-9944-c363e0f6b35e-maas-api-tls\") pod \"maas-api-e2e-worker-w1-e131af-58475b998f-bx6vp\" (UID: \"3c334eeb-957b-47ba-9944-c363e0f6b35e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e131af-58475b998f-bx6vp" Oct 29 22:50:46.083901 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:46.083869 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c334eeb-957b-47ba-9944-c363e0f6b35e-metrics-certs\") pod \"maas-api-e2e-worker-w1-e131af-58475b998f-bx6vp\" (UID: \"3c334eeb-957b-47ba-9944-c363e0f6b35e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e131af-58475b998f-bx6vp" Oct 29 22:50:46.102688 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:46.102663 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7d2j6\" (UniqueName: \"kubernetes.io/projected/3c334eeb-957b-47ba-9944-c363e0f6b35e-kube-api-access-7d2j6\") pod \"maas-api-e2e-worker-w1-e131af-58475b998f-bx6vp\" (UID: \"3c334eeb-957b-47ba-9944-c363e0f6b35e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e131af-58475b998f-bx6vp" Oct 29 22:50:46.111215 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:46.111192 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:50:46.216907 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:46.216878 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e131af-58475b998f-bx6vp" Oct 29 22:50:46.218937 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:46.218904 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:50:46.526483 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:46.526445 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-e131af-58475b998f-bx6vp"] Oct 29 22:50:46.528498 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:50:46.528474 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c334eeb_957b_47ba_9944_c363e0f6b35e.slice/crio-645e44971e30420b4c43d5295b253c00739bf855ca24d1b754bf1e2f3d9ef625 WatchSource:0}: Error finding container 645e44971e30420b4c43d5295b253c00739bf855ca24d1b754bf1e2f3d9ef625: Status 404 returned error can't find the container with id 645e44971e30420b4c43d5295b253c00739bf855ca24d1b754bf1e2f3d9ef625 Oct 29 22:50:46.873842 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:46.873815 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e131af-58475b998f-bx6vp" event={"ID":"3c334eeb-957b-47ba-9944-c363e0f6b35e","Type":"ContainerStarted","Data":"645e44971e30420b4c43d5295b253c00739bf855ca24d1b754bf1e2f3d9ef625"} Oct 29 22:50:47.881948 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:47.881908 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e131af-58475b998f-bx6vp" event={"ID":"3c334eeb-957b-47ba-9944-c363e0f6b35e","Type":"ContainerStarted","Data":"6c6ca1333e07a3e01388e13583348997f943c971b4f9de8b303868fe38113c7b"} Oct 29 22:50:47.882345 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:47.881974 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e131af-58475b998f-bx6vp" Oct 29 22:50:47.921882 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:47.921833 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e131af-58475b998f-bx6vp" podStartSLOduration=2.621592148 podStartE2EDuration="2.921821143s" totalImagesPullingTime="300.228995ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:50:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:50:47.920582298 +0000 UTC m=+1028.703795726" watchObservedRunningTime="2026-10-29 22:50:47.921821143 +0000 UTC m=+1028.705034631" Oct 29 22:50:47.957885 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:47.957845 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:50:49.027867 ip-10-0-140-74 kubenswrapper[5057]: E1029 22:50:49.027826 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d2dnn" containerName="extractor" Oct 29 22:50:49.139304 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:49.139271 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v2tfm"] Oct 29 22:50:49.797131 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:49.797102 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:50:51.982063 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:51.981974 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:50:53.896194 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:53.896162 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e131af-58475b998f-bx6vp" Oct 29 22:50:54.642057 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:54.642019 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:50:56.916137 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:56.916092 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:50:57.551866 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:57.551823 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:50:57.662055 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:57.661994 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:50:57.782854 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:57.782816 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:50:59.687957 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:59.687924 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:50:59.797438 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:50:59.797402 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:51:00.221584 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:00.221552 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-639067f9-6b77796fcb-plz6x"] Oct 29 22:51:00.258769 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:00.258730 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-639067f9-588f8ccf99-89pzl"] Oct 29 22:51:00.373500 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:00.373465 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:51:00.494529 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:00.494434 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:51:01.155090 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:01.155055 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-72818393-68f6b9db5b-2xnkt"] Oct 29 22:51:01.190921 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:01.190890 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-72818393-648db7d5c9-55zdf"] Oct 29 22:51:01.319566 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:01.319519 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:51:01.375604 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:01.375574 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-72818393-5c8b48595d-24b86"] Oct 29 22:51:01.375804 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:01.375763 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-72818393-5c8b48595d-24b86" podUID="3871fc17-a6ab-407c-944d-8f862b323e2d" containerName="maas-api" containerID="cri-o://43f95a561100305373e64427fa7f323bcef1903a18e3d9ed893bbf22483820ea" gracePeriod=30 Oct 29 22:51:01.433488 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:01.433446 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:51:01.552887 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:01.552725 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:51:01.653553 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:01.653526 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-72818393-5c8b48595d-24b86" Oct 29 22:51:01.673980 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:01.673911 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3871fc17-a6ab-407c-944d-8f862b323e2d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3871fc17-a6ab-407c-944d-8f862b323e2d-metrics-certs\") pod \"3871fc17-a6ab-407c-944d-8f862b323e2d\" (UID: \"3871fc17-a6ab-407c-944d-8f862b323e2d\") " Oct 29 22:51:01.673980 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:01.673953 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3871fc17-a6ab-407c-944d-8f862b323e2d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3871fc17-a6ab-407c-944d-8f862b323e2d-maas-api-tls\") pod \"3871fc17-a6ab-407c-944d-8f862b323e2d\" (UID: \"3871fc17-a6ab-407c-944d-8f862b323e2d\") " Oct 29 22:51:01.674147 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:01.674072 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3871fc17-a6ab-407c-944d-8f862b323e2d-kube-api-access-q4kgg\" (UniqueName: \"kubernetes.io/projected/3871fc17-a6ab-407c-944d-8f862b323e2d-kube-api-access-q4kgg\") pod \"3871fc17-a6ab-407c-944d-8f862b323e2d\" (UID: \"3871fc17-a6ab-407c-944d-8f862b323e2d\") " Oct 29 22:51:01.674147 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:01.674101 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3871fc17-a6ab-407c-944d-8f862b323e2d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3871fc17-a6ab-407c-944d-8f862b323e2d-openshift-service-ca\") pod \"3871fc17-a6ab-407c-944d-8f862b323e2d\" (UID: \"3871fc17-a6ab-407c-944d-8f862b323e2d\") " Oct 29 22:51:01.674598 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:01.674571 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3871fc17-a6ab-407c-944d-8f862b323e2d-openshift-service-ca" pod "3871fc17-a6ab-407c-944d-8f862b323e2d" (UID: "3871fc17-a6ab-407c-944d-8f862b323e2d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:51:01.675997 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:01.675973 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3871fc17-a6ab-407c-944d-8f862b323e2d-metrics-certs" pod "3871fc17-a6ab-407c-944d-8f862b323e2d" (UID: "3871fc17-a6ab-407c-944d-8f862b323e2d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:51:01.676108 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:01.676026 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3871fc17-a6ab-407c-944d-8f862b323e2d-maas-api-tls" pod "3871fc17-a6ab-407c-944d-8f862b323e2d" (UID: "3871fc17-a6ab-407c-944d-8f862b323e2d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:51:01.676178 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:01.676153 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3871fc17-a6ab-407c-944d-8f862b323e2d-kube-api-access-q4kgg" pod "3871fc17-a6ab-407c-944d-8f862b323e2d" (UID: "3871fc17-a6ab-407c-944d-8f862b323e2d"). InnerVolumeSpecName "kube-api-access-q4kgg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:51:01.775590 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:01.775555 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q4kgg\" (UniqueName: \"kubernetes.io/projected/3871fc17-a6ab-407c-944d-8f862b323e2d-kube-api-access-q4kgg\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 22:51:01.775590 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:01.775583 5057 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3871fc17-a6ab-407c-944d-8f862b323e2d-openshift-service-ca\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 22:51:01.775590 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:01.775593 5057 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3871fc17-a6ab-407c-944d-8f862b323e2d-metrics-certs\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 22:51:01.775820 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:01.775601 5057 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3871fc17-a6ab-407c-944d-8f862b323e2d-maas-api-tls\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 22:51:01.984823 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:01.984788 5057 generic.go:356] "Generic (PLEG): container finished" podID="3871fc17-a6ab-407c-944d-8f862b323e2d" containerID="43f95a561100305373e64427fa7f323bcef1903a18e3d9ed893bbf22483820ea" exitCode=0 Oct 29 22:51:01.985007 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:01.984847 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-72818393-5c8b48595d-24b86" Oct 29 22:51:01.985007 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:01.984873 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-72818393-5c8b48595d-24b86" event={"ID":"3871fc17-a6ab-407c-944d-8f862b323e2d","Type":"ContainerDied","Data":"43f95a561100305373e64427fa7f323bcef1903a18e3d9ed893bbf22483820ea"} Oct 29 22:51:01.985007 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:01.984949 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-72818393-5c8b48595d-24b86" event={"ID":"3871fc17-a6ab-407c-944d-8f862b323e2d","Type":"ContainerDied","Data":"6800319df2381c35c6e9f9f9d5c54f6a9401f9963c615dc2b51ea181b1bc05da"} Oct 29 22:51:01.985007 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:01.984973 5057 scope.go:122] "RemoveContainer" containerID="43f95a561100305373e64427fa7f323bcef1903a18e3d9ed893bbf22483820ea" Oct 29 22:51:01.995917 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:01.995887 5057 scope.go:122] "RemoveContainer" containerID="43f95a561100305373e64427fa7f323bcef1903a18e3d9ed893bbf22483820ea" Oct 29 22:51:01.996165 ip-10-0-140-74 kubenswrapper[5057]: E1029 22:51:01.996146 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43f95a561100305373e64427fa7f323bcef1903a18e3d9ed893bbf22483820ea\": container with ID starting with 43f95a561100305373e64427fa7f323bcef1903a18e3d9ed893bbf22483820ea not found: ID does not exist" containerID="43f95a561100305373e64427fa7f323bcef1903a18e3d9ed893bbf22483820ea" Oct 29 22:51:01.996240 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:01.996171 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43f95a561100305373e64427fa7f323bcef1903a18e3d9ed893bbf22483820ea"} err="failed to get container status \"43f95a561100305373e64427fa7f323bcef1903a18e3d9ed893bbf22483820ea\": rpc error: code = NotFound desc = could not find container \"43f95a561100305373e64427fa7f323bcef1903a18e3d9ed893bbf22483820ea\": container with ID starting with 43f95a561100305373e64427fa7f323bcef1903a18e3d9ed893bbf22483820ea not found: ID does not exist" Oct 29 22:51:02.030763 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:02.030738 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-72818393-5c8b48595d-24b86"] Oct 29 22:51:02.040419 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:02.040394 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-72818393-5c8b48595d-24b86"] Oct 29 22:51:02.264984 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:02.264907 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:51:02.680552 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:02.680466 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:51:02.790925 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:02.790886 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:51:03.328646 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:03.328605 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3871fc17-a6ab-407c-944d-8f862b323e2d" path="/var/lib/kubelet/pods/3871fc17-a6ab-407c-944d-8f862b323e2d/volumes" Oct 29 22:51:04.195461 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:04.195426 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:51:04.368156 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:04.368120 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:51:04.976071 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:04.976036 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v2tfm"] Oct 29 22:51:05.162044 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:05.162011 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:51:05.222721 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:05.222685 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-639067f9-6b77796fcb-plz6x" podUID="185a7421-26cf-46ae-a65d-ed283704904e" containerName="payload-processing" containerID="cri-o://b1b3daa8cd2ceaa05692f72b604e53b909e26770031a6e54af10e230600426bf" gracePeriod=25 Oct 29 22:51:05.259319 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:05.259283 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-639067f9-588f8ccf99-89pzl" podUID="31493bbc-acc0-423b-9968-3be0bca796b5" containerName="payload-pre-processing" containerID="cri-o://87c3a235f393e5b2eec80a4badf1b7023c9c05f17eb13b772474177ef4e331f1" gracePeriod=25 Oct 29 22:51:05.567910 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:05.567886 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-639067f9-6b77796fcb-plz6x" Oct 29 22:51:05.593694 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:05.593672 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-639067f9-588f8ccf99-89pzl" Oct 29 22:51:05.623221 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:05.623171 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/185a7421-26cf-46ae-a65d-ed283704904e-kube-api-access-7pp5g\" (UniqueName: \"kubernetes.io/projected/185a7421-26cf-46ae-a65d-ed283704904e-kube-api-access-7pp5g\") pod \"185a7421-26cf-46ae-a65d-ed283704904e\" (UID: \"185a7421-26cf-46ae-a65d-ed283704904e\") " Oct 29 22:51:05.623492 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:05.623238 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/185a7421-26cf-46ae-a65d-ed283704904e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/185a7421-26cf-46ae-a65d-ed283704904e-plugins-config-volume\") pod \"185a7421-26cf-46ae-a65d-ed283704904e\" (UID: \"185a7421-26cf-46ae-a65d-ed283704904e\") " Oct 29 22:51:05.623492 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:05.623313 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31493bbc-acc0-423b-9968-3be0bca796b5-kube-api-access-blphc\" (UniqueName: \"kubernetes.io/projected/31493bbc-acc0-423b-9968-3be0bca796b5-kube-api-access-blphc\") pod \"31493bbc-acc0-423b-9968-3be0bca796b5\" (UID: \"31493bbc-acc0-423b-9968-3be0bca796b5\") " Oct 29 22:51:05.623492 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:05.623443 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31493bbc-acc0-423b-9968-3be0bca796b5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31493bbc-acc0-423b-9968-3be0bca796b5-plugins-config-volume\") pod \"31493bbc-acc0-423b-9968-3be0bca796b5\" (UID: \"31493bbc-acc0-423b-9968-3be0bca796b5\") " Oct 29 22:51:05.624033 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:05.623802 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/185a7421-26cf-46ae-a65d-ed283704904e-plugins-config-volume" pod "185a7421-26cf-46ae-a65d-ed283704904e" (UID: "185a7421-26cf-46ae-a65d-ed283704904e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:51:05.624229 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:05.624201 5057 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/185a7421-26cf-46ae-a65d-ed283704904e-plugins-config-volume\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 22:51:05.624297 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:05.624233 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31493bbc-acc0-423b-9968-3be0bca796b5-plugins-config-volume" pod "31493bbc-acc0-423b-9968-3be0bca796b5" (UID: "31493bbc-acc0-423b-9968-3be0bca796b5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:51:05.626253 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:05.626229 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31493bbc-acc0-423b-9968-3be0bca796b5-kube-api-access-blphc" pod "31493bbc-acc0-423b-9968-3be0bca796b5" (UID: "31493bbc-acc0-423b-9968-3be0bca796b5"). InnerVolumeSpecName "kube-api-access-blphc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:51:05.626361 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:05.626258 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/185a7421-26cf-46ae-a65d-ed283704904e-kube-api-access-7pp5g" pod "185a7421-26cf-46ae-a65d-ed283704904e" (UID: "185a7421-26cf-46ae-a65d-ed283704904e"). InnerVolumeSpecName "kube-api-access-7pp5g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:51:05.726633 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:05.726602 5057 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31493bbc-acc0-423b-9968-3be0bca796b5-plugins-config-volume\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 22:51:05.726633 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:05.726632 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7pp5g\" (UniqueName: \"kubernetes.io/projected/185a7421-26cf-46ae-a65d-ed283704904e-kube-api-access-7pp5g\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 22:51:05.726862 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:05.726646 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-blphc\" (UniqueName: \"kubernetes.io/projected/31493bbc-acc0-423b-9968-3be0bca796b5-kube-api-access-blphc\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 22:51:05.776367 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:05.776275 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:51:05.893175 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:05.893137 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:51:05.947696 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:05.947650 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-639067f9-74c94b54c8-qjsft"] Oct 29 22:51:05.947963 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:05.947905 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-639067f9-74c94b54c8-qjsft" podUID="ee18842c-3a3f-4148-807f-09838ef6ccf9" containerName="maas-api" containerID="cri-o://5bbcf00c9a151a66497559fbf947831fe2e17b0a23b2a035092cd09046c65f79" gracePeriod=30 Oct 29 22:51:06.010774 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:06.010744 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1db2136c-5588cff679-gkt2z"] Oct 29 22:51:06.015610 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:06.015584 5057 generic.go:356] "Generic (PLEG): container finished" podID="185a7421-26cf-46ae-a65d-ed283704904e" containerID="b1b3daa8cd2ceaa05692f72b604e53b909e26770031a6e54af10e230600426bf" exitCode=0 Oct 29 22:51:06.015760 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:06.015625 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-639067f9-6b77796fcb-plz6x" Oct 29 22:51:06.015760 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:06.015635 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-639067f9-6b77796fcb-plz6x" event={"ID":"185a7421-26cf-46ae-a65d-ed283704904e","Type":"ContainerDied","Data":"b1b3daa8cd2ceaa05692f72b604e53b909e26770031a6e54af10e230600426bf"} Oct 29 22:51:06.015760 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:06.015705 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-639067f9-6b77796fcb-plz6x" event={"ID":"185a7421-26cf-46ae-a65d-ed283704904e","Type":"ContainerDied","Data":"bdbf276ce2c03ab3997feb5add02316f8c828ac62a19765ddc5765a7c04579d1"} Oct 29 22:51:06.015760 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:06.015737 5057 scope.go:122] "RemoveContainer" containerID="b1b3daa8cd2ceaa05692f72b604e53b909e26770031a6e54af10e230600426bf" Oct 29 22:51:06.017419 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:06.017394 5057 generic.go:356] "Generic (PLEG): container finished" podID="31493bbc-acc0-423b-9968-3be0bca796b5" containerID="87c3a235f393e5b2eec80a4badf1b7023c9c05f17eb13b772474177ef4e331f1" exitCode=0 Oct 29 22:51:06.017534 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:06.017437 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-639067f9-588f8ccf99-89pzl" Oct 29 22:51:06.017534 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:06.017475 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-639067f9-588f8ccf99-89pzl" event={"ID":"31493bbc-acc0-423b-9968-3be0bca796b5","Type":"ContainerDied","Data":"87c3a235f393e5b2eec80a4badf1b7023c9c05f17eb13b772474177ef4e331f1"} Oct 29 22:51:06.017534 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:06.017516 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-639067f9-588f8ccf99-89pzl" event={"ID":"31493bbc-acc0-423b-9968-3be0bca796b5","Type":"ContainerDied","Data":"aa97cb144251574f478405f6220e8c2a75f2225c4fe4ef1de074c2a1a769460f"} Oct 29 22:51:06.033495 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:06.033468 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:51:06.076915 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:06.076886 5057 scope.go:122] "RemoveContainer" containerID="b1b3daa8cd2ceaa05692f72b604e53b909e26770031a6e54af10e230600426bf" Oct 29 22:51:06.077304 ip-10-0-140-74 kubenswrapper[5057]: E1029 22:51:06.077270 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1b3daa8cd2ceaa05692f72b604e53b909e26770031a6e54af10e230600426bf\": container with ID starting with b1b3daa8cd2ceaa05692f72b604e53b909e26770031a6e54af10e230600426bf not found: ID does not exist" containerID="b1b3daa8cd2ceaa05692f72b604e53b909e26770031a6e54af10e230600426bf" Oct 29 22:51:06.077413 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:06.077317 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1b3daa8cd2ceaa05692f72b604e53b909e26770031a6e54af10e230600426bf"} err="failed to get container status \"b1b3daa8cd2ceaa05692f72b604e53b909e26770031a6e54af10e230600426bf\": rpc error: code = NotFound desc = could not find container \"b1b3daa8cd2ceaa05692f72b604e53b909e26770031a6e54af10e230600426bf\": container with ID starting with b1b3daa8cd2ceaa05692f72b604e53b909e26770031a6e54af10e230600426bf not found: ID does not exist" Oct 29 22:51:06.077413 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:06.077356 5057 scope.go:122] "RemoveContainer" containerID="87c3a235f393e5b2eec80a4badf1b7023c9c05f17eb13b772474177ef4e331f1" Oct 29 22:51:06.089157 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:06.089137 5057 scope.go:122] "RemoveContainer" containerID="87c3a235f393e5b2eec80a4badf1b7023c9c05f17eb13b772474177ef4e331f1" Oct 29 22:51:06.089569 ip-10-0-140-74 kubenswrapper[5057]: E1029 22:51:06.089471 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87c3a235f393e5b2eec80a4badf1b7023c9c05f17eb13b772474177ef4e331f1\": container with ID starting with 87c3a235f393e5b2eec80a4badf1b7023c9c05f17eb13b772474177ef4e331f1 not found: ID does not exist" containerID="87c3a235f393e5b2eec80a4badf1b7023c9c05f17eb13b772474177ef4e331f1" Oct 29 22:51:06.089569 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:06.089508 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87c3a235f393e5b2eec80a4badf1b7023c9c05f17eb13b772474177ef4e331f1"} err="failed to get container status \"87c3a235f393e5b2eec80a4badf1b7023c9c05f17eb13b772474177ef4e331f1\": rpc error: code = NotFound desc = could not find container \"87c3a235f393e5b2eec80a4badf1b7023c9c05f17eb13b772474177ef4e331f1\": container with ID starting with 87c3a235f393e5b2eec80a4badf1b7023c9c05f17eb13b772474177ef4e331f1 not found: ID does not exist" Oct 29 22:51:06.093278 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:06.093251 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-639067f9-588f8ccf99-89pzl"] Oct 29 22:51:06.096139 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:06.096114 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-639067f9-588f8ccf99-89pzl"] Oct 29 22:51:06.130012 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:06.129038 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-639067f9-6b77796fcb-plz6x"] Oct 29 22:51:06.134183 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:06.133695 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-639067f9-6b77796fcb-plz6x"] Oct 29 22:51:06.164362 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:06.150735 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:51:06.164362 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:06.155952 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-72818393-68f6b9db5b-2xnkt" podUID="9e333bec-919e-49c4-b9c0-a732247ce509" containerName="payload-processing" containerID="cri-o://7b49f6e3b69f95038d639b02f9bd69bcd2147641f8cbaca6f422baa84a02c8dc" gracePeriod=25 Oct 29 22:51:06.191714 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:06.191628 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-72818393-648db7d5c9-55zdf" podUID="89a6412b-1be7-4a77-bb4d-c8f81c7d6f43" containerName="payload-pre-processing" containerID="cri-o://741eab13ee0928fd6409b13ec80805bf759a43055e70c6fb0b4e30532665a4f0" gracePeriod=25 Oct 29 22:51:06.261392 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:06.261270 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:51:06.344381 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:06.343578 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-639067f9-74c94b54c8-qjsft" Oct 29 22:51:06.439441 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:06.439410 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ee18842c-3a3f-4148-807f-09838ef6ccf9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ee18842c-3a3f-4148-807f-09838ef6ccf9-maas-api-tls\") pod \"ee18842c-3a3f-4148-807f-09838ef6ccf9\" (UID: \"ee18842c-3a3f-4148-807f-09838ef6ccf9\") " Oct 29 22:51:06.439603 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:06.439459 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ee18842c-3a3f-4148-807f-09838ef6ccf9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ee18842c-3a3f-4148-807f-09838ef6ccf9-metrics-certs\") pod \"ee18842c-3a3f-4148-807f-09838ef6ccf9\" (UID: \"ee18842c-3a3f-4148-807f-09838ef6ccf9\") " Oct 29 22:51:06.439603 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:06.439524 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee18842c-3a3f-4148-807f-09838ef6ccf9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee18842c-3a3f-4148-807f-09838ef6ccf9-openshift-service-ca\") pod \"ee18842c-3a3f-4148-807f-09838ef6ccf9\" (UID: \"ee18842c-3a3f-4148-807f-09838ef6ccf9\") " Oct 29 22:51:06.439748 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:06.439711 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee18842c-3a3f-4148-807f-09838ef6ccf9-kube-api-access-v5g8k\" (UniqueName: \"kubernetes.io/projected/ee18842c-3a3f-4148-807f-09838ef6ccf9-kube-api-access-v5g8k\") pod \"ee18842c-3a3f-4148-807f-09838ef6ccf9\" (UID: \"ee18842c-3a3f-4148-807f-09838ef6ccf9\") " Oct 29 22:51:06.440140 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:06.440114 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee18842c-3a3f-4148-807f-09838ef6ccf9-openshift-service-ca" pod "ee18842c-3a3f-4148-807f-09838ef6ccf9" (UID: "ee18842c-3a3f-4148-807f-09838ef6ccf9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:51:06.441970 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:06.441945 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee18842c-3a3f-4148-807f-09838ef6ccf9-maas-api-tls" pod "ee18842c-3a3f-4148-807f-09838ef6ccf9" (UID: "ee18842c-3a3f-4148-807f-09838ef6ccf9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:51:06.441970 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:06.441955 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee18842c-3a3f-4148-807f-09838ef6ccf9-metrics-certs" pod "ee18842c-3a3f-4148-807f-09838ef6ccf9" (UID: "ee18842c-3a3f-4148-807f-09838ef6ccf9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:51:06.442243 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:06.442224 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee18842c-3a3f-4148-807f-09838ef6ccf9-kube-api-access-v5g8k" pod "ee18842c-3a3f-4148-807f-09838ef6ccf9" (UID: "ee18842c-3a3f-4148-807f-09838ef6ccf9"). InnerVolumeSpecName "kube-api-access-v5g8k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:51:06.489966 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:06.489932 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-72818393-68f6b9db5b-2xnkt" Oct 29 22:51:06.504760 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:06.504734 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-72818393-648db7d5c9-55zdf" Oct 29 22:51:06.541409 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:06.541371 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89a6412b-1be7-4a77-bb4d-c8f81c7d6f43-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89a6412b-1be7-4a77-bb4d-c8f81c7d6f43-plugins-config-volume\") pod \"89a6412b-1be7-4a77-bb4d-c8f81c7d6f43\" (UID: \"89a6412b-1be7-4a77-bb4d-c8f81c7d6f43\") " Oct 29 22:51:06.541606 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:06.541574 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e333bec-919e-49c4-b9c0-a732247ce509-kube-api-access-qfmwl\" (UniqueName: \"kubernetes.io/projected/9e333bec-919e-49c4-b9c0-a732247ce509-kube-api-access-qfmwl\") pod \"9e333bec-919e-49c4-b9c0-a732247ce509\" (UID: \"9e333bec-919e-49c4-b9c0-a732247ce509\") " Oct 29 22:51:06.541685 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:06.541625 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89a6412b-1be7-4a77-bb4d-c8f81c7d6f43-kube-api-access-nzfdj\" (UniqueName: \"kubernetes.io/projected/89a6412b-1be7-4a77-bb4d-c8f81c7d6f43-kube-api-access-nzfdj\") pod \"89a6412b-1be7-4a77-bb4d-c8f81c7d6f43\" (UID: \"89a6412b-1be7-4a77-bb4d-c8f81c7d6f43\") " Oct 29 22:51:06.541685 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:06.541675 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e333bec-919e-49c4-b9c0-a732247ce509-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e333bec-919e-49c4-b9c0-a732247ce509-plugins-config-volume\") pod \"9e333bec-919e-49c4-b9c0-a732247ce509\" (UID: \"9e333bec-919e-49c4-b9c0-a732247ce509\") " Oct 29 22:51:06.541825 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:06.541797 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89a6412b-1be7-4a77-bb4d-c8f81c7d6f43-plugins-config-volume" pod "89a6412b-1be7-4a77-bb4d-c8f81c7d6f43" (UID: "89a6412b-1be7-4a77-bb4d-c8f81c7d6f43"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:51:06.542169 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:06.542145 5057 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ee18842c-3a3f-4148-807f-09838ef6ccf9-maas-api-tls\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 22:51:06.542292 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:06.542172 5057 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ee18842c-3a3f-4148-807f-09838ef6ccf9-metrics-certs\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 22:51:06.542292 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:06.542186 5057 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee18842c-3a3f-4148-807f-09838ef6ccf9-openshift-service-ca\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 22:51:06.542292 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:06.542200 5057 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89a6412b-1be7-4a77-bb4d-c8f81c7d6f43-plugins-config-volume\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 22:51:06.542292 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:06.542214 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v5g8k\" (UniqueName: \"kubernetes.io/projected/ee18842c-3a3f-4148-807f-09838ef6ccf9-kube-api-access-v5g8k\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 22:51:06.542555 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:06.542311 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e333bec-919e-49c4-b9c0-a732247ce509-plugins-config-volume" pod "9e333bec-919e-49c4-b9c0-a732247ce509" (UID: "9e333bec-919e-49c4-b9c0-a732247ce509"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:51:06.543784 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:06.543760 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89a6412b-1be7-4a77-bb4d-c8f81c7d6f43-kube-api-access-nzfdj" pod "89a6412b-1be7-4a77-bb4d-c8f81c7d6f43" (UID: "89a6412b-1be7-4a77-bb4d-c8f81c7d6f43"). InnerVolumeSpecName "kube-api-access-nzfdj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:51:06.544060 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:06.544047 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e333bec-919e-49c4-b9c0-a732247ce509-kube-api-access-qfmwl" pod "9e333bec-919e-49c4-b9c0-a732247ce509" (UID: "9e333bec-919e-49c4-b9c0-a732247ce509"). InnerVolumeSpecName "kube-api-access-qfmwl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:51:06.644082 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:06.644004 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qfmwl\" (UniqueName: \"kubernetes.io/projected/9e333bec-919e-49c4-b9c0-a732247ce509-kube-api-access-qfmwl\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 22:51:06.644082 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:06.644035 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nzfdj\" (UniqueName: \"kubernetes.io/projected/89a6412b-1be7-4a77-bb4d-c8f81c7d6f43-kube-api-access-nzfdj\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 22:51:06.644082 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:06.644045 5057 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e333bec-919e-49c4-b9c0-a732247ce509-plugins-config-volume\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 22:51:06.717230 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:06.717184 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:51:06.845089 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:06.845054 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:51:06.953502 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:06.953476 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:51:07.030622 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:07.030592 5057 generic.go:356] "Generic (PLEG): container finished" podID="9e333bec-919e-49c4-b9c0-a732247ce509" containerID="7b49f6e3b69f95038d639b02f9bd69bcd2147641f8cbaca6f422baa84a02c8dc" exitCode=0 Oct 29 22:51:07.030818 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:07.030636 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-72818393-68f6b9db5b-2xnkt" Oct 29 22:51:07.030818 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:07.030670 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-72818393-68f6b9db5b-2xnkt" event={"ID":"9e333bec-919e-49c4-b9c0-a732247ce509","Type":"ContainerDied","Data":"7b49f6e3b69f95038d639b02f9bd69bcd2147641f8cbaca6f422baa84a02c8dc"} Oct 29 22:51:07.030818 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:07.030746 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-72818393-68f6b9db5b-2xnkt" event={"ID":"9e333bec-919e-49c4-b9c0-a732247ce509","Type":"ContainerDied","Data":"65c01662423592a486a854c518dcf784e7371a0babea4f1fcaf5ffbd94c869ab"} Oct 29 22:51:07.030818 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:07.030776 5057 scope.go:122] "RemoveContainer" containerID="7b49f6e3b69f95038d639b02f9bd69bcd2147641f8cbaca6f422baa84a02c8dc" Oct 29 22:51:07.032068 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:07.032039 5057 generic.go:356] "Generic (PLEG): container finished" podID="89a6412b-1be7-4a77-bb4d-c8f81c7d6f43" containerID="741eab13ee0928fd6409b13ec80805bf759a43055e70c6fb0b4e30532665a4f0" exitCode=0 Oct 29 22:51:07.032283 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:07.032085 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-72818393-648db7d5c9-55zdf" Oct 29 22:51:07.032410 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:07.032101 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-72818393-648db7d5c9-55zdf" event={"ID":"89a6412b-1be7-4a77-bb4d-c8f81c7d6f43","Type":"ContainerDied","Data":"741eab13ee0928fd6409b13ec80805bf759a43055e70c6fb0b4e30532665a4f0"} Oct 29 22:51:07.032478 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:07.032398 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-72818393-648db7d5c9-55zdf" event={"ID":"89a6412b-1be7-4a77-bb4d-c8f81c7d6f43","Type":"ContainerDied","Data":"8ae2a7b9f7b75538f858d10dd971749bbd90515bf67b5c7c9983dd2ce1b1f1bd"} Oct 29 22:51:07.034677 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:07.034652 5057 generic.go:356] "Generic (PLEG): container finished" podID="ee18842c-3a3f-4148-807f-09838ef6ccf9" containerID="5bbcf00c9a151a66497559fbf947831fe2e17b0a23b2a035092cd09046c65f79" exitCode=0 Oct 29 22:51:07.034752 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:07.034701 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-639067f9-74c94b54c8-qjsft" event={"ID":"ee18842c-3a3f-4148-807f-09838ef6ccf9","Type":"ContainerDied","Data":"5bbcf00c9a151a66497559fbf947831fe2e17b0a23b2a035092cd09046c65f79"} Oct 29 22:51:07.034752 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:07.034737 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-639067f9-74c94b54c8-qjsft" event={"ID":"ee18842c-3a3f-4148-807f-09838ef6ccf9","Type":"ContainerDied","Data":"de4b8d9ec40116eb05a28706f2ef6b5374b6e7bc7f8411b9d4b622cd9bb2e2f9"} Oct 29 22:51:07.034839 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:07.034805 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-639067f9-74c94b54c8-qjsft" Oct 29 22:51:07.043022 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:07.043006 5057 scope.go:122] "RemoveContainer" containerID="7b49f6e3b69f95038d639b02f9bd69bcd2147641f8cbaca6f422baa84a02c8dc" Oct 29 22:51:07.043413 ip-10-0-140-74 kubenswrapper[5057]: E1029 22:51:07.043274 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b49f6e3b69f95038d639b02f9bd69bcd2147641f8cbaca6f422baa84a02c8dc\": container with ID starting with 7b49f6e3b69f95038d639b02f9bd69bcd2147641f8cbaca6f422baa84a02c8dc not found: ID does not exist" containerID="7b49f6e3b69f95038d639b02f9bd69bcd2147641f8cbaca6f422baa84a02c8dc" Oct 29 22:51:07.043413 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:07.043305 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b49f6e3b69f95038d639b02f9bd69bcd2147641f8cbaca6f422baa84a02c8dc"} err="failed to get container status \"7b49f6e3b69f95038d639b02f9bd69bcd2147641f8cbaca6f422baa84a02c8dc\": rpc error: code = NotFound desc = could not find container \"7b49f6e3b69f95038d639b02f9bd69bcd2147641f8cbaca6f422baa84a02c8dc\": container with ID starting with 7b49f6e3b69f95038d639b02f9bd69bcd2147641f8cbaca6f422baa84a02c8dc not found: ID does not exist" Oct 29 22:51:07.043413 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:07.043326 5057 scope.go:122] "RemoveContainer" containerID="741eab13ee0928fd6409b13ec80805bf759a43055e70c6fb0b4e30532665a4f0" Oct 29 22:51:07.052941 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:07.052924 5057 scope.go:122] "RemoveContainer" containerID="741eab13ee0928fd6409b13ec80805bf759a43055e70c6fb0b4e30532665a4f0" Oct 29 22:51:07.053178 ip-10-0-140-74 kubenswrapper[5057]: E1029 22:51:07.053158 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"741eab13ee0928fd6409b13ec80805bf759a43055e70c6fb0b4e30532665a4f0\": container with ID starting with 741eab13ee0928fd6409b13ec80805bf759a43055e70c6fb0b4e30532665a4f0 not found: ID does not exist" containerID="741eab13ee0928fd6409b13ec80805bf759a43055e70c6fb0b4e30532665a4f0" Oct 29 22:51:07.053249 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:07.053188 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"741eab13ee0928fd6409b13ec80805bf759a43055e70c6fb0b4e30532665a4f0"} err="failed to get container status \"741eab13ee0928fd6409b13ec80805bf759a43055e70c6fb0b4e30532665a4f0\": rpc error: code = NotFound desc = could not find container \"741eab13ee0928fd6409b13ec80805bf759a43055e70c6fb0b4e30532665a4f0\": container with ID starting with 741eab13ee0928fd6409b13ec80805bf759a43055e70c6fb0b4e30532665a4f0 not found: ID does not exist" Oct 29 22:51:07.053249 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:07.053209 5057 scope.go:122] "RemoveContainer" containerID="5bbcf00c9a151a66497559fbf947831fe2e17b0a23b2a035092cd09046c65f79" Oct 29 22:51:07.061845 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:07.061823 5057 scope.go:122] "RemoveContainer" containerID="5bbcf00c9a151a66497559fbf947831fe2e17b0a23b2a035092cd09046c65f79" Oct 29 22:51:07.062110 ip-10-0-140-74 kubenswrapper[5057]: E1029 22:51:07.062092 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5bbcf00c9a151a66497559fbf947831fe2e17b0a23b2a035092cd09046c65f79\": container with ID starting with 5bbcf00c9a151a66497559fbf947831fe2e17b0a23b2a035092cd09046c65f79 not found: ID does not exist" containerID="5bbcf00c9a151a66497559fbf947831fe2e17b0a23b2a035092cd09046c65f79" Oct 29 22:51:07.062155 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:07.062116 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5bbcf00c9a151a66497559fbf947831fe2e17b0a23b2a035092cd09046c65f79"} err="failed to get container status \"5bbcf00c9a151a66497559fbf947831fe2e17b0a23b2a035092cd09046c65f79\": rpc error: code = NotFound desc = could not find container \"5bbcf00c9a151a66497559fbf947831fe2e17b0a23b2a035092cd09046c65f79\": container with ID starting with 5bbcf00c9a151a66497559fbf947831fe2e17b0a23b2a035092cd09046c65f79 not found: ID does not exist" Oct 29 22:51:07.067686 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:07.067662 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:51:07.104384 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:07.104354 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-639067f9-74c94b54c8-qjsft"] Oct 29 22:51:07.113908 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:07.113876 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-639067f9-74c94b54c8-qjsft"] Oct 29 22:51:07.138042 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:07.138015 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-72818393-648db7d5c9-55zdf"] Oct 29 22:51:07.143692 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:07.143669 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-72818393-648db7d5c9-55zdf"] Oct 29 22:51:07.174252 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:07.174230 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-72818393-68f6b9db5b-2xnkt"] Oct 29 22:51:07.176767 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:07.176744 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-72818393-68f6b9db5b-2xnkt"] Oct 29 22:51:07.328298 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:07.328264 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="185a7421-26cf-46ae-a65d-ed283704904e" path="/var/lib/kubelet/pods/185a7421-26cf-46ae-a65d-ed283704904e/volumes" Oct 29 22:51:07.328699 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:07.328683 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="31493bbc-acc0-423b-9968-3be0bca796b5" path="/var/lib/kubelet/pods/31493bbc-acc0-423b-9968-3be0bca796b5/volumes" Oct 29 22:51:07.329031 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:07.329018 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89a6412b-1be7-4a77-bb4d-c8f81c7d6f43" path="/var/lib/kubelet/pods/89a6412b-1be7-4a77-bb4d-c8f81c7d6f43/volumes" Oct 29 22:51:07.329381 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:07.329368 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e333bec-919e-49c4-b9c0-a732247ce509" path="/var/lib/kubelet/pods/9e333bec-919e-49c4-b9c0-a732247ce509/volumes" Oct 29 22:51:07.330041 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:07.330027 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee18842c-3a3f-4148-807f-09838ef6ccf9" path="/var/lib/kubelet/pods/ee18842c-3a3f-4148-807f-09838ef6ccf9/volumes" Oct 29 22:51:07.860619 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:07.860582 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:51:08.175074 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:08.174991 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:51:08.500422 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:08.500390 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:51:08.721142 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:08.721114 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:51:08.839419 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:08.839319 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:51:08.964193 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:08.964158 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:51:09.569666 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:09.569633 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:51:10.087513 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:10.087485 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:51:10.218157 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:10.218122 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:51:10.293677 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:10.292792 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-3c314108-67c4d8d458-hbl94"] Oct 29 22:51:10.332245 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:10.332213 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-3c314108-6769db8fbc-75ccb"] Oct 29 22:51:10.456662 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:10.456622 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:51:10.567054 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:10.567016 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:51:11.011772 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:11.011723 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-1db2136c-5588cff679-gkt2z" podUID="3b2af96a-99d4-46fc-9a2a-e773d96f5898" containerName="payload-pre-processing" containerID="cri-o://ec742484b685dad3ece4771eb72a9f5b79ff91398def48dbd3767b60aad81484" gracePeriod=25 Oct 29 22:51:11.301145 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:11.301111 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-1db2136c-5588cff679-gkt2z" Oct 29 22:51:11.411918 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:11.411884 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b2af96a-99d4-46fc-9a2a-e773d96f5898-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b2af96a-99d4-46fc-9a2a-e773d96f5898-plugins-config-volume\") pod \"3b2af96a-99d4-46fc-9a2a-e773d96f5898\" (UID: \"3b2af96a-99d4-46fc-9a2a-e773d96f5898\") " Oct 29 22:51:11.412097 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:11.411986 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b2af96a-99d4-46fc-9a2a-e773d96f5898-kube-api-access-dq5g8\" (UniqueName: \"kubernetes.io/projected/3b2af96a-99d4-46fc-9a2a-e773d96f5898-kube-api-access-dq5g8\") pod \"3b2af96a-99d4-46fc-9a2a-e773d96f5898\" (UID: \"3b2af96a-99d4-46fc-9a2a-e773d96f5898\") " Oct 29 22:51:11.412393 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:11.412365 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b2af96a-99d4-46fc-9a2a-e773d96f5898-plugins-config-volume" pod "3b2af96a-99d4-46fc-9a2a-e773d96f5898" (UID: "3b2af96a-99d4-46fc-9a2a-e773d96f5898"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:51:11.412541 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:11.412517 5057 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b2af96a-99d4-46fc-9a2a-e773d96f5898-plugins-config-volume\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 22:51:11.414262 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:11.414232 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b2af96a-99d4-46fc-9a2a-e773d96f5898-kube-api-access-dq5g8" pod "3b2af96a-99d4-46fc-9a2a-e773d96f5898" (UID: "3b2af96a-99d4-46fc-9a2a-e773d96f5898"). InnerVolumeSpecName "kube-api-access-dq5g8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:51:11.503076 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:11.503045 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:51:11.514065 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:11.514034 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dq5g8\" (UniqueName: \"kubernetes.io/projected/3b2af96a-99d4-46fc-9a2a-e773d96f5898-kube-api-access-dq5g8\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 22:51:11.763804 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:11.763773 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:51:11.889002 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:11.888965 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:51:11.953758 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:11.953728 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v2tfm"] Oct 29 22:51:12.072089 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:12.072001 5057 generic.go:356] "Generic (PLEG): container finished" podID="3b2af96a-99d4-46fc-9a2a-e773d96f5898" containerID="ec742484b685dad3ece4771eb72a9f5b79ff91398def48dbd3767b60aad81484" exitCode=0 Oct 29 22:51:12.072089 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:12.072047 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-1db2136c-5588cff679-gkt2z" Oct 29 22:51:12.072089 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:12.072080 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-1db2136c-5588cff679-gkt2z" event={"ID":"3b2af96a-99d4-46fc-9a2a-e773d96f5898","Type":"ContainerDied","Data":"ec742484b685dad3ece4771eb72a9f5b79ff91398def48dbd3767b60aad81484"} Oct 29 22:51:12.072395 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:12.072135 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-1db2136c-5588cff679-gkt2z" event={"ID":"3b2af96a-99d4-46fc-9a2a-e773d96f5898","Type":"ContainerDied","Data":"0744881ef1e4096bfac82431ea0c4a6ff13e3c66bc3aceb7021353ab04540e42"} Oct 29 22:51:12.072395 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:12.072164 5057 scope.go:122] "RemoveContainer" containerID="ec742484b685dad3ece4771eb72a9f5b79ff91398def48dbd3767b60aad81484" Oct 29 22:51:12.083703 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:12.083682 5057 scope.go:122] "RemoveContainer" containerID="ec742484b685dad3ece4771eb72a9f5b79ff91398def48dbd3767b60aad81484" Oct 29 22:51:12.083988 ip-10-0-140-74 kubenswrapper[5057]: E1029 22:51:12.083967 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec742484b685dad3ece4771eb72a9f5b79ff91398def48dbd3767b60aad81484\": container with ID starting with ec742484b685dad3ece4771eb72a9f5b79ff91398def48dbd3767b60aad81484 not found: ID does not exist" containerID="ec742484b685dad3ece4771eb72a9f5b79ff91398def48dbd3767b60aad81484" Oct 29 22:51:12.084081 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:12.083995 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec742484b685dad3ece4771eb72a9f5b79ff91398def48dbd3767b60aad81484"} err="failed to get container status \"ec742484b685dad3ece4771eb72a9f5b79ff91398def48dbd3767b60aad81484\": rpc error: code = NotFound desc = could not find container \"ec742484b685dad3ece4771eb72a9f5b79ff91398def48dbd3767b60aad81484\": container with ID starting with ec742484b685dad3ece4771eb72a9f5b79ff91398def48dbd3767b60aad81484 not found: ID does not exist" Oct 29 22:51:12.120581 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:12.120548 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1db2136c-5588cff679-gkt2z"] Oct 29 22:51:12.124482 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:12.124462 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1db2136c-5588cff679-gkt2z"] Oct 29 22:51:12.352126 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:12.352030 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:51:13.304234 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:13.304194 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:51:13.329303 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:13.329268 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b2af96a-99d4-46fc-9a2a-e773d96f5898" path="/var/lib/kubelet/pods/3b2af96a-99d4-46fc-9a2a-e773d96f5898/volumes" Oct 29 22:51:13.414695 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:13.414665 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:51:14.985116 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:14.985074 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:51:15.096062 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:15.096006 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:51:15.167136 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:15.167103 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-17394a-77794c47ff-djggj"] Oct 29 22:51:15.167734 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:15.167720 5057 cpu_manager.go:414] "containerMap: removing container" podUID="ee18842c-3a3f-4148-807f-09838ef6ccf9" containerName="maas-api" Oct 29 22:51:15.167800 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:15.167735 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee18842c-3a3f-4148-807f-09838ef6ccf9" containerName="maas-api" Oct 29 22:51:15.167800 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:15.167751 5057 cpu_manager.go:414] "containerMap: removing container" podUID="185a7421-26cf-46ae-a65d-ed283704904e" containerName="payload-processing" Oct 29 22:51:15.167800 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:15.167756 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="185a7421-26cf-46ae-a65d-ed283704904e" containerName="payload-processing" Oct 29 22:51:15.167800 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:15.167773 5057 cpu_manager.go:414] "containerMap: removing container" podUID="89a6412b-1be7-4a77-bb4d-c8f81c7d6f43" containerName="payload-pre-processing" Oct 29 22:51:15.167800 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:15.167778 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89a6412b-1be7-4a77-bb4d-c8f81c7d6f43" containerName="payload-pre-processing" Oct 29 22:51:15.167800 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:15.167792 5057 cpu_manager.go:414] "containerMap: removing container" podUID="9e333bec-919e-49c4-b9c0-a732247ce509" containerName="payload-processing" Oct 29 22:51:15.167800 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:15.167797 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e333bec-919e-49c4-b9c0-a732247ce509" containerName="payload-processing" Oct 29 22:51:15.168024 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:15.167808 5057 cpu_manager.go:414] "containerMap: removing container" podUID="3871fc17-a6ab-407c-944d-8f862b323e2d" containerName="maas-api" Oct 29 22:51:15.168024 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:15.167813 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3871fc17-a6ab-407c-944d-8f862b323e2d" containerName="maas-api" Oct 29 22:51:15.168024 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:15.167822 5057 cpu_manager.go:414] "containerMap: removing container" podUID="31493bbc-acc0-423b-9968-3be0bca796b5" containerName="payload-pre-processing" Oct 29 22:51:15.168024 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:15.167828 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31493bbc-acc0-423b-9968-3be0bca796b5" containerName="payload-pre-processing" Oct 29 22:51:15.168024 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:15.167852 5057 cpu_manager.go:414] "containerMap: removing container" podUID="3b2af96a-99d4-46fc-9a2a-e773d96f5898" containerName="payload-pre-processing" Oct 29 22:51:15.168024 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:15.167857 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b2af96a-99d4-46fc-9a2a-e773d96f5898" containerName="payload-pre-processing" Oct 29 22:51:15.168024 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:15.167957 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b2af96a-99d4-46fc-9a2a-e773d96f5898" containerName="payload-pre-processing" Oct 29 22:51:15.168024 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:15.167967 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee18842c-3a3f-4148-807f-09838ef6ccf9" containerName="maas-api" Oct 29 22:51:15.168024 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:15.167974 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="185a7421-26cf-46ae-a65d-ed283704904e" containerName="payload-processing" Oct 29 22:51:15.168024 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:15.167981 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="89a6412b-1be7-4a77-bb4d-c8f81c7d6f43" containerName="payload-pre-processing" Oct 29 22:51:15.168024 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:15.167988 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e333bec-919e-49c4-b9c0-a732247ce509" containerName="payload-processing" Oct 29 22:51:15.168024 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:15.167994 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="3871fc17-a6ab-407c-944d-8f862b323e2d" containerName="maas-api" Oct 29 22:51:15.168024 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:15.168000 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="31493bbc-acc0-423b-9968-3be0bca796b5" containerName="payload-pre-processing" Oct 29 22:51:15.172835 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:15.172816 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-17394a-77794c47ff-djggj" Oct 29 22:51:15.190077 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:15.190053 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-17394a-77794c47ff-djggj"] Oct 29 22:51:15.201941 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:15.196300 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-17394a\"" Oct 29 22:51:15.201941 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:15.196470 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-17394a-dockercfg-8625j\"" Oct 29 22:51:15.215318 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:15.215286 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:51:15.258353 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:15.258265 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5917154d-50a3-4166-94e0-533aef08f790-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-17394a-77794c47ff-djggj\" (UID: \"5917154d-50a3-4166-94e0-533aef08f790\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-17394a-77794c47ff-djggj" Oct 29 22:51:15.258469 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:15.258397 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxnz8\" (UniqueName: \"kubernetes.io/projected/5917154d-50a3-4166-94e0-533aef08f790-kube-api-access-nxnz8\") pod \"payload-pre-processing-e2e-worker-w3-17394a-77794c47ff-djggj\" (UID: \"5917154d-50a3-4166-94e0-533aef08f790\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-17394a-77794c47ff-djggj" Oct 29 22:51:15.295788 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:15.295742 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-3c314108-67c4d8d458-hbl94" podUID="a1c2e436-c3c1-4cc8-a362-f77f162d6508" containerName="payload-processing" containerID="cri-o://efcf2e9b80c3efdf2f7ed4f6709d590371b415e8eacf1b9602cb35c60521a41e" gracePeriod=25 Oct 29 22:51:15.332644 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:15.332609 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-3c314108-6769db8fbc-75ccb" podUID="846589eb-a0a2-49c1-a9e0-e793b456633b" containerName="payload-pre-processing" containerID="cri-o://5503e3d7d3366655f6141c398f673deb7a5a2a2781a20d56f65628835965c172" gracePeriod=25 Oct 29 22:51:15.360164 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:15.360134 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxnz8\" (UniqueName: \"kubernetes.io/projected/5917154d-50a3-4166-94e0-533aef08f790-kube-api-access-nxnz8\") pod \"payload-pre-processing-e2e-worker-w3-17394a-77794c47ff-djggj\" (UID: \"5917154d-50a3-4166-94e0-533aef08f790\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-17394a-77794c47ff-djggj" Oct 29 22:51:15.360359 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:15.360342 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5917154d-50a3-4166-94e0-533aef08f790-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-17394a-77794c47ff-djggj\" (UID: \"5917154d-50a3-4166-94e0-533aef08f790\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-17394a-77794c47ff-djggj" Oct 29 22:51:15.361034 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:15.361015 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5917154d-50a3-4166-94e0-533aef08f790-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-17394a-77794c47ff-djggj\" (UID: \"5917154d-50a3-4166-94e0-533aef08f790\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-17394a-77794c47ff-djggj" Oct 29 22:51:15.384164 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:15.384136 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxnz8\" (UniqueName: \"kubernetes.io/projected/5917154d-50a3-4166-94e0-533aef08f790-kube-api-access-nxnz8\") pod \"payload-pre-processing-e2e-worker-w3-17394a-77794c47ff-djggj\" (UID: \"5917154d-50a3-4166-94e0-533aef08f790\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-17394a-77794c47ff-djggj" Oct 29 22:51:15.490529 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:15.490418 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-17394a-77794c47ff-djggj" Oct 29 22:51:15.591159 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:15.591122 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-3c314108-67c4d8d458-hbl94" Oct 29 22:51:15.669840 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:15.669812 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-3c314108-6769db8fbc-75ccb" Oct 29 22:51:15.769652 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:15.769619 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/846589eb-a0a2-49c1-a9e0-e793b456633b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/846589eb-a0a2-49c1-a9e0-e793b456633b-plugins-config-volume\") pod \"846589eb-a0a2-49c1-a9e0-e793b456633b\" (UID: \"846589eb-a0a2-49c1-a9e0-e793b456633b\") " Oct 29 22:51:15.769837 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:15.769656 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1c2e436-c3c1-4cc8-a362-f77f162d6508-kube-api-access-j6m5f\" (UniqueName: \"kubernetes.io/projected/a1c2e436-c3c1-4cc8-a362-f77f162d6508-kube-api-access-j6m5f\") pod \"a1c2e436-c3c1-4cc8-a362-f77f162d6508\" (UID: \"a1c2e436-c3c1-4cc8-a362-f77f162d6508\") " Oct 29 22:51:15.769837 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:15.769816 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1c2e436-c3c1-4cc8-a362-f77f162d6508-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1c2e436-c3c1-4cc8-a362-f77f162d6508-plugins-config-volume\") pod \"a1c2e436-c3c1-4cc8-a362-f77f162d6508\" (UID: \"a1c2e436-c3c1-4cc8-a362-f77f162d6508\") " Oct 29 22:51:15.769961 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:15.769854 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/846589eb-a0a2-49c1-a9e0-e793b456633b-kube-api-access-47qnj\" (UniqueName: \"kubernetes.io/projected/846589eb-a0a2-49c1-a9e0-e793b456633b-kube-api-access-47qnj\") pod \"846589eb-a0a2-49c1-a9e0-e793b456633b\" (UID: \"846589eb-a0a2-49c1-a9e0-e793b456633b\") " Oct 29 22:51:15.770099 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:15.770074 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/846589eb-a0a2-49c1-a9e0-e793b456633b-plugins-config-volume" pod "846589eb-a0a2-49c1-a9e0-e793b456633b" (UID: "846589eb-a0a2-49c1-a9e0-e793b456633b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:51:15.770186 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:15.770165 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1c2e436-c3c1-4cc8-a362-f77f162d6508-plugins-config-volume" pod "a1c2e436-c3c1-4cc8-a362-f77f162d6508" (UID: "a1c2e436-c3c1-4cc8-a362-f77f162d6508"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:51:15.770301 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:15.770281 5057 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/846589eb-a0a2-49c1-a9e0-e793b456633b-plugins-config-volume\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 22:51:15.770301 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:15.770299 5057 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1c2e436-c3c1-4cc8-a362-f77f162d6508-plugins-config-volume\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 22:51:15.771794 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:15.771773 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1c2e436-c3c1-4cc8-a362-f77f162d6508-kube-api-access-j6m5f" pod "a1c2e436-c3c1-4cc8-a362-f77f162d6508" (UID: "a1c2e436-c3c1-4cc8-a362-f77f162d6508"). InnerVolumeSpecName "kube-api-access-j6m5f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:51:15.771882 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:15.771860 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/846589eb-a0a2-49c1-a9e0-e793b456633b-kube-api-access-47qnj" pod "846589eb-a0a2-49c1-a9e0-e793b456633b" (UID: "846589eb-a0a2-49c1-a9e0-e793b456633b"). InnerVolumeSpecName "kube-api-access-47qnj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:51:15.832471 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:15.832448 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-17394a-77794c47ff-djggj"] Oct 29 22:51:15.834624 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:51:15.834596 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5917154d_50a3_4166_94e0_533aef08f790.slice/crio-b4881508a0a3662a5d410605c3395bd491fd630daf1c35ec2835f00c41e53fbc WatchSource:0}: Error finding container b4881508a0a3662a5d410605c3395bd491fd630daf1c35ec2835f00c41e53fbc: Status 404 returned error can't find the container with id b4881508a0a3662a5d410605c3395bd491fd630daf1c35ec2835f00c41e53fbc Oct 29 22:51:15.870799 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:15.870778 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j6m5f\" (UniqueName: \"kubernetes.io/projected/a1c2e436-c3c1-4cc8-a362-f77f162d6508-kube-api-access-j6m5f\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 22:51:15.870897 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:15.870803 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-47qnj\" (UniqueName: \"kubernetes.io/projected/846589eb-a0a2-49c1-a9e0-e793b456633b-kube-api-access-47qnj\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 22:51:16.101716 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:16.101616 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-17394a-77794c47ff-djggj" event={"ID":"5917154d-50a3-4166-94e0-533aef08f790","Type":"ContainerStarted","Data":"f127c13a5f2b483e7e49b6a7ffb9d50ce77abeb89c03b2cc1d5971c9ccce63a7"} Oct 29 22:51:16.101716 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:16.101681 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-17394a-77794c47ff-djggj" event={"ID":"5917154d-50a3-4166-94e0-533aef08f790","Type":"ContainerStarted","Data":"b4881508a0a3662a5d410605c3395bd491fd630daf1c35ec2835f00c41e53fbc"} Oct 29 22:51:16.101716 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:16.101714 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-17394a-77794c47ff-djggj" Oct 29 22:51:16.102960 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:16.102936 5057 generic.go:356] "Generic (PLEG): container finished" podID="846589eb-a0a2-49c1-a9e0-e793b456633b" containerID="5503e3d7d3366655f6141c398f673deb7a5a2a2781a20d56f65628835965c172" exitCode=0 Oct 29 22:51:16.103085 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:16.102975 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-3c314108-6769db8fbc-75ccb" Oct 29 22:51:16.103085 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:16.103017 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-3c314108-6769db8fbc-75ccb" event={"ID":"846589eb-a0a2-49c1-a9e0-e793b456633b","Type":"ContainerDied","Data":"5503e3d7d3366655f6141c398f673deb7a5a2a2781a20d56f65628835965c172"} Oct 29 22:51:16.103085 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:16.103065 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-3c314108-6769db8fbc-75ccb" event={"ID":"846589eb-a0a2-49c1-a9e0-e793b456633b","Type":"ContainerDied","Data":"61025276b39889732ee6201bb70ae5bae1bed12eddbbe5314cb3f497d8062f45"} Oct 29 22:51:16.103256 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:16.103088 5057 scope.go:122] "RemoveContainer" containerID="5503e3d7d3366655f6141c398f673deb7a5a2a2781a20d56f65628835965c172" Oct 29 22:51:16.104643 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:16.104620 5057 generic.go:356] "Generic (PLEG): container finished" podID="a1c2e436-c3c1-4cc8-a362-f77f162d6508" containerID="efcf2e9b80c3efdf2f7ed4f6709d590371b415e8eacf1b9602cb35c60521a41e" exitCode=0 Oct 29 22:51:16.104777 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:16.104653 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-3c314108-67c4d8d458-hbl94" Oct 29 22:51:16.104777 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:16.104650 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-3c314108-67c4d8d458-hbl94" event={"ID":"a1c2e436-c3c1-4cc8-a362-f77f162d6508","Type":"ContainerDied","Data":"efcf2e9b80c3efdf2f7ed4f6709d590371b415e8eacf1b9602cb35c60521a41e"} Oct 29 22:51:16.104882 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:16.104776 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-3c314108-67c4d8d458-hbl94" event={"ID":"a1c2e436-c3c1-4cc8-a362-f77f162d6508","Type":"ContainerDied","Data":"8aa879d946825e44093a16b22638a241375d7dc7e578b33b056fc36202513185"} Oct 29 22:51:16.114697 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:16.114677 5057 scope.go:122] "RemoveContainer" containerID="5503e3d7d3366655f6141c398f673deb7a5a2a2781a20d56f65628835965c172" Oct 29 22:51:16.115044 ip-10-0-140-74 kubenswrapper[5057]: E1029 22:51:16.115022 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5503e3d7d3366655f6141c398f673deb7a5a2a2781a20d56f65628835965c172\": container with ID starting with 5503e3d7d3366655f6141c398f673deb7a5a2a2781a20d56f65628835965c172 not found: ID does not exist" containerID="5503e3d7d3366655f6141c398f673deb7a5a2a2781a20d56f65628835965c172" Oct 29 22:51:16.115123 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:16.115050 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5503e3d7d3366655f6141c398f673deb7a5a2a2781a20d56f65628835965c172"} err="failed to get container status \"5503e3d7d3366655f6141c398f673deb7a5a2a2781a20d56f65628835965c172\": rpc error: code = NotFound desc = could not find container \"5503e3d7d3366655f6141c398f673deb7a5a2a2781a20d56f65628835965c172\": container with ID starting with 5503e3d7d3366655f6141c398f673deb7a5a2a2781a20d56f65628835965c172 not found: ID does not exist" Oct 29 22:51:16.115123 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:16.115068 5057 scope.go:122] "RemoveContainer" containerID="efcf2e9b80c3efdf2f7ed4f6709d590371b415e8eacf1b9602cb35c60521a41e" Oct 29 22:51:16.127846 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:16.127825 5057 scope.go:122] "RemoveContainer" containerID="efcf2e9b80c3efdf2f7ed4f6709d590371b415e8eacf1b9602cb35c60521a41e" Oct 29 22:51:16.128253 ip-10-0-140-74 kubenswrapper[5057]: E1029 22:51:16.128230 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"efcf2e9b80c3efdf2f7ed4f6709d590371b415e8eacf1b9602cb35c60521a41e\": container with ID starting with efcf2e9b80c3efdf2f7ed4f6709d590371b415e8eacf1b9602cb35c60521a41e not found: ID does not exist" containerID="efcf2e9b80c3efdf2f7ed4f6709d590371b415e8eacf1b9602cb35c60521a41e" Oct 29 22:51:16.128360 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:16.128258 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"efcf2e9b80c3efdf2f7ed4f6709d590371b415e8eacf1b9602cb35c60521a41e"} err="failed to get container status \"efcf2e9b80c3efdf2f7ed4f6709d590371b415e8eacf1b9602cb35c60521a41e\": rpc error: code = NotFound desc = could not find container \"efcf2e9b80c3efdf2f7ed4f6709d590371b415e8eacf1b9602cb35c60521a41e\": container with ID starting with efcf2e9b80c3efdf2f7ed4f6709d590371b415e8eacf1b9602cb35c60521a41e not found: ID does not exist" Oct 29 22:51:16.142574 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:16.140927 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-17394a-77794c47ff-djggj" podStartSLOduration=1.1409130570000001 podStartE2EDuration="1.140913057s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:51:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:51:16.139093963 +0000 UTC m=+1056.922307390" watchObservedRunningTime="2026-10-29 22:51:16.140913057 +0000 UTC m=+1056.924126690" Oct 29 22:51:16.174205 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:16.174179 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-3c314108-67c4d8d458-hbl94"] Oct 29 22:51:16.178935 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:16.178912 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-3c314108-67c4d8d458-hbl94"] Oct 29 22:51:16.214725 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:16.214699 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-3c314108-6769db8fbc-75ccb"] Oct 29 22:51:16.222237 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:16.222215 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-3c314108-6769db8fbc-75ccb"] Oct 29 22:51:16.574954 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:16.574918 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:51:17.181352 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:17.173614 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:51:17.297055 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:17.296997 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:51:17.329503 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:17.329466 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="846589eb-a0a2-49c1-a9e0-e793b456633b" path="/var/lib/kubelet/pods/846589eb-a0a2-49c1-a9e0-e793b456633b/volumes" Oct 29 22:51:17.330011 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:17.329995 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1c2e436-c3c1-4cc8-a362-f77f162d6508" path="/var/lib/kubelet/pods/a1c2e436-c3c1-4cc8-a362-f77f162d6508/volumes" Oct 29 22:51:17.412280 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:17.412238 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:51:17.526008 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:17.525971 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:51:18.091793 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:18.091755 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:51:18.298239 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:18.298200 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:51:18.783884 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:18.783848 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:51:18.888238 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:18.888202 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-063b46-5895cc45b-jqqdh"] Oct 29 22:51:18.889150 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:18.889128 5057 cpu_manager.go:414] "containerMap: removing container" podUID="a1c2e436-c3c1-4cc8-a362-f77f162d6508" containerName="payload-processing" Oct 29 22:51:18.889235 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:18.889154 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1c2e436-c3c1-4cc8-a362-f77f162d6508" containerName="payload-processing" Oct 29 22:51:18.889235 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:18.889176 5057 cpu_manager.go:414] "containerMap: removing container" podUID="846589eb-a0a2-49c1-a9e0-e793b456633b" containerName="payload-pre-processing" Oct 29 22:51:18.889235 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:18.889186 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="846589eb-a0a2-49c1-a9e0-e793b456633b" containerName="payload-pre-processing" Oct 29 22:51:18.889419 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:18.889405 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1c2e436-c3c1-4cc8-a362-f77f162d6508" containerName="payload-processing" Oct 29 22:51:18.889457 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:18.889427 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="846589eb-a0a2-49c1-a9e0-e793b456633b" containerName="payload-pre-processing" Oct 29 22:51:18.893894 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:18.893839 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:51:18.895245 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:18.895226 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-063b46-5895cc45b-jqqdh" Oct 29 22:51:18.906514 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:18.906487 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8508fdad-a139-4803-affb-b63a6d52bd13-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-063b46-5895cc45b-jqqdh\" (UID: \"8508fdad-a139-4803-affb-b63a6d52bd13\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-063b46-5895cc45b-jqqdh" Oct 29 22:51:18.906707 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:18.906683 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9zqv\" (UniqueName: \"kubernetes.io/projected/8508fdad-a139-4803-affb-b63a6d52bd13-kube-api-access-t9zqv\") pod \"maas-api-e2e-worker-w4-063b46-5895cc45b-jqqdh\" (UID: \"8508fdad-a139-4803-affb-b63a6d52bd13\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-063b46-5895cc45b-jqqdh" Oct 29 22:51:18.906792 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:18.906773 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8508fdad-a139-4803-affb-b63a6d52bd13-maas-api-tls\") pod \"maas-api-e2e-worker-w4-063b46-5895cc45b-jqqdh\" (UID: \"8508fdad-a139-4803-affb-b63a6d52bd13\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-063b46-5895cc45b-jqqdh" Oct 29 22:51:18.906877 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:18.906858 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8508fdad-a139-4803-affb-b63a6d52bd13-metrics-certs\") pod \"maas-api-e2e-worker-w4-063b46-5895cc45b-jqqdh\" (UID: \"8508fdad-a139-4803-affb-b63a6d52bd13\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-063b46-5895cc45b-jqqdh" Oct 29 22:51:18.934375 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:18.918515 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-063b46-serving-cert\"" Oct 29 22:51:18.934375 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:18.922817 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-063b46-5895cc45b-jqqdh"] Oct 29 22:51:19.006147 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:19.006110 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:51:19.007806 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:19.007779 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8508fdad-a139-4803-affb-b63a6d52bd13-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-063b46-5895cc45b-jqqdh\" (UID: \"8508fdad-a139-4803-affb-b63a6d52bd13\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-063b46-5895cc45b-jqqdh" Oct 29 22:51:19.007911 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:19.007896 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9zqv\" (UniqueName: \"kubernetes.io/projected/8508fdad-a139-4803-affb-b63a6d52bd13-kube-api-access-t9zqv\") pod \"maas-api-e2e-worker-w4-063b46-5895cc45b-jqqdh\" (UID: \"8508fdad-a139-4803-affb-b63a6d52bd13\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-063b46-5895cc45b-jqqdh" Oct 29 22:51:19.007970 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:19.007941 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8508fdad-a139-4803-affb-b63a6d52bd13-maas-api-tls\") pod \"maas-api-e2e-worker-w4-063b46-5895cc45b-jqqdh\" (UID: \"8508fdad-a139-4803-affb-b63a6d52bd13\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-063b46-5895cc45b-jqqdh" Oct 29 22:51:19.008021 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:19.007992 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8508fdad-a139-4803-affb-b63a6d52bd13-metrics-certs\") pod \"maas-api-e2e-worker-w4-063b46-5895cc45b-jqqdh\" (UID: \"8508fdad-a139-4803-affb-b63a6d52bd13\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-063b46-5895cc45b-jqqdh" Oct 29 22:51:19.008311 ip-10-0-140-74 kubenswrapper[5057]: E1029 22:51:19.008291 5057 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w4-063b46-serving-cert: secret "maas-api-e2e-worker-w4-063b46-serving-cert" not found Oct 29 22:51:19.008387 ip-10-0-140-74 kubenswrapper[5057]: E1029 22:51:19.008374 5057 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8508fdad-a139-4803-affb-b63a6d52bd13-maas-api-tls podName:8508fdad-a139-4803-affb-b63a6d52bd13 nodeName:}" failed. No retries permitted until 2026-10-29 22:51:19.508355242 +0000 UTC m=+1060.291568655 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8508fdad-a139-4803-affb-b63a6d52bd13-maas-api-tls") pod "maas-api-e2e-worker-w4-063b46-5895cc45b-jqqdh" (UID: "8508fdad-a139-4803-affb-b63a6d52bd13") : secret "maas-api-e2e-worker-w4-063b46-serving-cert" not found Oct 29 22:51:19.008453 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:19.008425 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8508fdad-a139-4803-affb-b63a6d52bd13-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-063b46-5895cc45b-jqqdh\" (UID: \"8508fdad-a139-4803-affb-b63a6d52bd13\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-063b46-5895cc45b-jqqdh" Oct 29 22:51:19.012354 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:19.012095 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8508fdad-a139-4803-affb-b63a6d52bd13-metrics-certs\") pod \"maas-api-e2e-worker-w4-063b46-5895cc45b-jqqdh\" (UID: \"8508fdad-a139-4803-affb-b63a6d52bd13\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-063b46-5895cc45b-jqqdh" Oct 29 22:51:19.032040 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:19.032011 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9zqv\" (UniqueName: \"kubernetes.io/projected/8508fdad-a139-4803-affb-b63a6d52bd13-kube-api-access-t9zqv\") pod \"maas-api-e2e-worker-w4-063b46-5895cc45b-jqqdh\" (UID: \"8508fdad-a139-4803-affb-b63a6d52bd13\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-063b46-5895cc45b-jqqdh" Oct 29 22:51:19.046445 ip-10-0-140-74 kubenswrapper[5057]: E1029 22:51:19.039454 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d2dnn" containerName="extractor" Oct 29 22:51:19.117018 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:19.116977 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:51:19.294803 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:19.294763 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:51:19.513439 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:19.513401 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8508fdad-a139-4803-affb-b63a6d52bd13-maas-api-tls\") pod \"maas-api-e2e-worker-w4-063b46-5895cc45b-jqqdh\" (UID: \"8508fdad-a139-4803-affb-b63a6d52bd13\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-063b46-5895cc45b-jqqdh" Oct 29 22:51:19.515765 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:19.515741 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8508fdad-a139-4803-affb-b63a6d52bd13-maas-api-tls\") pod \"maas-api-e2e-worker-w4-063b46-5895cc45b-jqqdh\" (UID: \"8508fdad-a139-4803-affb-b63a6d52bd13\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-063b46-5895cc45b-jqqdh" Oct 29 22:51:19.813521 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:19.813435 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-063b46-5895cc45b-jqqdh" Oct 29 22:51:20.117456 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:20.117427 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-063b46-5895cc45b-jqqdh"] Oct 29 22:51:20.120024 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:51:20.119991 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8508fdad_a139_4803_affb_b63a6d52bd13.slice/crio-b7431cd94d1d89632ab3ba4449c4e76976d635f8f1d92042d30a360e403ec39f WatchSource:0}: Error finding container b7431cd94d1d89632ab3ba4449c4e76976d635f8f1d92042d30a360e403ec39f: Status 404 returned error can't find the container with id b7431cd94d1d89632ab3ba4449c4e76976d635f8f1d92042d30a360e403ec39f Oct 29 22:51:20.145265 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:20.145237 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-063b46-5895cc45b-jqqdh" event={"ID":"8508fdad-a139-4803-affb-b63a6d52bd13","Type":"ContainerStarted","Data":"b7431cd94d1d89632ab3ba4449c4e76976d635f8f1d92042d30a360e403ec39f"} Oct 29 22:51:20.323549 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:20.322964 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-3c314108-7d6d7db8c7-s2s47"] Oct 29 22:51:20.323549 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:20.323191 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3c314108-7d6d7db8c7-s2s47" podUID="029233d7-bcbd-41e8-af12-f3276061c833" containerName="maas-api" containerID="cri-o://ee67ee37bab54f052b904641ec1aaa82b9b9a46fd30ef03958e588faa9efd7f7" gracePeriod=30 Oct 29 22:51:20.385321 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:20.385183 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:51:20.685927 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:20.685893 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3c314108-7d6d7db8c7-s2s47" Oct 29 22:51:20.726922 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:20.726887 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/029233d7-bcbd-41e8-af12-f3276061c833-metrics-certs\" (UniqueName: \"kubernetes.io/secret/029233d7-bcbd-41e8-af12-f3276061c833-metrics-certs\") pod \"029233d7-bcbd-41e8-af12-f3276061c833\" (UID: \"029233d7-bcbd-41e8-af12-f3276061c833\") " Oct 29 22:51:20.727029 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:20.726993 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/029233d7-bcbd-41e8-af12-f3276061c833-kube-api-access-vn8fq\" (UniqueName: \"kubernetes.io/projected/029233d7-bcbd-41e8-af12-f3276061c833-kube-api-access-vn8fq\") pod \"029233d7-bcbd-41e8-af12-f3276061c833\" (UID: \"029233d7-bcbd-41e8-af12-f3276061c833\") " Oct 29 22:51:20.727094 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:20.727082 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/029233d7-bcbd-41e8-af12-f3276061c833-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/029233d7-bcbd-41e8-af12-f3276061c833-openshift-service-ca\") pod \"029233d7-bcbd-41e8-af12-f3276061c833\" (UID: \"029233d7-bcbd-41e8-af12-f3276061c833\") " Oct 29 22:51:20.727164 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:20.727115 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/029233d7-bcbd-41e8-af12-f3276061c833-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/029233d7-bcbd-41e8-af12-f3276061c833-maas-api-tls\") pod \"029233d7-bcbd-41e8-af12-f3276061c833\" (UID: \"029233d7-bcbd-41e8-af12-f3276061c833\") " Oct 29 22:51:20.728404 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:20.728120 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/029233d7-bcbd-41e8-af12-f3276061c833-openshift-service-ca" pod "029233d7-bcbd-41e8-af12-f3276061c833" (UID: "029233d7-bcbd-41e8-af12-f3276061c833"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:51:20.729538 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:20.729474 5057 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/029233d7-bcbd-41e8-af12-f3276061c833-openshift-service-ca\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 22:51:20.730638 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:20.730613 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/029233d7-bcbd-41e8-af12-f3276061c833-maas-api-tls" pod "029233d7-bcbd-41e8-af12-f3276061c833" (UID: "029233d7-bcbd-41e8-af12-f3276061c833"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:51:20.730726 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:20.730694 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/029233d7-bcbd-41e8-af12-f3276061c833-metrics-certs" pod "029233d7-bcbd-41e8-af12-f3276061c833" (UID: "029233d7-bcbd-41e8-af12-f3276061c833"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:51:20.732568 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:20.732544 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/029233d7-bcbd-41e8-af12-f3276061c833-kube-api-access-vn8fq" pod "029233d7-bcbd-41e8-af12-f3276061c833" (UID: "029233d7-bcbd-41e8-af12-f3276061c833"). InnerVolumeSpecName "kube-api-access-vn8fq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:51:20.831006 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:20.830977 5057 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/029233d7-bcbd-41e8-af12-f3276061c833-metrics-certs\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 22:51:20.831006 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:20.831003 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vn8fq\" (UniqueName: \"kubernetes.io/projected/029233d7-bcbd-41e8-af12-f3276061c833-kube-api-access-vn8fq\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 22:51:20.831006 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:20.831013 5057 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/029233d7-bcbd-41e8-af12-f3276061c833-maas-api-tls\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 22:51:20.885734 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:20.885702 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:51:21.006453 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:21.006417 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:51:21.119120 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:21.119083 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:51:21.155919 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:21.155700 5057 generic.go:356] "Generic (PLEG): container finished" podID="029233d7-bcbd-41e8-af12-f3276061c833" containerID="ee67ee37bab54f052b904641ec1aaa82b9b9a46fd30ef03958e588faa9efd7f7" exitCode=0 Oct 29 22:51:21.155919 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:21.155760 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3c314108-7d6d7db8c7-s2s47" Oct 29 22:51:21.155919 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:21.155801 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3c314108-7d6d7db8c7-s2s47" event={"ID":"029233d7-bcbd-41e8-af12-f3276061c833","Type":"ContainerDied","Data":"ee67ee37bab54f052b904641ec1aaa82b9b9a46fd30ef03958e588faa9efd7f7"} Oct 29 22:51:21.155919 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:21.155848 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3c314108-7d6d7db8c7-s2s47" event={"ID":"029233d7-bcbd-41e8-af12-f3276061c833","Type":"ContainerDied","Data":"a54fd8a947ae5c5683351fe415e12d0b735e94be7add2910ad2d58d3a002bd33"} Oct 29 22:51:21.155919 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:21.155877 5057 scope.go:122] "RemoveContainer" containerID="ee67ee37bab54f052b904641ec1aaa82b9b9a46fd30ef03958e588faa9efd7f7" Oct 29 22:51:21.163437 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:21.163407 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-063b46-5895cc45b-jqqdh" event={"ID":"8508fdad-a139-4803-affb-b63a6d52bd13","Type":"ContainerStarted","Data":"22aa141fdbcfb4ff64996355576208767fc6947554c48afe4e4134b76164a9c2"} Oct 29 22:51:21.163690 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:21.163649 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-063b46-5895cc45b-jqqdh" Oct 29 22:51:21.174158 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:21.174139 5057 scope.go:122] "RemoveContainer" containerID="ee67ee37bab54f052b904641ec1aaa82b9b9a46fd30ef03958e588faa9efd7f7" Oct 29 22:51:21.174463 ip-10-0-140-74 kubenswrapper[5057]: E1029 22:51:21.174440 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee67ee37bab54f052b904641ec1aaa82b9b9a46fd30ef03958e588faa9efd7f7\": container with ID starting with ee67ee37bab54f052b904641ec1aaa82b9b9a46fd30ef03958e588faa9efd7f7 not found: ID does not exist" containerID="ee67ee37bab54f052b904641ec1aaa82b9b9a46fd30ef03958e588faa9efd7f7" Oct 29 22:51:21.174546 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:21.174473 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee67ee37bab54f052b904641ec1aaa82b9b9a46fd30ef03958e588faa9efd7f7"} err="failed to get container status \"ee67ee37bab54f052b904641ec1aaa82b9b9a46fd30ef03958e588faa9efd7f7\": rpc error: code = NotFound desc = could not find container \"ee67ee37bab54f052b904641ec1aaa82b9b9a46fd30ef03958e588faa9efd7f7\": container with ID starting with ee67ee37bab54f052b904641ec1aaa82b9b9a46fd30ef03958e588faa9efd7f7 not found: ID does not exist" Oct 29 22:51:21.210282 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:21.210241 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-063b46-5895cc45b-jqqdh" podStartSLOduration=2.7565057570000002 podStartE2EDuration="3.210227484s" totalImagesPullingTime="453.721727ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:51:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:51:21.207263079 +0000 UTC m=+1061.990476505" watchObservedRunningTime="2026-10-29 22:51:21.210227484 +0000 UTC m=+1061.993440910" Oct 29 22:51:21.236562 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:21.236536 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:51:21.244162 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:21.244143 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-3c314108-7d6d7db8c7-s2s47"] Oct 29 22:51:21.251268 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:21.251250 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-3c314108-7d6d7db8c7-s2s47"] Oct 29 22:51:21.330190 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:21.330120 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="029233d7-bcbd-41e8-af12-f3276061c833" path="/var/lib/kubelet/pods/029233d7-bcbd-41e8-af12-f3276061c833/volumes" Oct 29 22:51:21.988882 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:21.988845 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v2tfm"] Oct 29 22:51:22.016653 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:22.016613 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:51:22.625491 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:22.625459 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:51:22.700434 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:22.700403 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-17394a-f64cd8977-xhjpk"] Oct 29 22:51:22.701054 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:22.701040 5057 cpu_manager.go:414] "containerMap: removing container" podUID="029233d7-bcbd-41e8-af12-f3276061c833" containerName="maas-api" Oct 29 22:51:22.701099 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:22.701057 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="029233d7-bcbd-41e8-af12-f3276061c833" containerName="maas-api" Oct 29 22:51:22.701193 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:22.701184 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="029233d7-bcbd-41e8-af12-f3276061c833" containerName="maas-api" Oct 29 22:51:22.706265 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:22.706246 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-17394a-f64cd8977-xhjpk" Oct 29 22:51:22.730458 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:22.730430 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w3-17394a-serving-cert\"" Oct 29 22:51:22.733785 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:22.733756 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-17394a-f64cd8977-xhjpk"] Oct 29 22:51:22.750779 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:22.750755 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/21c72ab9-0742-4f8a-aee1-8f820d967c94-metrics-certs\") pod \"maas-api-e2e-worker-w3-17394a-f64cd8977-xhjpk\" (UID: \"21c72ab9-0742-4f8a-aee1-8f820d967c94\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-17394a-f64cd8977-xhjpk" Oct 29 22:51:22.750878 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:22.750805 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/21c72ab9-0742-4f8a-aee1-8f820d967c94-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-17394a-f64cd8977-xhjpk\" (UID: \"21c72ab9-0742-4f8a-aee1-8f820d967c94\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-17394a-f64cd8977-xhjpk" Oct 29 22:51:22.750929 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:22.750885 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57z5j\" (UniqueName: \"kubernetes.io/projected/21c72ab9-0742-4f8a-aee1-8f820d967c94-kube-api-access-57z5j\") pod \"maas-api-e2e-worker-w3-17394a-f64cd8977-xhjpk\" (UID: \"21c72ab9-0742-4f8a-aee1-8f820d967c94\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-17394a-f64cd8977-xhjpk" Oct 29 22:51:22.750969 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:22.750949 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21c72ab9-0742-4f8a-aee1-8f820d967c94-maas-api-tls\") pod \"maas-api-e2e-worker-w3-17394a-f64cd8977-xhjpk\" (UID: \"21c72ab9-0742-4f8a-aee1-8f820d967c94\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-17394a-f64cd8977-xhjpk" Oct 29 22:51:22.835535 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:22.835489 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:51:22.851623 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:22.851595 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/21c72ab9-0742-4f8a-aee1-8f820d967c94-metrics-certs\") pod \"maas-api-e2e-worker-w3-17394a-f64cd8977-xhjpk\" (UID: \"21c72ab9-0742-4f8a-aee1-8f820d967c94\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-17394a-f64cd8977-xhjpk" Oct 29 22:51:22.851769 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:22.851653 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/21c72ab9-0742-4f8a-aee1-8f820d967c94-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-17394a-f64cd8977-xhjpk\" (UID: \"21c72ab9-0742-4f8a-aee1-8f820d967c94\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-17394a-f64cd8977-xhjpk" Oct 29 22:51:22.851769 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:22.851697 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57z5j\" (UniqueName: \"kubernetes.io/projected/21c72ab9-0742-4f8a-aee1-8f820d967c94-kube-api-access-57z5j\") pod \"maas-api-e2e-worker-w3-17394a-f64cd8977-xhjpk\" (UID: \"21c72ab9-0742-4f8a-aee1-8f820d967c94\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-17394a-f64cd8977-xhjpk" Oct 29 22:51:22.851888 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:22.851772 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21c72ab9-0742-4f8a-aee1-8f820d967c94-maas-api-tls\") pod \"maas-api-e2e-worker-w3-17394a-f64cd8977-xhjpk\" (UID: \"21c72ab9-0742-4f8a-aee1-8f820d967c94\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-17394a-f64cd8977-xhjpk" Oct 29 22:51:22.852022 ip-10-0-140-74 kubenswrapper[5057]: E1029 22:51:22.852000 5057 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w3-17394a-serving-cert: secret "maas-api-e2e-worker-w3-17394a-serving-cert" not found Oct 29 22:51:22.852122 ip-10-0-140-74 kubenswrapper[5057]: E1029 22:51:22.852111 5057 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21c72ab9-0742-4f8a-aee1-8f820d967c94-maas-api-tls podName:21c72ab9-0742-4f8a-aee1-8f820d967c94 nodeName:}" failed. No retries permitted until 2026-10-29 22:51:23.352091599 +0000 UTC m=+1064.135305008 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/21c72ab9-0742-4f8a-aee1-8f820d967c94-maas-api-tls") pod "maas-api-e2e-worker-w3-17394a-f64cd8977-xhjpk" (UID: "21c72ab9-0742-4f8a-aee1-8f820d967c94") : secret "maas-api-e2e-worker-w3-17394a-serving-cert" not found Oct 29 22:51:22.852408 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:22.852390 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/21c72ab9-0742-4f8a-aee1-8f820d967c94-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-17394a-f64cd8977-xhjpk\" (UID: \"21c72ab9-0742-4f8a-aee1-8f820d967c94\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-17394a-f64cd8977-xhjpk" Oct 29 22:51:22.854147 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:22.854131 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/21c72ab9-0742-4f8a-aee1-8f820d967c94-metrics-certs\") pod \"maas-api-e2e-worker-w3-17394a-f64cd8977-xhjpk\" (UID: \"21c72ab9-0742-4f8a-aee1-8f820d967c94\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-17394a-f64cd8977-xhjpk" Oct 29 22:51:22.877088 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:22.877024 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-57z5j\" (UniqueName: \"kubernetes.io/projected/21c72ab9-0742-4f8a-aee1-8f820d967c94-kube-api-access-57z5j\") pod \"maas-api-e2e-worker-w3-17394a-f64cd8977-xhjpk\" (UID: \"21c72ab9-0742-4f8a-aee1-8f820d967c94\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-17394a-f64cd8977-xhjpk" Oct 29 22:51:22.968242 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:22.968206 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:51:23.082045 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:23.082016 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:51:23.202010 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:23.201975 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:51:23.358409 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:23.358375 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21c72ab9-0742-4f8a-aee1-8f820d967c94-maas-api-tls\") pod \"maas-api-e2e-worker-w3-17394a-f64cd8977-xhjpk\" (UID: \"21c72ab9-0742-4f8a-aee1-8f820d967c94\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-17394a-f64cd8977-xhjpk" Oct 29 22:51:23.360815 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:23.360786 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21c72ab9-0742-4f8a-aee1-8f820d967c94-maas-api-tls\") pod \"maas-api-e2e-worker-w3-17394a-f64cd8977-xhjpk\" (UID: \"21c72ab9-0742-4f8a-aee1-8f820d967c94\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-17394a-f64cd8977-xhjpk" Oct 29 22:51:23.620294 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:23.620214 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-17394a-f64cd8977-xhjpk" Oct 29 22:51:23.927548 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:23.927526 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-17394a-f64cd8977-xhjpk"] Oct 29 22:51:23.929728 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:51:23.929701 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21c72ab9_0742_4f8a_aee1_8f820d967c94.slice/crio-ae37a6d5fc513a559ad0a4cde9b6d50e0b1bb1bc09ceb1bb6b03c76edeae082f WatchSource:0}: Error finding container ae37a6d5fc513a559ad0a4cde9b6d50e0b1bb1bc09ceb1bb6b03c76edeae082f: Status 404 returned error can't find the container with id ae37a6d5fc513a559ad0a4cde9b6d50e0b1bb1bc09ceb1bb6b03c76edeae082f Oct 29 22:51:24.188541 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:24.188458 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-17394a-f64cd8977-xhjpk" event={"ID":"21c72ab9-0742-4f8a-aee1-8f820d967c94","Type":"ContainerStarted","Data":"ae37a6d5fc513a559ad0a4cde9b6d50e0b1bb1bc09ceb1bb6b03c76edeae082f"} Oct 29 22:51:24.726906 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:24.726872 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:51:24.883484 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:24.883450 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:51:24.932450 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:24.932408 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-575d59-65c58457dd-c4d2z"] Oct 29 22:51:24.937070 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:24.937045 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-575d59-65c58457dd-c4d2z" Oct 29 22:51:24.970467 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:24.970433 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-575d59-65c58457dd-c4d2z"] Oct 29 22:51:24.985621 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:24.985545 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-575d59\"" Oct 29 22:51:24.989502 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:24.986761 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-575d59-dockercfg-vqbj5\"" Oct 29 22:51:24.997079 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:24.997044 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7k8g\" (UniqueName: \"kubernetes.io/projected/3bb1016d-e1c5-4e72-ab08-a063c94e3079-kube-api-access-w7k8g\") pod \"payload-processing-e2e-worker-w2-575d59-65c58457dd-c4d2z\" (UID: \"3bb1016d-e1c5-4e72-ab08-a063c94e3079\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-575d59-65c58457dd-c4d2z" Oct 29 22:51:25.000536 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:25.000504 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3bb1016d-e1c5-4e72-ab08-a063c94e3079-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-575d59-65c58457dd-c4d2z\" (UID: \"3bb1016d-e1c5-4e72-ab08-a063c94e3079\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-575d59-65c58457dd-c4d2z" Oct 29 22:51:25.102070 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:25.102040 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3bb1016d-e1c5-4e72-ab08-a063c94e3079-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-575d59-65c58457dd-c4d2z\" (UID: \"3bb1016d-e1c5-4e72-ab08-a063c94e3079\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-575d59-65c58457dd-c4d2z" Oct 29 22:51:25.102256 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:25.102138 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7k8g\" (UniqueName: \"kubernetes.io/projected/3bb1016d-e1c5-4e72-ab08-a063c94e3079-kube-api-access-w7k8g\") pod \"payload-processing-e2e-worker-w2-575d59-65c58457dd-c4d2z\" (UID: \"3bb1016d-e1c5-4e72-ab08-a063c94e3079\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-575d59-65c58457dd-c4d2z" Oct 29 22:51:25.102835 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:25.102808 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3bb1016d-e1c5-4e72-ab08-a063c94e3079-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-575d59-65c58457dd-c4d2z\" (UID: \"3bb1016d-e1c5-4e72-ab08-a063c94e3079\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-575d59-65c58457dd-c4d2z" Oct 29 22:51:25.123580 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:25.123553 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7k8g\" (UniqueName: \"kubernetes.io/projected/3bb1016d-e1c5-4e72-ab08-a063c94e3079-kube-api-access-w7k8g\") pod \"payload-processing-e2e-worker-w2-575d59-65c58457dd-c4d2z\" (UID: \"3bb1016d-e1c5-4e72-ab08-a063c94e3079\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-575d59-65c58457dd-c4d2z" Oct 29 22:51:25.196130 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:25.196099 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-17394a-f64cd8977-xhjpk" event={"ID":"21c72ab9-0742-4f8a-aee1-8f820d967c94","Type":"ContainerStarted","Data":"2c0666f47fdff3cddd2313a8765c7664b08dec0ba9817811ef1d341c9571d9c1"} Oct 29 22:51:25.196479 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:25.196176 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-17394a-f64cd8977-xhjpk" Oct 29 22:51:25.237838 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:25.237742 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-17394a-f64cd8977-xhjpk" podStartSLOduration=2.87174359 podStartE2EDuration="3.237727911s" totalImagesPullingTime="365.984321ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:51:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:51:25.236305813 +0000 UTC m=+1066.019519239" watchObservedRunningTime="2026-10-29 22:51:25.237727911 +0000 UTC m=+1066.020941338" Oct 29 22:51:25.274768 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:25.274735 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-575d59-65c58457dd-c4d2z" Oct 29 22:51:25.579050 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:25.579012 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-575d59-65c58457dd-c4d2z"] Oct 29 22:51:25.582632 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:51:25.582601 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bb1016d_e1c5_4e72_ab08_a063c94e3079.slice/crio-00645d9564df01312c0f0618aa9ead63ee22f4a6507f2a9141023f7b531f99e9 WatchSource:0}: Error finding container 00645d9564df01312c0f0618aa9ead63ee22f4a6507f2a9141023f7b531f99e9: Status 404 returned error can't find the container with id 00645d9564df01312c0f0618aa9ead63ee22f4a6507f2a9141023f7b531f99e9 Oct 29 22:51:26.204670 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:26.204633 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-575d59-65c58457dd-c4d2z" event={"ID":"3bb1016d-e1c5-4e72-ab08-a063c94e3079","Type":"ContainerStarted","Data":"859f4b40a2735141c396ace5bb8a76e1e729fbc5ccac7db116b1d72c0087416b"} Oct 29 22:51:26.205149 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:26.204726 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-575d59-65c58457dd-c4d2z" Oct 29 22:51:26.205149 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:26.204746 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-575d59-65c58457dd-c4d2z" event={"ID":"3bb1016d-e1c5-4e72-ab08-a063c94e3079","Type":"ContainerStarted","Data":"00645d9564df01312c0f0618aa9ead63ee22f4a6507f2a9141023f7b531f99e9"} Oct 29 22:51:26.244677 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:26.244625 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w2-575d59-65c58457dd-c4d2z" podStartSLOduration=2.244612729 podStartE2EDuration="2.244612729s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:51:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:51:26.243387358 +0000 UTC m=+1067.026600787" watchObservedRunningTime="2026-10-29 22:51:26.244612729 +0000 UTC m=+1067.027826154" Oct 29 22:51:26.366607 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:26.366574 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:51:26.480099 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:26.480007 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:51:26.659116 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:26.659082 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:51:27.127447 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:27.127417 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-17394a-77794c47ff-djggj" Oct 29 22:51:27.179397 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:27.179368 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-063b46-5895cc45b-jqqdh" Oct 29 22:51:28.095984 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:28.095935 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:51:28.220197 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:28.220160 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:51:28.543327 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:28.543292 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:51:30.226605 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:30.226572 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:51:30.921033 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:30.920998 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:51:31.211022 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:31.210995 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-17394a-f64cd8977-xhjpk" Oct 29 22:51:31.785233 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:31.785197 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:51:33.516546 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:33.516515 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-575d59-55c4d89464-snt8r"] Oct 29 22:51:33.521446 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:33.521366 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-575d59-55c4d89464-snt8r" Oct 29 22:51:33.534814 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:33.534790 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w2-575d59-serving-cert\"" Oct 29 22:51:33.550372 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:33.550344 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-575d59-55c4d89464-snt8r"] Oct 29 22:51:33.594208 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:33.594169 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/90a14eb2-ffff-495b-a714-a6a950d37b7c-metrics-certs\") pod \"maas-api-e2e-worker-w2-575d59-55c4d89464-snt8r\" (UID: \"90a14eb2-ffff-495b-a714-a6a950d37b7c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-575d59-55c4d89464-snt8r" Oct 29 22:51:33.594434 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:33.594247 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/90a14eb2-ffff-495b-a714-a6a950d37b7c-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-575d59-55c4d89464-snt8r\" (UID: \"90a14eb2-ffff-495b-a714-a6a950d37b7c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-575d59-55c4d89464-snt8r" Oct 29 22:51:33.594434 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:33.594300 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/90a14eb2-ffff-495b-a714-a6a950d37b7c-maas-api-tls\") pod \"maas-api-e2e-worker-w2-575d59-55c4d89464-snt8r\" (UID: \"90a14eb2-ffff-495b-a714-a6a950d37b7c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-575d59-55c4d89464-snt8r" Oct 29 22:51:33.594434 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:33.594348 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5z9vp\" (UniqueName: \"kubernetes.io/projected/90a14eb2-ffff-495b-a714-a6a950d37b7c-kube-api-access-5z9vp\") pod \"maas-api-e2e-worker-w2-575d59-55c4d89464-snt8r\" (UID: \"90a14eb2-ffff-495b-a714-a6a950d37b7c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-575d59-55c4d89464-snt8r" Oct 29 22:51:33.631435 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:33.631383 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:51:33.695379 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:33.695313 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/90a14eb2-ffff-495b-a714-a6a950d37b7c-maas-api-tls\") pod \"maas-api-e2e-worker-w2-575d59-55c4d89464-snt8r\" (UID: \"90a14eb2-ffff-495b-a714-a6a950d37b7c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-575d59-55c4d89464-snt8r" Oct 29 22:51:33.695596 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:33.695401 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5z9vp\" (UniqueName: \"kubernetes.io/projected/90a14eb2-ffff-495b-a714-a6a950d37b7c-kube-api-access-5z9vp\") pod \"maas-api-e2e-worker-w2-575d59-55c4d89464-snt8r\" (UID: \"90a14eb2-ffff-495b-a714-a6a950d37b7c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-575d59-55c4d89464-snt8r" Oct 29 22:51:33.695727 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:33.695629 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/90a14eb2-ffff-495b-a714-a6a950d37b7c-metrics-certs\") pod \"maas-api-e2e-worker-w2-575d59-55c4d89464-snt8r\" (UID: \"90a14eb2-ffff-495b-a714-a6a950d37b7c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-575d59-55c4d89464-snt8r" Oct 29 22:51:33.695727 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:33.695691 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/90a14eb2-ffff-495b-a714-a6a950d37b7c-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-575d59-55c4d89464-snt8r\" (UID: \"90a14eb2-ffff-495b-a714-a6a950d37b7c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-575d59-55c4d89464-snt8r" Oct 29 22:51:33.696528 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:33.696495 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/90a14eb2-ffff-495b-a714-a6a950d37b7c-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-575d59-55c4d89464-snt8r\" (UID: \"90a14eb2-ffff-495b-a714-a6a950d37b7c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-575d59-55c4d89464-snt8r" Oct 29 22:51:33.698036 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:33.698013 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/90a14eb2-ffff-495b-a714-a6a950d37b7c-metrics-certs\") pod \"maas-api-e2e-worker-w2-575d59-55c4d89464-snt8r\" (UID: \"90a14eb2-ffff-495b-a714-a6a950d37b7c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-575d59-55c4d89464-snt8r" Oct 29 22:51:33.698126 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:33.698052 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/90a14eb2-ffff-495b-a714-a6a950d37b7c-maas-api-tls\") pod \"maas-api-e2e-worker-w2-575d59-55c4d89464-snt8r\" (UID: \"90a14eb2-ffff-495b-a714-a6a950d37b7c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-575d59-55c4d89464-snt8r" Oct 29 22:51:33.720207 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:33.720179 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5z9vp\" (UniqueName: \"kubernetes.io/projected/90a14eb2-ffff-495b-a714-a6a950d37b7c-kube-api-access-5z9vp\") pod \"maas-api-e2e-worker-w2-575d59-55c4d89464-snt8r\" (UID: \"90a14eb2-ffff-495b-a714-a6a950d37b7c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-575d59-55c4d89464-snt8r" Oct 29 22:51:33.736779 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:33.736754 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v2tfm"] Oct 29 22:51:33.753575 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:33.753551 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:51:33.836684 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:33.836598 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-575d59-55c4d89464-snt8r" Oct 29 22:51:34.145627 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:34.145604 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-575d59-55c4d89464-snt8r"] Oct 29 22:51:34.148006 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:51:34.147980 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90a14eb2_ffff_495b_a714_a6a950d37b7c.slice/crio-cdf5d3e275de4435ecce4b717d26e612a50b7bb761ea1317ac822e0a78f507f0 WatchSource:0}: Error finding container cdf5d3e275de4435ecce4b717d26e612a50b7bb761ea1317ac822e0a78f507f0: Status 404 returned error can't find the container with id cdf5d3e275de4435ecce4b717d26e612a50b7bb761ea1317ac822e0a78f507f0 Oct 29 22:51:34.260050 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:34.260012 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-575d59-55c4d89464-snt8r" event={"ID":"90a14eb2-ffff-495b-a714-a6a950d37b7c","Type":"ContainerStarted","Data":"cdf5d3e275de4435ecce4b717d26e612a50b7bb761ea1317ac822e0a78f507f0"} Oct 29 22:51:35.268944 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:35.268903 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-575d59-55c4d89464-snt8r" event={"ID":"90a14eb2-ffff-495b-a714-a6a950d37b7c","Type":"ContainerStarted","Data":"1a035f869846e51d38ca36ad761885487601310ea5871d58f4c4da835f1112aa"} Oct 29 22:51:35.269321 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:35.268988 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-575d59-55c4d89464-snt8r" Oct 29 22:51:35.275423 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:35.275384 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:51:35.310275 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:35.310223 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-575d59-55c4d89464-snt8r" podStartSLOduration=2.006362736 podStartE2EDuration="2.310211692s" totalImagesPullingTime="303.848956ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:51:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:51:35.306676742 +0000 UTC m=+1076.089890167" watchObservedRunningTime="2026-10-29 22:51:35.310211692 +0000 UTC m=+1076.093425114" Oct 29 22:51:35.696775 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:35.696692 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:51:35.883632 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:35.883599 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:51:35.995788 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:35.995755 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:51:36.119156 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:36.119121 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:51:37.217929 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:37.217900 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-575d59-65c58457dd-c4d2z" Oct 29 22:51:37.500100 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:37.500009 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:51:38.451486 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:38.451457 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:51:39.079869 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:39.079829 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:51:39.104162 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:39.104133 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-b2921b-6599749ddd-kbw8v"] Oct 29 22:51:39.108715 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:39.108693 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-b2921b-6599749ddd-kbw8v" Oct 29 22:51:39.128030 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:39.127257 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w5-b2921b-serving-cert\"" Oct 29 22:51:39.140241 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:39.136855 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-b2921b-6599749ddd-kbw8v"] Oct 29 22:51:39.207801 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:39.207766 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:51:39.261444 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:39.261413 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g98km\" (UniqueName: \"kubernetes.io/projected/02be284e-3e49-4fa7-8475-a5a79f8997b1-kube-api-access-g98km\") pod \"maas-api-e2e-worker-w5-b2921b-6599749ddd-kbw8v\" (UID: \"02be284e-3e49-4fa7-8475-a5a79f8997b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-b2921b-6599749ddd-kbw8v" Oct 29 22:51:39.261620 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:39.261464 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/02be284e-3e49-4fa7-8475-a5a79f8997b1-metrics-certs\") pod \"maas-api-e2e-worker-w5-b2921b-6599749ddd-kbw8v\" (UID: \"02be284e-3e49-4fa7-8475-a5a79f8997b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-b2921b-6599749ddd-kbw8v" Oct 29 22:51:39.261675 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:39.261640 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/02be284e-3e49-4fa7-8475-a5a79f8997b1-maas-api-tls\") pod \"maas-api-e2e-worker-w5-b2921b-6599749ddd-kbw8v\" (UID: \"02be284e-3e49-4fa7-8475-a5a79f8997b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-b2921b-6599749ddd-kbw8v" Oct 29 22:51:39.261718 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:39.261700 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/02be284e-3e49-4fa7-8475-a5a79f8997b1-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-b2921b-6599749ddd-kbw8v\" (UID: \"02be284e-3e49-4fa7-8475-a5a79f8997b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-b2921b-6599749ddd-kbw8v" Oct 29 22:51:39.323472 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:39.323438 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:51:39.362527 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:39.362439 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g98km\" (UniqueName: \"kubernetes.io/projected/02be284e-3e49-4fa7-8475-a5a79f8997b1-kube-api-access-g98km\") pod \"maas-api-e2e-worker-w5-b2921b-6599749ddd-kbw8v\" (UID: \"02be284e-3e49-4fa7-8475-a5a79f8997b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-b2921b-6599749ddd-kbw8v" Oct 29 22:51:39.362527 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:39.362494 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/02be284e-3e49-4fa7-8475-a5a79f8997b1-metrics-certs\") pod \"maas-api-e2e-worker-w5-b2921b-6599749ddd-kbw8v\" (UID: \"02be284e-3e49-4fa7-8475-a5a79f8997b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-b2921b-6599749ddd-kbw8v" Oct 29 22:51:39.362737 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:39.362595 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/02be284e-3e49-4fa7-8475-a5a79f8997b1-maas-api-tls\") pod \"maas-api-e2e-worker-w5-b2921b-6599749ddd-kbw8v\" (UID: \"02be284e-3e49-4fa7-8475-a5a79f8997b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-b2921b-6599749ddd-kbw8v" Oct 29 22:51:39.362737 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:39.362667 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/02be284e-3e49-4fa7-8475-a5a79f8997b1-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-b2921b-6599749ddd-kbw8v\" (UID: \"02be284e-3e49-4fa7-8475-a5a79f8997b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-b2921b-6599749ddd-kbw8v" Oct 29 22:51:39.363421 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:39.363396 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/02be284e-3e49-4fa7-8475-a5a79f8997b1-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-b2921b-6599749ddd-kbw8v\" (UID: \"02be284e-3e49-4fa7-8475-a5a79f8997b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-b2921b-6599749ddd-kbw8v" Oct 29 22:51:39.364981 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:39.364960 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/02be284e-3e49-4fa7-8475-a5a79f8997b1-metrics-certs\") pod \"maas-api-e2e-worker-w5-b2921b-6599749ddd-kbw8v\" (UID: \"02be284e-3e49-4fa7-8475-a5a79f8997b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-b2921b-6599749ddd-kbw8v" Oct 29 22:51:39.374594 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:39.374577 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w5-b2921b-serving-cert\"" Oct 29 22:51:39.384161 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:39.384142 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g98km\" (UniqueName: \"kubernetes.io/projected/02be284e-3e49-4fa7-8475-a5a79f8997b1-kube-api-access-g98km\") pod \"maas-api-e2e-worker-w5-b2921b-6599749ddd-kbw8v\" (UID: \"02be284e-3e49-4fa7-8475-a5a79f8997b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-b2921b-6599749ddd-kbw8v" Oct 29 22:51:39.386017 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:39.385999 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/02be284e-3e49-4fa7-8475-a5a79f8997b1-maas-api-tls\") pod \"maas-api-e2e-worker-w5-b2921b-6599749ddd-kbw8v\" (UID: \"02be284e-3e49-4fa7-8475-a5a79f8997b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-b2921b-6599749ddd-kbw8v" Oct 29 22:51:39.423875 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:39.423854 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-b2921b-6599749ddd-kbw8v" Oct 29 22:51:39.737076 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:39.736483 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-b2921b-6599749ddd-kbw8v"] Oct 29 22:51:39.739082 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:51:39.739051 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02be284e_3e49_4fa7_8475_a5a79f8997b1.slice/crio-6246f3109c9c5fa654478bf6806e90fc848528247cd96cec81ce573e05cca639 WatchSource:0}: Error finding container 6246f3109c9c5fa654478bf6806e90fc848528247cd96cec81ce573e05cca639: Status 404 returned error can't find the container with id 6246f3109c9c5fa654478bf6806e90fc848528247cd96cec81ce573e05cca639 Oct 29 22:51:40.311322 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:40.311237 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-b2921b-6599749ddd-kbw8v" event={"ID":"02be284e-3e49-4fa7-8475-a5a79f8997b1","Type":"ContainerStarted","Data":"6277901ee055d20006326dce028a7d0607d9325974928333951b57d526290e85"} Oct 29 22:51:40.311322 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:40.311314 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-b2921b-6599749ddd-kbw8v" event={"ID":"02be284e-3e49-4fa7-8475-a5a79f8997b1","Type":"ContainerStarted","Data":"6246f3109c9c5fa654478bf6806e90fc848528247cd96cec81ce573e05cca639"} Oct 29 22:51:40.311593 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:40.311400 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-b2921b-6599749ddd-kbw8v" Oct 29 22:51:40.354273 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:40.354231 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-b2921b-6599749ddd-kbw8v" podStartSLOduration=1.035326479 podStartE2EDuration="1.354217735s" totalImagesPullingTime="318.891256ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:51:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:51:40.351395401 +0000 UTC m=+1081.134608828" watchObservedRunningTime="2026-10-29 22:51:40.354217735 +0000 UTC m=+1081.137431161" Oct 29 22:51:41.130803 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:41.130768 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:51:41.285651 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:41.285620 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-575d59-55c4d89464-snt8r" Oct 29 22:51:42.655355 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:42.655280 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:51:44.159246 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:44.159214 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:51:45.775369 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:45.775309 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:51:46.325657 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:46.325622 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-b2921b-6599749ddd-kbw8v" Oct 29 22:51:46.991027 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:46.990994 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v2tfm"] Oct 29 22:51:47.390655 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:47.390562 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:51:47.505112 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:47.505075 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:51:49.030019 ip-10-0-140-74 kubenswrapper[5057]: E1029 22:51:49.029976 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d2dnn" containerName="extractor" Oct 29 22:51:49.087810 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:49.087777 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:51:50.832207 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:50.832168 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:51:51.847984 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:51.847948 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:51:52.622279 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:52.622228 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:51:53.149922 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:53.149868 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:51:53.343987 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:53.343959 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:51:53.771652 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:53.771618 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:51:53.892791 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:53.892749 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:51:54.000701 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:54.000663 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:51:54.514545 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:54.514510 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:51:56.387186 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:56.387146 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:51:56.673269 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:56.673175 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:51:57.305129 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:57.305096 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:51:57.420605 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:57.419545 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:51:57.544975 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:57.544937 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:51:57.998983 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:57.998943 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:51:58.545364 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:58.545319 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v2tfm"] Oct 29 22:51:59.683495 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:51:59.683454 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:52:01.323114 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:52:01.323081 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:52:03.117286 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:52:03.117235 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:52:04.657259 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:52:04.657223 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:52:06.092927 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:52:06.092891 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:52:06.540618 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:52:06.540576 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:52:07.968891 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:52:07.968855 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:52:09.389709 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:52:09.389669 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:52:10.907283 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:52:10.907250 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:52:12.510526 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:52:12.510491 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:52:13.163954 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:52:13.163917 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:52:14.085515 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:52:14.085479 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v2tfm"] Oct 29 22:52:14.110651 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:52:14.110615 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:52:15.712938 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:52:15.712906 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:52:17.196101 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:52:17.196061 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:52:18.790062 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:52:18.790021 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:52:19.028079 ip-10-0-140-74 kubenswrapper[5057]: E1029 22:52:19.028046 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d2dnn" containerName="extractor" Oct 29 22:52:20.280158 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:52:20.280120 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:52:20.482029 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:52:20.481994 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-pk92s"] Oct 29 22:52:20.488693 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:52:20.488667 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-pk92s" Oct 29 22:52:20.513537 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:52:20.509498 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-pk92s"] Oct 29 22:52:20.514687 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:52:20.514650 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-31226c-openshift-default-dockercfg-cmhxj\"" Oct 29 22:52:20.553721 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:52:20.553652 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9033e6ee-4fd2-41a2-8a7d-132b6ca20128-istio-data\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-pk92s\" (UID: \"9033e6ee-4fd2-41a2-8a7d-132b6ca20128\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-pk92s" Oct 29 22:52:20.553835 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:52:20.553717 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9033e6ee-4fd2-41a2-8a7d-132b6ca20128-credential-socket\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-pk92s\" (UID: \"9033e6ee-4fd2-41a2-8a7d-132b6ca20128\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-pk92s" Oct 29 22:52:20.553835 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:52:20.553821 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9033e6ee-4fd2-41a2-8a7d-132b6ca20128-workload-certs\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-pk92s\" (UID: \"9033e6ee-4fd2-41a2-8a7d-132b6ca20128\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-pk92s" Oct 29 22:52:20.553918 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:52:20.553905 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9033e6ee-4fd2-41a2-8a7d-132b6ca20128-workload-socket\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-pk92s\" (UID: \"9033e6ee-4fd2-41a2-8a7d-132b6ca20128\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-pk92s" Oct 29 22:52:20.553982 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:52:20.553962 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9033e6ee-4fd2-41a2-8a7d-132b6ca20128-istio-envoy\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-pk92s\" (UID: \"9033e6ee-4fd2-41a2-8a7d-132b6ca20128\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-pk92s" Oct 29 22:52:20.554022 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:52:20.554009 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9033e6ee-4fd2-41a2-8a7d-132b6ca20128-istio-token\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-pk92s\" (UID: \"9033e6ee-4fd2-41a2-8a7d-132b6ca20128\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-pk92s" Oct 29 22:52:20.554071 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:52:20.554036 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9033e6ee-4fd2-41a2-8a7d-132b6ca20128-istio-podinfo\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-pk92s\" (UID: \"9033e6ee-4fd2-41a2-8a7d-132b6ca20128\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-pk92s" Oct 29 22:52:20.554117 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:52:20.554103 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrkk5\" (UniqueName: \"kubernetes.io/projected/9033e6ee-4fd2-41a2-8a7d-132b6ca20128-kube-api-access-wrkk5\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-pk92s\" (UID: \"9033e6ee-4fd2-41a2-8a7d-132b6ca20128\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-pk92s" Oct 29 22:52:20.554193 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:52:20.554173 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9033e6ee-4fd2-41a2-8a7d-132b6ca20128-istiod-ca-cert\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-pk92s\" (UID: \"9033e6ee-4fd2-41a2-8a7d-132b6ca20128\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-pk92s" Oct 29 22:52:20.655173 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:52:20.655143 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrkk5\" (UniqueName: \"kubernetes.io/projected/9033e6ee-4fd2-41a2-8a7d-132b6ca20128-kube-api-access-wrkk5\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-pk92s\" (UID: \"9033e6ee-4fd2-41a2-8a7d-132b6ca20128\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-pk92s" Oct 29 22:52:20.655369 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:52:20.655180 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9033e6ee-4fd2-41a2-8a7d-132b6ca20128-istiod-ca-cert\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-pk92s\" (UID: \"9033e6ee-4fd2-41a2-8a7d-132b6ca20128\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-pk92s" Oct 29 22:52:20.655369 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:52:20.655316 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9033e6ee-4fd2-41a2-8a7d-132b6ca20128-istio-data\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-pk92s\" (UID: \"9033e6ee-4fd2-41a2-8a7d-132b6ca20128\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-pk92s" Oct 29 22:52:20.655500 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:52:20.655392 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9033e6ee-4fd2-41a2-8a7d-132b6ca20128-credential-socket\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-pk92s\" (UID: \"9033e6ee-4fd2-41a2-8a7d-132b6ca20128\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-pk92s" Oct 29 22:52:20.655500 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:52:20.655458 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9033e6ee-4fd2-41a2-8a7d-132b6ca20128-workload-certs\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-pk92s\" (UID: \"9033e6ee-4fd2-41a2-8a7d-132b6ca20128\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-pk92s" Oct 29 22:52:20.655609 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:52:20.655536 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9033e6ee-4fd2-41a2-8a7d-132b6ca20128-workload-socket\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-pk92s\" (UID: \"9033e6ee-4fd2-41a2-8a7d-132b6ca20128\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-pk92s" Oct 29 22:52:20.655609 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:52:20.655592 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9033e6ee-4fd2-41a2-8a7d-132b6ca20128-istio-envoy\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-pk92s\" (UID: \"9033e6ee-4fd2-41a2-8a7d-132b6ca20128\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-pk92s" Oct 29 22:52:20.655715 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:52:20.655653 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9033e6ee-4fd2-41a2-8a7d-132b6ca20128-istio-token\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-pk92s\" (UID: \"9033e6ee-4fd2-41a2-8a7d-132b6ca20128\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-pk92s" Oct 29 22:52:20.655715 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:52:20.655701 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9033e6ee-4fd2-41a2-8a7d-132b6ca20128-istio-podinfo\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-pk92s\" (UID: \"9033e6ee-4fd2-41a2-8a7d-132b6ca20128\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-pk92s" Oct 29 22:52:20.655851 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:52:20.655716 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9033e6ee-4fd2-41a2-8a7d-132b6ca20128-istio-data\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-pk92s\" (UID: \"9033e6ee-4fd2-41a2-8a7d-132b6ca20128\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-pk92s" Oct 29 22:52:20.655851 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:52:20.655755 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9033e6ee-4fd2-41a2-8a7d-132b6ca20128-credential-socket\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-pk92s\" (UID: \"9033e6ee-4fd2-41a2-8a7d-132b6ca20128\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-pk92s" Oct 29 22:52:20.655988 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:52:20.655966 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9033e6ee-4fd2-41a2-8a7d-132b6ca20128-istiod-ca-cert\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-pk92s\" (UID: \"9033e6ee-4fd2-41a2-8a7d-132b6ca20128\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-pk92s" Oct 29 22:52:20.656110 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:52:20.656091 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9033e6ee-4fd2-41a2-8a7d-132b6ca20128-workload-certs\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-pk92s\" (UID: \"9033e6ee-4fd2-41a2-8a7d-132b6ca20128\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-pk92s" Oct 29 22:52:20.656212 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:52:20.656193 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9033e6ee-4fd2-41a2-8a7d-132b6ca20128-workload-socket\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-pk92s\" (UID: \"9033e6ee-4fd2-41a2-8a7d-132b6ca20128\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-pk92s" Oct 29 22:52:20.658093 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:52:20.658065 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9033e6ee-4fd2-41a2-8a7d-132b6ca20128-istio-envoy\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-pk92s\" (UID: \"9033e6ee-4fd2-41a2-8a7d-132b6ca20128\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-pk92s" Oct 29 22:52:20.658209 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:52:20.658190 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9033e6ee-4fd2-41a2-8a7d-132b6ca20128-istio-podinfo\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-pk92s\" (UID: \"9033e6ee-4fd2-41a2-8a7d-132b6ca20128\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-pk92s" Oct 29 22:52:20.676581 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:52:20.676563 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9033e6ee-4fd2-41a2-8a7d-132b6ca20128-istio-token\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-pk92s\" (UID: \"9033e6ee-4fd2-41a2-8a7d-132b6ca20128\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-pk92s" Oct 29 22:52:20.677015 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:52:20.676980 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrkk5\" (UniqueName: \"kubernetes.io/projected/9033e6ee-4fd2-41a2-8a7d-132b6ca20128-kube-api-access-wrkk5\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-pk92s\" (UID: \"9033e6ee-4fd2-41a2-8a7d-132b6ca20128\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-pk92s" Oct 29 22:52:20.808010 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:52:20.807899 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-pk92s" Oct 29 22:52:21.125721 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:52:21.125689 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-pk92s"] Oct 29 22:52:21.127145 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:52:21.127116 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9033e6ee_4fd2_41a2_8a7d_132b6ca20128.slice/crio-40b9e2c0215dee2358c1f1676159574d7c697dca9fbbda58fd701876e5dc48f9 WatchSource:0}: Error finding container 40b9e2c0215dee2358c1f1676159574d7c697dca9fbbda58fd701876e5dc48f9: Status 404 returned error can't find the container with id 40b9e2c0215dee2358c1f1676159574d7c697dca9fbbda58fd701876e5dc48f9 Oct 29 22:52:21.130387 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:52:21.130088 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:52:21.132165 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:52:21.130308 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:52:21.132165 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:52:21.130666 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:52:21.581996 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:52:21.581953 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-pk92s" event={"ID":"9033e6ee-4fd2-41a2-8a7d-132b6ca20128","Type":"ContainerStarted","Data":"9c6dd6a2b4c881e2526567b87b68d980e665674e84932f76ecd320f922734f62"} Oct 29 22:52:21.582529 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:52:21.582013 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-pk92s" event={"ID":"9033e6ee-4fd2-41a2-8a7d-132b6ca20128","Type":"ContainerStarted","Data":"40b9e2c0215dee2358c1f1676159574d7c697dca9fbbda58fd701876e5dc48f9"} Oct 29 22:52:21.638285 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:52:21.638228 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-pk92s" podStartSLOduration=1.6382118079999999 podStartE2EDuration="1.638211808s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:52:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:52:21.635932764 +0000 UTC m=+1122.419146216" watchObservedRunningTime="2026-10-29 22:52:21.638211808 +0000 UTC m=+1122.421425235" Oct 29 22:52:21.808525 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:52:21.808482 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-pk92s" Oct 29 22:52:21.970661 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:52:21.970616 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:52:22.810233 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:52:22.810198 5057 patch_prober.go:29] interesting pod/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-pk92s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Oct 29 22:52:22.810740 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:52:22.810268 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-pk92s" podUID="9033e6ee-4fd2-41a2-8a7d-132b6ca20128" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Oct 29 22:52:23.468064 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:52:23.468016 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:52:23.808950 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:52:23.808869 5057 patch_prober.go:29] interesting pod/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-pk92s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Oct 29 22:52:23.809167 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:52:23.809142 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-pk92s" podUID="9033e6ee-4fd2-41a2-8a7d-132b6ca20128" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Oct 29 22:52:24.808791 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:52:24.808752 5057 patch_prober.go:29] interesting pod/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-pk92s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Oct 29 22:52:24.809281 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:52:24.808805 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-pk92s" podUID="9033e6ee-4fd2-41a2-8a7d-132b6ca20128" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Oct 29 22:52:24.993356 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:52:24.993276 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:52:25.808930 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:52:25.808856 5057 patch_prober.go:29] interesting pod/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-pk92s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Oct 29 22:52:25.808930 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:52:25.808913 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-pk92s" podUID="9033e6ee-4fd2-41a2-8a7d-132b6ca20128" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Oct 29 22:52:26.597915 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:52:26.597829 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:52:26.808903 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:52:26.808866 5057 patch_prober.go:29] interesting pod/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-pk92s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Oct 29 22:52:26.809215 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:52:26.809174 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-pk92s" podUID="9033e6ee-4fd2-41a2-8a7d-132b6ca20128" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Oct 29 22:52:27.812785 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:52:27.812756 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-pk92s" Oct 29 22:52:27.813211 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:52:27.813183 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-pk92s" Oct 29 22:52:27.813842 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:52:27.813817 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-pk92s" Oct 29 22:52:28.180187 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:52:28.180099 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:52:28.183225 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:52:28.183197 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v2tfm"] Oct 29 22:52:28.833970 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:52:28.833933 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:52:29.465670 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:52:29.465634 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-4djh7"] Oct 29 22:52:29.469878 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:52:29.469857 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-4djh7" Oct 29 22:52:29.500581 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:52:29.500554 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-e131af-openshift-default-dockercfg-796wj\"" Oct 29 22:52:29.505682 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:52:29.505655 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-4djh7"] Oct 29 22:52:29.587182 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:52:29.587142 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3637f82c-7b89-4a17-aa32-5a99d1647969-credential-socket\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-4djh7\" (UID: \"3637f82c-7b89-4a17-aa32-5a99d1647969\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-4djh7" Oct 29 22:52:29.587369 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:52:29.587242 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3637f82c-7b89-4a17-aa32-5a99d1647969-istio-envoy\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-4djh7\" (UID: \"3637f82c-7b89-4a17-aa32-5a99d1647969\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-4djh7" Oct 29 22:52:29.587369 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:52:29.587270 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3637f82c-7b89-4a17-aa32-5a99d1647969-istio-podinfo\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-4djh7\" (UID: \"3637f82c-7b89-4a17-aa32-5a99d1647969\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-4djh7" Oct 29 22:52:29.587369 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:52:29.587292 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3637f82c-7b89-4a17-aa32-5a99d1647969-workload-socket\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-4djh7\" (UID: \"3637f82c-7b89-4a17-aa32-5a99d1647969\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-4djh7" Oct 29 22:52:29.587369 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:52:29.587310 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3637f82c-7b89-4a17-aa32-5a99d1647969-istio-token\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-4djh7\" (UID: \"3637f82c-7b89-4a17-aa32-5a99d1647969\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-4djh7" Oct 29 22:52:29.587722 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:52:29.587403 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3637f82c-7b89-4a17-aa32-5a99d1647969-istiod-ca-cert\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-4djh7\" (UID: \"3637f82c-7b89-4a17-aa32-5a99d1647969\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-4djh7" Oct 29 22:52:29.587722 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:52:29.587442 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mjdk\" (UniqueName: \"kubernetes.io/projected/3637f82c-7b89-4a17-aa32-5a99d1647969-kube-api-access-4mjdk\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-4djh7\" (UID: \"3637f82c-7b89-4a17-aa32-5a99d1647969\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-4djh7" Oct 29 22:52:29.587722 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:52:29.587472 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3637f82c-7b89-4a17-aa32-5a99d1647969-workload-certs\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-4djh7\" (UID: \"3637f82c-7b89-4a17-aa32-5a99d1647969\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-4djh7" Oct 29 22:52:29.587883 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:52:29.587837 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3637f82c-7b89-4a17-aa32-5a99d1647969-istio-data\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-4djh7\" (UID: \"3637f82c-7b89-4a17-aa32-5a99d1647969\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-4djh7" Oct 29 22:52:29.682517 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:52:29.682481 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:52:29.688723 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:52:29.688696 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3637f82c-7b89-4a17-aa32-5a99d1647969-workload-socket\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-4djh7\" (UID: \"3637f82c-7b89-4a17-aa32-5a99d1647969\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-4djh7" Oct 29 22:52:29.688836 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:52:29.688730 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3637f82c-7b89-4a17-aa32-5a99d1647969-istio-token\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-4djh7\" (UID: \"3637f82c-7b89-4a17-aa32-5a99d1647969\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-4djh7" Oct 29 22:52:29.688836 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:52:29.688748 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3637f82c-7b89-4a17-aa32-5a99d1647969-istiod-ca-cert\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-4djh7\" (UID: \"3637f82c-7b89-4a17-aa32-5a99d1647969\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-4djh7" Oct 29 22:52:29.688836 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:52:29.688773 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mjdk\" (UniqueName: \"kubernetes.io/projected/3637f82c-7b89-4a17-aa32-5a99d1647969-kube-api-access-4mjdk\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-4djh7\" (UID: \"3637f82c-7b89-4a17-aa32-5a99d1647969\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-4djh7" Oct 29 22:52:29.688836 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:52:29.688796 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3637f82c-7b89-4a17-aa32-5a99d1647969-workload-certs\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-4djh7\" (UID: \"3637f82c-7b89-4a17-aa32-5a99d1647969\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-4djh7" Oct 29 22:52:29.689041 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:52:29.689024 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3637f82c-7b89-4a17-aa32-5a99d1647969-istio-data\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-4djh7\" (UID: \"3637f82c-7b89-4a17-aa32-5a99d1647969\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-4djh7" Oct 29 22:52:29.689152 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:52:29.689126 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3637f82c-7b89-4a17-aa32-5a99d1647969-workload-socket\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-4djh7\" (UID: \"3637f82c-7b89-4a17-aa32-5a99d1647969\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-4djh7" Oct 29 22:52:29.689245 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:52:29.689172 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3637f82c-7b89-4a17-aa32-5a99d1647969-credential-socket\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-4djh7\" (UID: \"3637f82c-7b89-4a17-aa32-5a99d1647969\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-4djh7" Oct 29 22:52:29.689245 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:52:29.689227 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3637f82c-7b89-4a17-aa32-5a99d1647969-workload-certs\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-4djh7\" (UID: \"3637f82c-7b89-4a17-aa32-5a99d1647969\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-4djh7" Oct 29 22:52:29.689349 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:52:29.689257 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3637f82c-7b89-4a17-aa32-5a99d1647969-istio-envoy\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-4djh7\" (UID: \"3637f82c-7b89-4a17-aa32-5a99d1647969\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-4djh7" Oct 29 22:52:29.689349 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:52:29.689300 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3637f82c-7b89-4a17-aa32-5a99d1647969-istio-podinfo\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-4djh7\" (UID: \"3637f82c-7b89-4a17-aa32-5a99d1647969\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-4djh7" Oct 29 22:52:29.689441 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:52:29.689394 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3637f82c-7b89-4a17-aa32-5a99d1647969-istio-data\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-4djh7\" (UID: \"3637f82c-7b89-4a17-aa32-5a99d1647969\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-4djh7" Oct 29 22:52:29.689509 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:52:29.689490 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3637f82c-7b89-4a17-aa32-5a99d1647969-istiod-ca-cert\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-4djh7\" (UID: \"3637f82c-7b89-4a17-aa32-5a99d1647969\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-4djh7" Oct 29 22:52:29.689550 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:52:29.689508 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3637f82c-7b89-4a17-aa32-5a99d1647969-credential-socket\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-4djh7\" (UID: \"3637f82c-7b89-4a17-aa32-5a99d1647969\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-4djh7" Oct 29 22:52:29.691384 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:52:29.691356 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3637f82c-7b89-4a17-aa32-5a99d1647969-istio-envoy\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-4djh7\" (UID: \"3637f82c-7b89-4a17-aa32-5a99d1647969\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-4djh7" Oct 29 22:52:29.691981 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:52:29.691951 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3637f82c-7b89-4a17-aa32-5a99d1647969-istio-podinfo\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-4djh7\" (UID: \"3637f82c-7b89-4a17-aa32-5a99d1647969\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-4djh7" Oct 29 22:52:29.709768 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:52:29.709747 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3637f82c-7b89-4a17-aa32-5a99d1647969-istio-token\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-4djh7\" (UID: \"3637f82c-7b89-4a17-aa32-5a99d1647969\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-4djh7" Oct 29 22:52:29.709940 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:52:29.709922 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mjdk\" (UniqueName: \"kubernetes.io/projected/3637f82c-7b89-4a17-aa32-5a99d1647969-kube-api-access-4mjdk\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-4djh7\" (UID: \"3637f82c-7b89-4a17-aa32-5a99d1647969\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-4djh7" Oct 29 22:52:29.785617 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:52:29.785532 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-4djh7" Oct 29 22:52:30.111128 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:52:30.111099 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-4djh7"] Oct 29 22:52:30.112454 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:52:30.112425 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3637f82c_7b89_4a17_aa32_5a99d1647969.slice/crio-11902b12f09c433ec14e784ae489a99cfaa1bd479bc5cb51144749a97d05a502 WatchSource:0}: Error finding container 11902b12f09c433ec14e784ae489a99cfaa1bd479bc5cb51144749a97d05a502: Status 404 returned error can't find the container with id 11902b12f09c433ec14e784ae489a99cfaa1bd479bc5cb51144749a97d05a502 Oct 29 22:52:30.114945 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:52:30.114911 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:52:30.115056 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:52:30.114976 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:52:30.115056 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:52:30.115004 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:52:30.667925 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:52:30.667879 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-4djh7" event={"ID":"3637f82c-7b89-4a17-aa32-5a99d1647969","Type":"ContainerStarted","Data":"69a57ffaa5960e83b11d95ab053b106f4a165c5cfeab2b1a38f0812f04a96877"} Oct 29 22:52:30.668099 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:52:30.667957 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-4djh7" event={"ID":"3637f82c-7b89-4a17-aa32-5a99d1647969","Type":"ContainerStarted","Data":"11902b12f09c433ec14e784ae489a99cfaa1bd479bc5cb51144749a97d05a502"} Oct 29 22:52:30.714660 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:52:30.714600 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-4djh7" podStartSLOduration=1.714586054 podStartE2EDuration="1.714586054s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:52:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:52:30.712882783 +0000 UTC m=+1131.496096200" watchObservedRunningTime="2026-10-29 22:52:30.714586054 +0000 UTC m=+1131.497799479" Oct 29 22:52:30.793354 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:52:30.788429 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-4djh7" Oct 29 22:52:31.049906 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:52:31.049866 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:52:31.305441 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:52:31.305355 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:52:31.792356 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:52:31.791426 5057 patch_prober.go:29] interesting pod/e2e-worker-w1-e131af-openshift-default-c476c699c-4djh7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Oct 29 22:52:31.792356 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:52:31.791522 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-4djh7" podUID="3637f82c-7b89-4a17-aa32-5a99d1647969" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Oct 29 22:52:32.756575 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:52:32.756539 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:52:32.788703 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:52:32.788669 5057 patch_prober.go:29] interesting pod/e2e-worker-w1-e131af-openshift-default-c476c699c-4djh7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Oct 29 22:52:32.788947 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:52:32.788920 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-4djh7" podUID="3637f82c-7b89-4a17-aa32-5a99d1647969" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Oct 29 22:52:33.788355 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:52:33.786693 5057 patch_prober.go:29] interesting pod/e2e-worker-w1-e131af-openshift-default-c476c699c-4djh7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Oct 29 22:52:33.788355 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:52:33.786763 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-4djh7" podUID="3637f82c-7b89-4a17-aa32-5a99d1647969" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Oct 29 22:52:33.899356 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:52:33.895305 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:52:34.008151 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:52:34.008113 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:52:34.335324 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:52:34.335281 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:52:34.802469 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:52:34.802435 5057 patch_prober.go:29] interesting pod/e2e-worker-w1-e131af-openshift-default-c476c699c-4djh7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Oct 29 22:52:34.803085 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:52:34.803048 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-4djh7" podUID="3637f82c-7b89-4a17-aa32-5a99d1647969" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Oct 29 22:52:35.790203 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:52:35.789732 5057 patch_prober.go:29] interesting pod/e2e-worker-w1-e131af-openshift-default-c476c699c-4djh7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Oct 29 22:52:35.790203 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:52:35.789799 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-4djh7" podUID="3637f82c-7b89-4a17-aa32-5a99d1647969" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Oct 29 22:52:35.898357 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:52:35.893775 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:52:37.592257 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:52:37.592216 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-4djh7" Oct 29 22:52:37.593342 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:52:37.593304 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-4djh7" Oct 29 22:52:37.599856 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:52:37.599826 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-4djh7" Oct 29 22:52:37.698975 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:52:37.698944 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:52:39.295390 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:52:39.295360 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:52:40.205493 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:52:40.203689 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:52:40.890565 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:52:40.890528 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:52:42.490076 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:52:42.490030 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:52:42.601561 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:52:42.601525 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:52:44.009311 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:52:44.009274 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:52:45.168944 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:52:45.168902 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v2tfm"] Oct 29 22:52:45.514649 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:52:45.514617 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:52:47.094961 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:52:47.094919 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:52:47.656121 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:52:47.656084 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:52:48.702013 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:52:48.701977 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:52:49.029119 ip-10-0-140-74 kubenswrapper[5057]: E1029 22:52:49.029038 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d2dnn" containerName="extractor" Oct 29 22:52:50.211952 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:52:50.211908 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:52:51.778837 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:52:51.778797 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:52:53.201760 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:52:53.201726 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:52:54.598632 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:52:54.598580 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:52:56.110933 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:52:56.110889 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:52:56.472649 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:52:56.472606 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:52:57.886104 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:52:57.886060 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:52:59.497192 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:52:59.497154 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:53:00.907483 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:00.907449 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:53:02.380765 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:02.380727 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:53:03.796970 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:03.796918 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:53:05.284904 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:05.284866 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:53:06.081134 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:06.081099 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-lfnnh"] Oct 29 22:53:06.085743 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:06.085719 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-lfnnh" Oct 29 22:53:06.100261 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:06.100232 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-063b46-openshift-default-dockercfg-hj7tc\"" Oct 29 22:53:06.121113 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:06.121085 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-lfnnh"] Oct 29 22:53:06.218190 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:06.218151 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e344401d-57e1-47dc-bf88-dfee4e4b0914-istiod-ca-cert\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-lfnnh\" (UID: \"e344401d-57e1-47dc-bf88-dfee4e4b0914\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-lfnnh" Oct 29 22:53:06.218190 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:06.218200 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e344401d-57e1-47dc-bf88-dfee4e4b0914-workload-certs\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-lfnnh\" (UID: \"e344401d-57e1-47dc-bf88-dfee4e4b0914\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-lfnnh" Oct 29 22:53:06.218451 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:06.218219 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e344401d-57e1-47dc-bf88-dfee4e4b0914-istio-podinfo\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-lfnnh\" (UID: \"e344401d-57e1-47dc-bf88-dfee4e4b0914\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-lfnnh" Oct 29 22:53:06.218451 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:06.218409 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e344401d-57e1-47dc-bf88-dfee4e4b0914-credential-socket\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-lfnnh\" (UID: \"e344401d-57e1-47dc-bf88-dfee4e4b0914\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-lfnnh" Oct 29 22:53:06.218570 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:06.218455 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e344401d-57e1-47dc-bf88-dfee4e4b0914-istio-token\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-lfnnh\" (UID: \"e344401d-57e1-47dc-bf88-dfee4e4b0914\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-lfnnh" Oct 29 22:53:06.218570 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:06.218546 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8htl\" (UniqueName: \"kubernetes.io/projected/e344401d-57e1-47dc-bf88-dfee4e4b0914-kube-api-access-j8htl\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-lfnnh\" (UID: \"e344401d-57e1-47dc-bf88-dfee4e4b0914\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-lfnnh" Oct 29 22:53:06.218677 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:06.218583 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e344401d-57e1-47dc-bf88-dfee4e4b0914-istio-data\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-lfnnh\" (UID: \"e344401d-57e1-47dc-bf88-dfee4e4b0914\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-lfnnh" Oct 29 22:53:06.218730 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:06.218681 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e344401d-57e1-47dc-bf88-dfee4e4b0914-workload-socket\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-lfnnh\" (UID: \"e344401d-57e1-47dc-bf88-dfee4e4b0914\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-lfnnh" Oct 29 22:53:06.218730 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:06.218720 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e344401d-57e1-47dc-bf88-dfee4e4b0914-istio-envoy\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-lfnnh\" (UID: \"e344401d-57e1-47dc-bf88-dfee4e4b0914\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-lfnnh" Oct 29 22:53:06.319821 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:06.319787 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e344401d-57e1-47dc-bf88-dfee4e4b0914-istio-token\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-lfnnh\" (UID: \"e344401d-57e1-47dc-bf88-dfee4e4b0914\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-lfnnh" Oct 29 22:53:06.320245 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:06.319872 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8htl\" (UniqueName: \"kubernetes.io/projected/e344401d-57e1-47dc-bf88-dfee4e4b0914-kube-api-access-j8htl\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-lfnnh\" (UID: \"e344401d-57e1-47dc-bf88-dfee4e4b0914\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-lfnnh" Oct 29 22:53:06.320245 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:06.319911 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e344401d-57e1-47dc-bf88-dfee4e4b0914-istio-data\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-lfnnh\" (UID: \"e344401d-57e1-47dc-bf88-dfee4e4b0914\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-lfnnh" Oct 29 22:53:06.320245 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:06.319967 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e344401d-57e1-47dc-bf88-dfee4e4b0914-workload-socket\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-lfnnh\" (UID: \"e344401d-57e1-47dc-bf88-dfee4e4b0914\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-lfnnh" Oct 29 22:53:06.320245 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:06.320004 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e344401d-57e1-47dc-bf88-dfee4e4b0914-istio-envoy\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-lfnnh\" (UID: \"e344401d-57e1-47dc-bf88-dfee4e4b0914\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-lfnnh" Oct 29 22:53:06.320245 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:06.320145 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e344401d-57e1-47dc-bf88-dfee4e4b0914-istiod-ca-cert\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-lfnnh\" (UID: \"e344401d-57e1-47dc-bf88-dfee4e4b0914\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-lfnnh" Oct 29 22:53:06.320245 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:06.320178 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e344401d-57e1-47dc-bf88-dfee4e4b0914-workload-certs\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-lfnnh\" (UID: \"e344401d-57e1-47dc-bf88-dfee4e4b0914\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-lfnnh" Oct 29 22:53:06.320245 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:06.320200 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e344401d-57e1-47dc-bf88-dfee4e4b0914-istio-podinfo\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-lfnnh\" (UID: \"e344401d-57e1-47dc-bf88-dfee4e4b0914\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-lfnnh" Oct 29 22:53:06.320610 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:06.320260 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e344401d-57e1-47dc-bf88-dfee4e4b0914-credential-socket\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-lfnnh\" (UID: \"e344401d-57e1-47dc-bf88-dfee4e4b0914\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-lfnnh" Oct 29 22:53:06.320610 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:06.320492 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e344401d-57e1-47dc-bf88-dfee4e4b0914-istio-data\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-lfnnh\" (UID: \"e344401d-57e1-47dc-bf88-dfee4e4b0914\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-lfnnh" Oct 29 22:53:06.320723 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:06.320669 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e344401d-57e1-47dc-bf88-dfee4e4b0914-credential-socket\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-lfnnh\" (UID: \"e344401d-57e1-47dc-bf88-dfee4e4b0914\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-lfnnh" Oct 29 22:53:06.320783 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:06.320762 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e344401d-57e1-47dc-bf88-dfee4e4b0914-workload-socket\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-lfnnh\" (UID: \"e344401d-57e1-47dc-bf88-dfee4e4b0914\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-lfnnh" Oct 29 22:53:06.320963 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:06.320906 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e344401d-57e1-47dc-bf88-dfee4e4b0914-workload-certs\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-lfnnh\" (UID: \"e344401d-57e1-47dc-bf88-dfee4e4b0914\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-lfnnh" Oct 29 22:53:06.321464 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:06.321438 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e344401d-57e1-47dc-bf88-dfee4e4b0914-istiod-ca-cert\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-lfnnh\" (UID: \"e344401d-57e1-47dc-bf88-dfee4e4b0914\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-lfnnh" Oct 29 22:53:06.322823 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:06.322797 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e344401d-57e1-47dc-bf88-dfee4e4b0914-istio-envoy\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-lfnnh\" (UID: \"e344401d-57e1-47dc-bf88-dfee4e4b0914\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-lfnnh" Oct 29 22:53:06.323190 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:06.323162 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e344401d-57e1-47dc-bf88-dfee4e4b0914-istio-podinfo\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-lfnnh\" (UID: \"e344401d-57e1-47dc-bf88-dfee4e4b0914\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-lfnnh" Oct 29 22:53:06.341456 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:06.341399 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e344401d-57e1-47dc-bf88-dfee4e4b0914-istio-token\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-lfnnh\" (UID: \"e344401d-57e1-47dc-bf88-dfee4e4b0914\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-lfnnh" Oct 29 22:53:06.341551 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:06.341500 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8htl\" (UniqueName: \"kubernetes.io/projected/e344401d-57e1-47dc-bf88-dfee4e4b0914-kube-api-access-j8htl\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-lfnnh\" (UID: \"e344401d-57e1-47dc-bf88-dfee4e4b0914\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-lfnnh" Oct 29 22:53:06.399851 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:06.399824 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-lfnnh" Oct 29 22:53:06.720157 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:06.720125 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-lfnnh"] Oct 29 22:53:06.723045 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:53:06.723015 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode344401d_57e1_47dc_bf88_dfee4e4b0914.slice/crio-f907355d4d947f52e9652e1db188db95407dddb50d94046a27b4950b47ef1b18 WatchSource:0}: Error finding container f907355d4d947f52e9652e1db188db95407dddb50d94046a27b4950b47ef1b18: Status 404 returned error can't find the container with id f907355d4d947f52e9652e1db188db95407dddb50d94046a27b4950b47ef1b18 Oct 29 22:53:06.725646 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:06.725614 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:53:06.725717 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:06.725677 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:53:06.725717 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:06.725706 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:53:06.778988 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:06.778954 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:53:06.931647 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:06.931608 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-lfnnh" event={"ID":"e344401d-57e1-47dc-bf88-dfee4e4b0914","Type":"ContainerStarted","Data":"0286fb961c72f6b96a5a3fa53564bef1a75256f6dd1408cda4e5d78f2d7d92af"} Oct 29 22:53:06.931864 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:06.931676 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-lfnnh" event={"ID":"e344401d-57e1-47dc-bf88-dfee4e4b0914","Type":"ContainerStarted","Data":"f907355d4d947f52e9652e1db188db95407dddb50d94046a27b4950b47ef1b18"} Oct 29 22:53:06.977451 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:06.977367 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-lfnnh" podStartSLOduration=0.97735365 podStartE2EDuration="977.35365ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:53:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:53:06.974663753 +0000 UTC m=+1167.757877179" watchObservedRunningTime="2026-10-29 22:53:06.97735365 +0000 UTC m=+1167.760567073" Oct 29 22:53:07.403820 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:07.403501 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-lfnnh" Oct 29 22:53:07.408490 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:07.408457 5057 patch_prober.go:29] interesting pod/e2e-worker-w4-063b46-openshift-default-799d8d64cf-lfnnh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Oct 29 22:53:07.408762 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:07.408721 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-lfnnh" podUID="e344401d-57e1-47dc-bf88-dfee4e4b0914" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Oct 29 22:53:07.447464 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:07.447428 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v2tfm"] Oct 29 22:53:08.404703 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:08.404666 5057 patch_prober.go:29] interesting pod/e2e-worker-w4-063b46-openshift-default-799d8d64cf-lfnnh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Oct 29 22:53:08.406683 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:08.406649 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-lfnnh" podUID="e344401d-57e1-47dc-bf88-dfee4e4b0914" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Oct 29 22:53:08.459030 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:08.458990 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:53:09.411356 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:09.407702 5057 patch_prober.go:29] interesting pod/e2e-worker-w4-063b46-openshift-default-799d8d64cf-lfnnh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Oct 29 22:53:09.411356 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:09.407762 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-lfnnh" podUID="e344401d-57e1-47dc-bf88-dfee4e4b0914" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Oct 29 22:53:10.212091 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:10.212056 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:53:10.408565 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:10.408459 5057 patch_prober.go:29] interesting pod/e2e-worker-w4-063b46-openshift-default-799d8d64cf-lfnnh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Oct 29 22:53:10.408565 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:10.408526 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-lfnnh" podUID="e344401d-57e1-47dc-bf88-dfee4e4b0914" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Oct 29 22:53:11.401647 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:11.401610 5057 patch_prober.go:29] interesting pod/e2e-worker-w4-063b46-openshift-default-799d8d64cf-lfnnh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Oct 29 22:53:11.402219 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:11.402189 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-lfnnh" podUID="e344401d-57e1-47dc-bf88-dfee4e4b0914" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Oct 29 22:53:11.606551 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:11.606513 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:53:12.403689 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:12.403643 5057 patch_prober.go:29] interesting pod/e2e-worker-w4-063b46-openshift-default-799d8d64cf-lfnnh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Oct 29 22:53:12.404285 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:12.404255 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-lfnnh" podUID="e344401d-57e1-47dc-bf88-dfee4e4b0914" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Oct 29 22:53:13.110146 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:13.110097 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:53:14.218243 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:14.218209 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-lfnnh" Oct 29 22:53:14.218785 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:14.218664 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-lfnnh" Oct 29 22:53:14.219523 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:14.219499 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-lfnnh" Oct 29 22:53:14.688419 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:14.688309 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:53:16.206028 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:16.205993 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:53:17.795949 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:17.795912 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:53:18.639402 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:18.639357 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:53:19.028698 ip-10-0-140-74 kubenswrapper[5057]: E1029 22:53:19.028667 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d2dnn" containerName="extractor" Oct 29 22:53:19.306063 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:19.305979 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:53:20.901379 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:20.901344 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:53:22.376390 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:22.376355 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:53:22.932920 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:22.932885 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-dv8l2"] Oct 29 22:53:22.940447 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:22.940412 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-dv8l2" Oct 29 22:53:22.965103 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:22.965073 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-b2921b-openshift-default-dockercfg-652vx\"" Oct 29 22:53:22.969044 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:22.969013 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-dv8l2"] Oct 29 22:53:23.057259 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:23.057224 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxjrb\" (UniqueName: \"kubernetes.io/projected/a0d8f2a6-a6b5-44dd-99fb-38aef41899e4-kube-api-access-pxjrb\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-dv8l2\" (UID: \"a0d8f2a6-a6b5-44dd-99fb-38aef41899e4\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-dv8l2" Oct 29 22:53:23.057429 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:23.057274 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0d8f2a6-a6b5-44dd-99fb-38aef41899e4-istio-token\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-dv8l2\" (UID: \"a0d8f2a6-a6b5-44dd-99fb-38aef41899e4\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-dv8l2" Oct 29 22:53:23.057429 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:23.057371 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0d8f2a6-a6b5-44dd-99fb-38aef41899e4-istiod-ca-cert\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-dv8l2\" (UID: \"a0d8f2a6-a6b5-44dd-99fb-38aef41899e4\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-dv8l2" Oct 29 22:53:23.057511 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:23.057426 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0d8f2a6-a6b5-44dd-99fb-38aef41899e4-istio-data\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-dv8l2\" (UID: \"a0d8f2a6-a6b5-44dd-99fb-38aef41899e4\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-dv8l2" Oct 29 22:53:23.057511 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:23.057491 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0d8f2a6-a6b5-44dd-99fb-38aef41899e4-workload-certs\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-dv8l2\" (UID: \"a0d8f2a6-a6b5-44dd-99fb-38aef41899e4\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-dv8l2" Oct 29 22:53:23.057582 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:23.057519 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0d8f2a6-a6b5-44dd-99fb-38aef41899e4-istio-envoy\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-dv8l2\" (UID: \"a0d8f2a6-a6b5-44dd-99fb-38aef41899e4\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-dv8l2" Oct 29 22:53:23.057582 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:23.057534 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0d8f2a6-a6b5-44dd-99fb-38aef41899e4-istio-podinfo\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-dv8l2\" (UID: \"a0d8f2a6-a6b5-44dd-99fb-38aef41899e4\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-dv8l2" Oct 29 22:53:23.057664 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:23.057650 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0d8f2a6-a6b5-44dd-99fb-38aef41899e4-workload-socket\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-dv8l2\" (UID: \"a0d8f2a6-a6b5-44dd-99fb-38aef41899e4\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-dv8l2" Oct 29 22:53:23.057702 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:23.057674 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0d8f2a6-a6b5-44dd-99fb-38aef41899e4-credential-socket\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-dv8l2\" (UID: \"a0d8f2a6-a6b5-44dd-99fb-38aef41899e4\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-dv8l2" Oct 29 22:53:23.158542 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:23.158513 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxjrb\" (UniqueName: \"kubernetes.io/projected/a0d8f2a6-a6b5-44dd-99fb-38aef41899e4-kube-api-access-pxjrb\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-dv8l2\" (UID: \"a0d8f2a6-a6b5-44dd-99fb-38aef41899e4\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-dv8l2" Oct 29 22:53:23.158703 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:23.158555 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0d8f2a6-a6b5-44dd-99fb-38aef41899e4-istio-token\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-dv8l2\" (UID: \"a0d8f2a6-a6b5-44dd-99fb-38aef41899e4\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-dv8l2" Oct 29 22:53:23.158703 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:23.158572 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0d8f2a6-a6b5-44dd-99fb-38aef41899e4-istiod-ca-cert\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-dv8l2\" (UID: \"a0d8f2a6-a6b5-44dd-99fb-38aef41899e4\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-dv8l2" Oct 29 22:53:23.158703 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:23.158603 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0d8f2a6-a6b5-44dd-99fb-38aef41899e4-istio-data\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-dv8l2\" (UID: \"a0d8f2a6-a6b5-44dd-99fb-38aef41899e4\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-dv8l2" Oct 29 22:53:23.158703 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:23.158630 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0d8f2a6-a6b5-44dd-99fb-38aef41899e4-workload-certs\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-dv8l2\" (UID: \"a0d8f2a6-a6b5-44dd-99fb-38aef41899e4\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-dv8l2" Oct 29 22:53:23.158703 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:23.158653 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0d8f2a6-a6b5-44dd-99fb-38aef41899e4-istio-envoy\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-dv8l2\" (UID: \"a0d8f2a6-a6b5-44dd-99fb-38aef41899e4\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-dv8l2" Oct 29 22:53:23.158703 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:23.158668 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0d8f2a6-a6b5-44dd-99fb-38aef41899e4-istio-podinfo\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-dv8l2\" (UID: \"a0d8f2a6-a6b5-44dd-99fb-38aef41899e4\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-dv8l2" Oct 29 22:53:23.159019 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:23.158746 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0d8f2a6-a6b5-44dd-99fb-38aef41899e4-workload-socket\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-dv8l2\" (UID: \"a0d8f2a6-a6b5-44dd-99fb-38aef41899e4\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-dv8l2" Oct 29 22:53:23.159019 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:23.158783 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0d8f2a6-a6b5-44dd-99fb-38aef41899e4-credential-socket\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-dv8l2\" (UID: \"a0d8f2a6-a6b5-44dd-99fb-38aef41899e4\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-dv8l2" Oct 29 22:53:23.159132 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:23.159032 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0d8f2a6-a6b5-44dd-99fb-38aef41899e4-istio-data\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-dv8l2\" (UID: \"a0d8f2a6-a6b5-44dd-99fb-38aef41899e4\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-dv8l2" Oct 29 22:53:23.159132 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:23.159117 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0d8f2a6-a6b5-44dd-99fb-38aef41899e4-workload-certs\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-dv8l2\" (UID: \"a0d8f2a6-a6b5-44dd-99fb-38aef41899e4\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-dv8l2" Oct 29 22:53:23.159255 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:23.159161 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0d8f2a6-a6b5-44dd-99fb-38aef41899e4-credential-socket\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-dv8l2\" (UID: \"a0d8f2a6-a6b5-44dd-99fb-38aef41899e4\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-dv8l2" Oct 29 22:53:23.159649 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:23.159622 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0d8f2a6-a6b5-44dd-99fb-38aef41899e4-workload-socket\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-dv8l2\" (UID: \"a0d8f2a6-a6b5-44dd-99fb-38aef41899e4\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-dv8l2" Oct 29 22:53:23.159807 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:23.159791 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0d8f2a6-a6b5-44dd-99fb-38aef41899e4-istiod-ca-cert\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-dv8l2\" (UID: \"a0d8f2a6-a6b5-44dd-99fb-38aef41899e4\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-dv8l2" Oct 29 22:53:23.161104 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:23.161084 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0d8f2a6-a6b5-44dd-99fb-38aef41899e4-istio-podinfo\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-dv8l2\" (UID: \"a0d8f2a6-a6b5-44dd-99fb-38aef41899e4\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-dv8l2" Oct 29 22:53:23.161405 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:23.161317 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0d8f2a6-a6b5-44dd-99fb-38aef41899e4-istio-envoy\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-dv8l2\" (UID: \"a0d8f2a6-a6b5-44dd-99fb-38aef41899e4\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-dv8l2" Oct 29 22:53:23.180559 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:23.180540 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0d8f2a6-a6b5-44dd-99fb-38aef41899e4-istio-token\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-dv8l2\" (UID: \"a0d8f2a6-a6b5-44dd-99fb-38aef41899e4\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-dv8l2" Oct 29 22:53:23.180559 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:23.180553 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxjrb\" (UniqueName: \"kubernetes.io/projected/a0d8f2a6-a6b5-44dd-99fb-38aef41899e4-kube-api-access-pxjrb\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-dv8l2\" (UID: \"a0d8f2a6-a6b5-44dd-99fb-38aef41899e4\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-dv8l2" Oct 29 22:53:23.257445 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:23.257421 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-dv8l2" Oct 29 22:53:23.580344 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:23.580312 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-dv8l2"] Oct 29 22:53:23.582318 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:53:23.582294 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0d8f2a6_a6b5_44dd_99fb_38aef41899e4.slice/crio-7b7bdbf94b72dc8c9a557b42fa35cbc6d775153ad9dabe0b410d27f80094ab61 WatchSource:0}: Error finding container 7b7bdbf94b72dc8c9a557b42fa35cbc6d775153ad9dabe0b410d27f80094ab61: Status 404 returned error can't find the container with id 7b7bdbf94b72dc8c9a557b42fa35cbc6d775153ad9dabe0b410d27f80094ab61 Oct 29 22:53:23.585234 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:23.585195 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:53:23.585380 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:23.585277 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:53:23.585380 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:23.585320 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:53:23.791748 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:23.791715 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:53:24.055390 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:24.055340 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-dv8l2" event={"ID":"a0d8f2a6-a6b5-44dd-99fb-38aef41899e4","Type":"ContainerStarted","Data":"edb8f1cc17a92e3ed0cb999632507fc77704b3a8071b599d7aae9050d179ddf2"} Oct 29 22:53:24.055594 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:24.055407 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-dv8l2" event={"ID":"a0d8f2a6-a6b5-44dd-99fb-38aef41899e4","Type":"ContainerStarted","Data":"7b7bdbf94b72dc8c9a557b42fa35cbc6d775153ad9dabe0b410d27f80094ab61"} Oct 29 22:53:24.103569 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:24.103502 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-dv8l2" podStartSLOduration=2.103483872 podStartE2EDuration="2.103483872s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:53:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:53:24.099617358 +0000 UTC m=+1184.882830784" watchObservedRunningTime="2026-10-29 22:53:24.103483872 +0000 UTC m=+1184.886697304" Oct 29 22:53:24.258385 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:24.258352 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-dv8l2" Oct 29 22:53:24.259781 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:24.259753 5057 patch_prober.go:29] interesting pod/e2e-worker-w5-b2921b-openshift-default-5547c88fff-dv8l2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Oct 29 22:53:24.259916 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:24.259807 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-dv8l2" podUID="a0d8f2a6-a6b5-44dd-99fb-38aef41899e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Oct 29 22:53:25.258040 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:25.258005 5057 patch_prober.go:29] interesting pod/e2e-worker-w5-b2921b-openshift-default-5547c88fff-dv8l2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Oct 29 22:53:25.258449 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:25.258054 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-dv8l2" podUID="a0d8f2a6-a6b5-44dd-99fb-38aef41899e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Oct 29 22:53:25.522400 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:25.521832 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:53:26.258363 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:26.258315 5057 patch_prober.go:29] interesting pod/e2e-worker-w5-b2921b-openshift-default-5547c88fff-dv8l2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Oct 29 22:53:26.258744 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:26.258380 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-dv8l2" podUID="a0d8f2a6-a6b5-44dd-99fb-38aef41899e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Oct 29 22:53:26.918762 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:26.918717 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:53:27.258691 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:27.258652 5057 patch_prober.go:29] interesting pod/e2e-worker-w5-b2921b-openshift-default-5547c88fff-dv8l2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Oct 29 22:53:27.259177 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:27.258715 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-dv8l2" podUID="a0d8f2a6-a6b5-44dd-99fb-38aef41899e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Oct 29 22:53:28.258077 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:28.258040 5057 patch_prober.go:29] interesting pod/e2e-worker-w5-b2921b-openshift-default-5547c88fff-dv8l2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Oct 29 22:53:28.258269 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:28.258090 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-dv8l2" podUID="a0d8f2a6-a6b5-44dd-99fb-38aef41899e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Oct 29 22:53:28.386935 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:28.386903 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:53:29.257921 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:29.257883 5057 patch_prober.go:29] interesting pod/e2e-worker-w5-b2921b-openshift-default-5547c88fff-dv8l2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Oct 29 22:53:29.258101 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:29.257945 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-dv8l2" podUID="a0d8f2a6-a6b5-44dd-99fb-38aef41899e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Oct 29 22:53:29.651461 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:29.651376 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-4hl56"] Oct 29 22:53:29.656063 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:29.656038 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-4hl56" Oct 29 22:53:29.683598 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:29.683568 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-4hl56"] Oct 29 22:53:29.757462 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:29.757415 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9qpj\" (UniqueName: \"kubernetes.io/projected/d6a73f9e-f3a4-4ed8-ac2e-ff657d266514-kube-api-access-f9qpj\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-4hl56\" (UID: \"d6a73f9e-f3a4-4ed8-ac2e-ff657d266514\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-4hl56" Oct 29 22:53:29.757637 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:29.757485 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6a73f9e-f3a4-4ed8-ac2e-ff657d266514-istio-data\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-4hl56\" (UID: \"d6a73f9e-f3a4-4ed8-ac2e-ff657d266514\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-4hl56" Oct 29 22:53:29.757637 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:29.757578 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6a73f9e-f3a4-4ed8-ac2e-ff657d266514-workload-certs\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-4hl56\" (UID: \"d6a73f9e-f3a4-4ed8-ac2e-ff657d266514\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-4hl56" Oct 29 22:53:29.757721 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:29.757688 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6a73f9e-f3a4-4ed8-ac2e-ff657d266514-istio-token\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-4hl56\" (UID: \"d6a73f9e-f3a4-4ed8-ac2e-ff657d266514\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-4hl56" Oct 29 22:53:29.757762 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:29.757732 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6a73f9e-f3a4-4ed8-ac2e-ff657d266514-credential-socket\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-4hl56\" (UID: \"d6a73f9e-f3a4-4ed8-ac2e-ff657d266514\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-4hl56" Oct 29 22:53:29.757805 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:29.757767 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6a73f9e-f3a4-4ed8-ac2e-ff657d266514-istio-envoy\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-4hl56\" (UID: \"d6a73f9e-f3a4-4ed8-ac2e-ff657d266514\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-4hl56" Oct 29 22:53:29.757805 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:29.757795 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6a73f9e-f3a4-4ed8-ac2e-ff657d266514-istio-podinfo\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-4hl56\" (UID: \"d6a73f9e-f3a4-4ed8-ac2e-ff657d266514\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-4hl56" Oct 29 22:53:29.757878 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:29.757844 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6a73f9e-f3a4-4ed8-ac2e-ff657d266514-istiod-ca-cert\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-4hl56\" (UID: \"d6a73f9e-f3a4-4ed8-ac2e-ff657d266514\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-4hl56" Oct 29 22:53:29.757923 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:29.757912 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6a73f9e-f3a4-4ed8-ac2e-ff657d266514-workload-socket\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-4hl56\" (UID: \"d6a73f9e-f3a4-4ed8-ac2e-ff657d266514\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-4hl56" Oct 29 22:53:29.858943 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:29.858910 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6a73f9e-f3a4-4ed8-ac2e-ff657d266514-istio-token\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-4hl56\" (UID: \"d6a73f9e-f3a4-4ed8-ac2e-ff657d266514\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-4hl56" Oct 29 22:53:29.858943 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:29.858955 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6a73f9e-f3a4-4ed8-ac2e-ff657d266514-credential-socket\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-4hl56\" (UID: \"d6a73f9e-f3a4-4ed8-ac2e-ff657d266514\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-4hl56" Oct 29 22:53:29.859187 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:29.858982 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6a73f9e-f3a4-4ed8-ac2e-ff657d266514-istio-envoy\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-4hl56\" (UID: \"d6a73f9e-f3a4-4ed8-ac2e-ff657d266514\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-4hl56" Oct 29 22:53:29.859187 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:29.859009 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6a73f9e-f3a4-4ed8-ac2e-ff657d266514-istio-podinfo\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-4hl56\" (UID: \"d6a73f9e-f3a4-4ed8-ac2e-ff657d266514\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-4hl56" Oct 29 22:53:29.859187 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:29.859074 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6a73f9e-f3a4-4ed8-ac2e-ff657d266514-istiod-ca-cert\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-4hl56\" (UID: \"d6a73f9e-f3a4-4ed8-ac2e-ff657d266514\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-4hl56" Oct 29 22:53:29.859187 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:29.859116 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6a73f9e-f3a4-4ed8-ac2e-ff657d266514-workload-socket\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-4hl56\" (UID: \"d6a73f9e-f3a4-4ed8-ac2e-ff657d266514\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-4hl56" Oct 29 22:53:29.859434 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:29.859197 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9qpj\" (UniqueName: \"kubernetes.io/projected/d6a73f9e-f3a4-4ed8-ac2e-ff657d266514-kube-api-access-f9qpj\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-4hl56\" (UID: \"d6a73f9e-f3a4-4ed8-ac2e-ff657d266514\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-4hl56" Oct 29 22:53:29.859434 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:29.859262 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6a73f9e-f3a4-4ed8-ac2e-ff657d266514-istio-data\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-4hl56\" (UID: \"d6a73f9e-f3a4-4ed8-ac2e-ff657d266514\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-4hl56" Oct 29 22:53:29.859434 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:29.859316 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6a73f9e-f3a4-4ed8-ac2e-ff657d266514-workload-certs\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-4hl56\" (UID: \"d6a73f9e-f3a4-4ed8-ac2e-ff657d266514\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-4hl56" Oct 29 22:53:29.859641 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:29.859590 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6a73f9e-f3a4-4ed8-ac2e-ff657d266514-workload-socket\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-4hl56\" (UID: \"d6a73f9e-f3a4-4ed8-ac2e-ff657d266514\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-4hl56" Oct 29 22:53:29.859797 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:29.859771 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6a73f9e-f3a4-4ed8-ac2e-ff657d266514-workload-certs\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-4hl56\" (UID: \"d6a73f9e-f3a4-4ed8-ac2e-ff657d266514\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-4hl56" Oct 29 22:53:29.859903 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:29.859882 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6a73f9e-f3a4-4ed8-ac2e-ff657d266514-credential-socket\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-4hl56\" (UID: \"d6a73f9e-f3a4-4ed8-ac2e-ff657d266514\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-4hl56" Oct 29 22:53:29.859968 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:29.859882 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6a73f9e-f3a4-4ed8-ac2e-ff657d266514-istio-data\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-4hl56\" (UID: \"d6a73f9e-f3a4-4ed8-ac2e-ff657d266514\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-4hl56" Oct 29 22:53:29.860429 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:29.860402 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6a73f9e-f3a4-4ed8-ac2e-ff657d266514-istiod-ca-cert\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-4hl56\" (UID: \"d6a73f9e-f3a4-4ed8-ac2e-ff657d266514\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-4hl56" Oct 29 22:53:29.861907 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:29.861864 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6a73f9e-f3a4-4ed8-ac2e-ff657d266514-istio-podinfo\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-4hl56\" (UID: \"d6a73f9e-f3a4-4ed8-ac2e-ff657d266514\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-4hl56" Oct 29 22:53:29.862016 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:29.862002 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6a73f9e-f3a4-4ed8-ac2e-ff657d266514-istio-envoy\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-4hl56\" (UID: \"d6a73f9e-f3a4-4ed8-ac2e-ff657d266514\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-4hl56" Oct 29 22:53:29.884188 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:29.884156 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6a73f9e-f3a4-4ed8-ac2e-ff657d266514-istio-token\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-4hl56\" (UID: \"d6a73f9e-f3a4-4ed8-ac2e-ff657d266514\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-4hl56" Oct 29 22:53:29.885214 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:29.885192 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9qpj\" (UniqueName: \"kubernetes.io/projected/d6a73f9e-f3a4-4ed8-ac2e-ff657d266514-kube-api-access-f9qpj\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-4hl56\" (UID: \"d6a73f9e-f3a4-4ed8-ac2e-ff657d266514\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-4hl56" Oct 29 22:53:29.971752 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:29.971715 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-4hl56" Oct 29 22:53:30.021069 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:30.021014 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:53:30.315052 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:30.315021 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-4hl56"] Oct 29 22:53:30.316277 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:53:30.316244 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6a73f9e_f3a4_4ed8_ac2e_ff657d266514.slice/crio-e8a14577541f109aca0405ac3de2c523839a00b4921b8bfd61c9793cd1299048 WatchSource:0}: Error finding container e8a14577541f109aca0405ac3de2c523839a00b4921b8bfd61c9793cd1299048: Status 404 returned error can't find the container with id e8a14577541f109aca0405ac3de2c523839a00b4921b8bfd61c9793cd1299048 Oct 29 22:53:30.318789 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:30.318758 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:53:30.318915 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:30.318820 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:53:30.318915 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:30.318849 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:53:31.044527 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:31.044497 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-dv8l2" Oct 29 22:53:31.047051 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:31.044963 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-dv8l2" Oct 29 22:53:31.047051 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:31.045919 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-dv8l2" Oct 29 22:53:31.129126 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:31.129091 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-4hl56" event={"ID":"d6a73f9e-f3a4-4ed8-ac2e-ff657d266514","Type":"ContainerStarted","Data":"94a64970649f984d97dda6e5464a9ac044586483af1dcaf6d5805691d5f33105"} Oct 29 22:53:31.129316 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:31.129144 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-4hl56" event={"ID":"d6a73f9e-f3a4-4ed8-ac2e-ff657d266514","Type":"ContainerStarted","Data":"e8a14577541f109aca0405ac3de2c523839a00b4921b8bfd61c9793cd1299048"} Oct 29 22:53:31.211863 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:31.211831 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:53:31.212511 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:31.212441 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-4hl56" podStartSLOduration=2.212423735 podStartE2EDuration="2.212423735s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:53:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:53:31.175650603 +0000 UTC m=+1191.958864030" watchObservedRunningTime="2026-10-29 22:53:31.212423735 +0000 UTC m=+1191.995637161" Oct 29 22:53:31.627979 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:31.627935 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:53:31.972673 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:31.972641 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-4hl56" Oct 29 22:53:31.974725 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:31.974700 5057 patch_prober.go:29] interesting pod/e2e-worker-w1-e131af-openshift-default-c476c699c-4hl56 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Oct 29 22:53:31.974850 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:31.974751 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-4hl56" podUID="d6a73f9e-f3a4-4ed8-ac2e-ff657d266514" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Oct 29 22:53:32.972963 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:32.972930 5057 patch_prober.go:29] interesting pod/e2e-worker-w1-e131af-openshift-default-c476c699c-4hl56 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Oct 29 22:53:32.973359 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:32.972991 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-4hl56" podUID="d6a73f9e-f3a4-4ed8-ac2e-ff657d266514" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Oct 29 22:53:33.109577 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:33.109539 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:53:33.605866 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:33.605831 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:53:33.972493 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:33.972460 5057 patch_prober.go:29] interesting pod/e2e-worker-w1-e131af-openshift-default-c476c699c-4hl56 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Oct 29 22:53:33.972653 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:33.972521 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-4hl56" podUID="d6a73f9e-f3a4-4ed8-ac2e-ff657d266514" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Oct 29 22:53:34.827186 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:34.827151 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:53:34.972054 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:34.972018 5057 patch_prober.go:29] interesting pod/e2e-worker-w1-e131af-openshift-default-c476c699c-4hl56 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Oct 29 22:53:34.972216 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:34.972075 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-4hl56" podUID="d6a73f9e-f3a4-4ed8-ac2e-ff657d266514" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Oct 29 22:53:35.972799 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:35.972765 5057 patch_prober.go:29] interesting pod/e2e-worker-w1-e131af-openshift-default-c476c699c-4hl56 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Oct 29 22:53:35.973192 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:35.972819 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-4hl56" podUID="d6a73f9e-f3a4-4ed8-ac2e-ff657d266514" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Oct 29 22:53:36.435545 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:36.435462 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:53:37.338532 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:37.338448 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-4hl56" Oct 29 22:53:37.339212 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:37.339190 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-4hl56" Oct 29 22:53:37.339776 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:37.339753 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-4hl56" Oct 29 22:53:37.858207 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:37.858175 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:53:37.968748 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:37.968717 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:53:39.484914 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:39.484882 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-vj6md_aa6a190e-3283-458a-9dfd-2de7cb193fc5/console-operator/2.log" Oct 29 22:53:39.485556 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:39.485515 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:53:39.492568 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:39.492547 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-vj6md_aa6a190e-3283-458a-9dfd-2de7cb193fc5/console-operator/2.log" Oct 29 22:53:40.182184 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:40.182148 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v2tfm"] Oct 29 22:53:41.093690 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:41.093655 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:53:41.643090 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:41.643050 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:53:42.565426 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:42.565390 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:53:42.673113 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:42.673080 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:53:44.080673 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:44.080639 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:53:45.582560 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:45.582522 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:53:47.060786 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:47.060753 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:53:48.446921 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:48.446886 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:53:49.027518 ip-10-0-140-74 kubenswrapper[5057]: E1029 22:53:49.027479 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d2dnn" containerName="extractor" Oct 29 22:53:49.971572 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:49.971531 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:53:51.445009 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:51.444907 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:53:53.101865 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:53.101821 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:53:53.433951 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:53.433860 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-08e15d/distinct-w0-08e15d-kserve-85695c78bb-p67gk"] Oct 29 22:53:53.440207 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:53.440183 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-08e15d/distinct-w0-08e15d-kserve-85695c78bb-p67gk" Oct 29 22:53:53.458146 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:53.458116 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-08e15d/distinct-w0-08e15d-kserve-85695c78bb-p67gk"] Oct 29 22:53:53.463906 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:53.462928 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-08e15d\"/\"default-dockercfg-246bt\"" Oct 29 22:53:53.463906 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:53.462971 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-08e15d\"/\"openshift-service-ca.crt\"" Oct 29 22:53:53.463906 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:53.462974 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-08e15d\"/\"kube-root-ca.crt\"" Oct 29 22:53:53.464111 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:53.463914 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-08e15d\"/\"distinct-w0-08e15d-kserve-self-signed-certs\"" Oct 29 22:53:53.481768 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:53.481735 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7rc9\" (UniqueName: \"kubernetes.io/projected/e664f012-3b80-4a8a-ac19-d767d92c0a33-kube-api-access-v7rc9\") pod \"distinct-w0-08e15d-kserve-85695c78bb-p67gk\" (UID: \"e664f012-3b80-4a8a-ac19-d767d92c0a33\") " pod="e2e-models-e2e-worker-w0-08e15d/distinct-w0-08e15d-kserve-85695c78bb-p67gk" Oct 29 22:53:53.481885 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:53.481801 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e664f012-3b80-4a8a-ac19-d767d92c0a33-dshm\") pod \"distinct-w0-08e15d-kserve-85695c78bb-p67gk\" (UID: \"e664f012-3b80-4a8a-ac19-d767d92c0a33\") " pod="e2e-models-e2e-worker-w0-08e15d/distinct-w0-08e15d-kserve-85695c78bb-p67gk" Oct 29 22:53:53.481885 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:53.481856 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e664f012-3b80-4a8a-ac19-d767d92c0a33-tls-certs\") pod \"distinct-w0-08e15d-kserve-85695c78bb-p67gk\" (UID: \"e664f012-3b80-4a8a-ac19-d767d92c0a33\") " pod="e2e-models-e2e-worker-w0-08e15d/distinct-w0-08e15d-kserve-85695c78bb-p67gk" Oct 29 22:53:53.482118 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:53.482097 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e664f012-3b80-4a8a-ac19-d767d92c0a33-model-cache\") pod \"distinct-w0-08e15d-kserve-85695c78bb-p67gk\" (UID: \"e664f012-3b80-4a8a-ac19-d767d92c0a33\") " pod="e2e-models-e2e-worker-w0-08e15d/distinct-w0-08e15d-kserve-85695c78bb-p67gk" Oct 29 22:53:53.482179 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:53.482148 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e664f012-3b80-4a8a-ac19-d767d92c0a33-home\") pod \"distinct-w0-08e15d-kserve-85695c78bb-p67gk\" (UID: \"e664f012-3b80-4a8a-ac19-d767d92c0a33\") " pod="e2e-models-e2e-worker-w0-08e15d/distinct-w0-08e15d-kserve-85695c78bb-p67gk" Oct 29 22:53:53.482288 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:53.482268 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e664f012-3b80-4a8a-ac19-d767d92c0a33-tmp-dir\") pod \"distinct-w0-08e15d-kserve-85695c78bb-p67gk\" (UID: \"e664f012-3b80-4a8a-ac19-d767d92c0a33\") " pod="e2e-models-e2e-worker-w0-08e15d/distinct-w0-08e15d-kserve-85695c78bb-p67gk" Oct 29 22:53:53.498546 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:53.498521 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:53:53.583904 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:53.583872 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e664f012-3b80-4a8a-ac19-d767d92c0a33-model-cache\") pod \"distinct-w0-08e15d-kserve-85695c78bb-p67gk\" (UID: \"e664f012-3b80-4a8a-ac19-d767d92c0a33\") " pod="e2e-models-e2e-worker-w0-08e15d/distinct-w0-08e15d-kserve-85695c78bb-p67gk" Oct 29 22:53:53.584040 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:53.583914 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e664f012-3b80-4a8a-ac19-d767d92c0a33-home\") pod \"distinct-w0-08e15d-kserve-85695c78bb-p67gk\" (UID: \"e664f012-3b80-4a8a-ac19-d767d92c0a33\") " pod="e2e-models-e2e-worker-w0-08e15d/distinct-w0-08e15d-kserve-85695c78bb-p67gk" Oct 29 22:53:53.584095 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:53.584032 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e664f012-3b80-4a8a-ac19-d767d92c0a33-tmp-dir\") pod \"distinct-w0-08e15d-kserve-85695c78bb-p67gk\" (UID: \"e664f012-3b80-4a8a-ac19-d767d92c0a33\") " pod="e2e-models-e2e-worker-w0-08e15d/distinct-w0-08e15d-kserve-85695c78bb-p67gk" Oct 29 22:53:53.584166 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:53.584150 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7rc9\" (UniqueName: \"kubernetes.io/projected/e664f012-3b80-4a8a-ac19-d767d92c0a33-kube-api-access-v7rc9\") pod \"distinct-w0-08e15d-kserve-85695c78bb-p67gk\" (UID: \"e664f012-3b80-4a8a-ac19-d767d92c0a33\") " pod="e2e-models-e2e-worker-w0-08e15d/distinct-w0-08e15d-kserve-85695c78bb-p67gk" Oct 29 22:53:53.584228 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:53.584189 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e664f012-3b80-4a8a-ac19-d767d92c0a33-dshm\") pod \"distinct-w0-08e15d-kserve-85695c78bb-p67gk\" (UID: \"e664f012-3b80-4a8a-ac19-d767d92c0a33\") " pod="e2e-models-e2e-worker-w0-08e15d/distinct-w0-08e15d-kserve-85695c78bb-p67gk" Oct 29 22:53:53.584228 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:53.584208 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e664f012-3b80-4a8a-ac19-d767d92c0a33-tls-certs\") pod \"distinct-w0-08e15d-kserve-85695c78bb-p67gk\" (UID: \"e664f012-3b80-4a8a-ac19-d767d92c0a33\") " pod="e2e-models-e2e-worker-w0-08e15d/distinct-w0-08e15d-kserve-85695c78bb-p67gk" Oct 29 22:53:53.584379 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:53.584316 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e664f012-3b80-4a8a-ac19-d767d92c0a33-model-cache\") pod \"distinct-w0-08e15d-kserve-85695c78bb-p67gk\" (UID: \"e664f012-3b80-4a8a-ac19-d767d92c0a33\") " pod="e2e-models-e2e-worker-w0-08e15d/distinct-w0-08e15d-kserve-85695c78bb-p67gk" Oct 29 22:53:53.584451 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:53.584375 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e664f012-3b80-4a8a-ac19-d767d92c0a33-home\") pod \"distinct-w0-08e15d-kserve-85695c78bb-p67gk\" (UID: \"e664f012-3b80-4a8a-ac19-d767d92c0a33\") " pod="e2e-models-e2e-worker-w0-08e15d/distinct-w0-08e15d-kserve-85695c78bb-p67gk" Oct 29 22:53:53.584451 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:53.584420 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e664f012-3b80-4a8a-ac19-d767d92c0a33-tmp-dir\") pod \"distinct-w0-08e15d-kserve-85695c78bb-p67gk\" (UID: \"e664f012-3b80-4a8a-ac19-d767d92c0a33\") " pod="e2e-models-e2e-worker-w0-08e15d/distinct-w0-08e15d-kserve-85695c78bb-p67gk" Oct 29 22:53:53.586769 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:53.586747 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e664f012-3b80-4a8a-ac19-d767d92c0a33-dshm\") pod \"distinct-w0-08e15d-kserve-85695c78bb-p67gk\" (UID: \"e664f012-3b80-4a8a-ac19-d767d92c0a33\") " pod="e2e-models-e2e-worker-w0-08e15d/distinct-w0-08e15d-kserve-85695c78bb-p67gk" Oct 29 22:53:53.586869 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:53.586855 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e664f012-3b80-4a8a-ac19-d767d92c0a33-tls-certs\") pod \"distinct-w0-08e15d-kserve-85695c78bb-p67gk\" (UID: \"e664f012-3b80-4a8a-ac19-d767d92c0a33\") " pod="e2e-models-e2e-worker-w0-08e15d/distinct-w0-08e15d-kserve-85695c78bb-p67gk" Oct 29 22:53:53.605609 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:53.605580 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7rc9\" (UniqueName: \"kubernetes.io/projected/e664f012-3b80-4a8a-ac19-d767d92c0a33-kube-api-access-v7rc9\") pod \"distinct-w0-08e15d-kserve-85695c78bb-p67gk\" (UID: \"e664f012-3b80-4a8a-ac19-d767d92c0a33\") " pod="e2e-models-e2e-worker-w0-08e15d/distinct-w0-08e15d-kserve-85695c78bb-p67gk" Oct 29 22:53:53.763424 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:53.763397 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-08e15d/distinct-w0-08e15d-kserve-85695c78bb-p67gk" Oct 29 22:53:54.074521 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:54.074495 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-08e15d/distinct-w0-08e15d-kserve-85695c78bb-p67gk"] Oct 29 22:53:54.076041 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:53:54.076021 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode664f012_3b80_4a8a_ac19_d767d92c0a33.slice/crio-bd3bb9de238db28a11e1e2b25ce393dc7e441c105ec3d05a75e783dfc770afad WatchSource:0}: Error finding container bd3bb9de238db28a11e1e2b25ce393dc7e441c105ec3d05a75e783dfc770afad: Status 404 returned error can't find the container with id bd3bb9de238db28a11e1e2b25ce393dc7e441c105ec3d05a75e783dfc770afad Oct 29 22:53:54.302123 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:54.302082 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-08e15d/distinct-w0-08e15d-kserve-85695c78bb-p67gk" event={"ID":"e664f012-3b80-4a8a-ac19-d767d92c0a33","Type":"ContainerStarted","Data":"90918fdd096be65fde5ed90d07b1582c4de7dffa60bed26cc6a010a2bdaf19ba"} Oct 29 22:53:54.302123 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:54.302129 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-08e15d/distinct-w0-08e15d-kserve-85695c78bb-p67gk" event={"ID":"e664f012-3b80-4a8a-ac19-d767d92c0a33","Type":"ContainerStarted","Data":"bd3bb9de238db28a11e1e2b25ce393dc7e441c105ec3d05a75e783dfc770afad"} Oct 29 22:53:54.343480 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:54.343391 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-08e15d/distinct-w0-08e15d-kserve-85695c78bb-p67gk" podStartSLOduration=1.343376936 podStartE2EDuration="1.343376936s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:53:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:53:54.339711046 +0000 UTC m=+1215.122924473" watchObservedRunningTime="2026-10-29 22:53:54.343376936 +0000 UTC m=+1215.126590362" Oct 29 22:53:54.764068 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:54.764039 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-08e15d/distinct-w0-08e15d-kserve-85695c78bb-p67gk" Oct 29 22:53:54.894821 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:54.894787 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:53:55.002347 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:55.002295 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:53:56.547134 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:56.547099 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:53:57.663223 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:57.663187 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-31226c/distinct-w6-31226c-kserve-858d95fd76-bkf7c"] Oct 29 22:53:57.670617 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:57.670403 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:53:57.750878 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:57.750847 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-31226c/distinct-w6-31226c-kserve-858d95fd76-bkf7c"] Oct 29 22:53:57.762431 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:57.751154 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-31226c/distinct-w6-31226c-kserve-858d95fd76-bkf7c" Oct 29 22:53:57.782756 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:57.782535 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-31226c\"/\"distinct-w6-31226c-kserve-self-signed-certs\"" Oct 29 22:53:57.789353 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:57.787259 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-31226c\"/\"kube-root-ca.crt\"" Oct 29 22:53:57.789353 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:57.787416 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-31226c\"/\"default-dockercfg-hqr2k\"" Oct 29 22:53:57.789353 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:57.787484 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-31226c\"/\"openshift-service-ca.crt\"" Oct 29 22:53:57.871396 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:57.871320 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3fc11c6-03c9-4314-99c3-f857b149415c-tmp-dir\") pod \"distinct-w6-31226c-kserve-858d95fd76-bkf7c\" (UID: \"f3fc11c6-03c9-4314-99c3-f857b149415c\") " pod="e2e-models-e2e-worker-w6-31226c/distinct-w6-31226c-kserve-858d95fd76-bkf7c" Oct 29 22:53:57.871585 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:57.871448 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3fc11c6-03c9-4314-99c3-f857b149415c-model-cache\") pod \"distinct-w6-31226c-kserve-858d95fd76-bkf7c\" (UID: \"f3fc11c6-03c9-4314-99c3-f857b149415c\") " pod="e2e-models-e2e-worker-w6-31226c/distinct-w6-31226c-kserve-858d95fd76-bkf7c" Oct 29 22:53:57.871654 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:57.871619 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3fc11c6-03c9-4314-99c3-f857b149415c-tls-certs\") pod \"distinct-w6-31226c-kserve-858d95fd76-bkf7c\" (UID: \"f3fc11c6-03c9-4314-99c3-f857b149415c\") " pod="e2e-models-e2e-worker-w6-31226c/distinct-w6-31226c-kserve-858d95fd76-bkf7c" Oct 29 22:53:57.871749 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:57.871731 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3fc11c6-03c9-4314-99c3-f857b149415c-dshm\") pod \"distinct-w6-31226c-kserve-858d95fd76-bkf7c\" (UID: \"f3fc11c6-03c9-4314-99c3-f857b149415c\") " pod="e2e-models-e2e-worker-w6-31226c/distinct-w6-31226c-kserve-858d95fd76-bkf7c" Oct 29 22:53:57.871839 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:57.871817 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xfn5w\" (UniqueName: \"kubernetes.io/projected/f3fc11c6-03c9-4314-99c3-f857b149415c-kube-api-access-xfn5w\") pod \"distinct-w6-31226c-kserve-858d95fd76-bkf7c\" (UID: \"f3fc11c6-03c9-4314-99c3-f857b149415c\") " pod="e2e-models-e2e-worker-w6-31226c/distinct-w6-31226c-kserve-858d95fd76-bkf7c" Oct 29 22:53:57.871901 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:57.871869 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3fc11c6-03c9-4314-99c3-f857b149415c-home\") pod \"distinct-w6-31226c-kserve-858d95fd76-bkf7c\" (UID: \"f3fc11c6-03c9-4314-99c3-f857b149415c\") " pod="e2e-models-e2e-worker-w6-31226c/distinct-w6-31226c-kserve-858d95fd76-bkf7c" Oct 29 22:53:57.972832 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:57.972798 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3fc11c6-03c9-4314-99c3-f857b149415c-tls-certs\") pod \"distinct-w6-31226c-kserve-858d95fd76-bkf7c\" (UID: \"f3fc11c6-03c9-4314-99c3-f857b149415c\") " pod="e2e-models-e2e-worker-w6-31226c/distinct-w6-31226c-kserve-858d95fd76-bkf7c" Oct 29 22:53:57.973044 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:57.972912 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3fc11c6-03c9-4314-99c3-f857b149415c-dshm\") pod \"distinct-w6-31226c-kserve-858d95fd76-bkf7c\" (UID: \"f3fc11c6-03c9-4314-99c3-f857b149415c\") " pod="e2e-models-e2e-worker-w6-31226c/distinct-w6-31226c-kserve-858d95fd76-bkf7c" Oct 29 22:53:57.973044 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:57.972953 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xfn5w\" (UniqueName: \"kubernetes.io/projected/f3fc11c6-03c9-4314-99c3-f857b149415c-kube-api-access-xfn5w\") pod \"distinct-w6-31226c-kserve-858d95fd76-bkf7c\" (UID: \"f3fc11c6-03c9-4314-99c3-f857b149415c\") " pod="e2e-models-e2e-worker-w6-31226c/distinct-w6-31226c-kserve-858d95fd76-bkf7c" Oct 29 22:53:57.973044 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:57.972984 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3fc11c6-03c9-4314-99c3-f857b149415c-home\") pod \"distinct-w6-31226c-kserve-858d95fd76-bkf7c\" (UID: \"f3fc11c6-03c9-4314-99c3-f857b149415c\") " pod="e2e-models-e2e-worker-w6-31226c/distinct-w6-31226c-kserve-858d95fd76-bkf7c" Oct 29 22:53:57.973215 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:57.973047 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3fc11c6-03c9-4314-99c3-f857b149415c-tmp-dir\") pod \"distinct-w6-31226c-kserve-858d95fd76-bkf7c\" (UID: \"f3fc11c6-03c9-4314-99c3-f857b149415c\") " pod="e2e-models-e2e-worker-w6-31226c/distinct-w6-31226c-kserve-858d95fd76-bkf7c" Oct 29 22:53:57.973215 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:57.973069 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3fc11c6-03c9-4314-99c3-f857b149415c-model-cache\") pod \"distinct-w6-31226c-kserve-858d95fd76-bkf7c\" (UID: \"f3fc11c6-03c9-4314-99c3-f857b149415c\") " pod="e2e-models-e2e-worker-w6-31226c/distinct-w6-31226c-kserve-858d95fd76-bkf7c" Oct 29 22:53:57.973534 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:57.973506 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3fc11c6-03c9-4314-99c3-f857b149415c-tmp-dir\") pod \"distinct-w6-31226c-kserve-858d95fd76-bkf7c\" (UID: \"f3fc11c6-03c9-4314-99c3-f857b149415c\") " pod="e2e-models-e2e-worker-w6-31226c/distinct-w6-31226c-kserve-858d95fd76-bkf7c" Oct 29 22:53:57.973670 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:57.973542 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3fc11c6-03c9-4314-99c3-f857b149415c-home\") pod \"distinct-w6-31226c-kserve-858d95fd76-bkf7c\" (UID: \"f3fc11c6-03c9-4314-99c3-f857b149415c\") " pod="e2e-models-e2e-worker-w6-31226c/distinct-w6-31226c-kserve-858d95fd76-bkf7c" Oct 29 22:53:57.973670 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:57.973573 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3fc11c6-03c9-4314-99c3-f857b149415c-model-cache\") pod \"distinct-w6-31226c-kserve-858d95fd76-bkf7c\" (UID: \"f3fc11c6-03c9-4314-99c3-f857b149415c\") " pod="e2e-models-e2e-worker-w6-31226c/distinct-w6-31226c-kserve-858d95fd76-bkf7c" Oct 29 22:53:57.975202 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:57.975174 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3fc11c6-03c9-4314-99c3-f857b149415c-dshm\") pod \"distinct-w6-31226c-kserve-858d95fd76-bkf7c\" (UID: \"f3fc11c6-03c9-4314-99c3-f857b149415c\") " pod="e2e-models-e2e-worker-w6-31226c/distinct-w6-31226c-kserve-858d95fd76-bkf7c" Oct 29 22:53:57.975509 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:57.975489 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3fc11c6-03c9-4314-99c3-f857b149415c-tls-certs\") pod \"distinct-w6-31226c-kserve-858d95fd76-bkf7c\" (UID: \"f3fc11c6-03c9-4314-99c3-f857b149415c\") " pod="e2e-models-e2e-worker-w6-31226c/distinct-w6-31226c-kserve-858d95fd76-bkf7c" Oct 29 22:53:58.019351 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:58.008045 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xfn5w\" (UniqueName: \"kubernetes.io/projected/f3fc11c6-03c9-4314-99c3-f857b149415c-kube-api-access-xfn5w\") pod \"distinct-w6-31226c-kserve-858d95fd76-bkf7c\" (UID: \"f3fc11c6-03c9-4314-99c3-f857b149415c\") " pod="e2e-models-e2e-worker-w6-31226c/distinct-w6-31226c-kserve-858d95fd76-bkf7c" Oct 29 22:53:58.130843 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:58.130812 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-31226c/distinct-w6-31226c-kserve-858d95fd76-bkf7c" Oct 29 22:53:58.439566 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:58.439536 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-31226c/distinct-w6-31226c-kserve-858d95fd76-bkf7c"] Oct 29 22:53:58.440201 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:53:58.440170 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf3fc11c6_03c9_4314_99c3_f857b149415c.slice/crio-54d0af1252aa238c5e891f1c8dd85f8deb17d528e68f7710a3942f0999a4be22 WatchSource:0}: Error finding container 54d0af1252aa238c5e891f1c8dd85f8deb17d528e68f7710a3942f0999a4be22: Status 404 returned error can't find the container with id 54d0af1252aa238c5e891f1c8dd85f8deb17d528e68f7710a3942f0999a4be22 Oct 29 22:53:58.759535 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:58.759495 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-31226c/distinct-w6-31226c-kserve-858d95fd76-bkf7c" event={"ID":"f3fc11c6-03c9-4314-99c3-f857b149415c","Type":"ContainerStarted","Data":"ba7af072d51cf9d892a2aff973911964540cba2c84ebbb4916b5d0d2bd97e8f7"} Oct 29 22:53:58.759949 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:58.759557 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-31226c/distinct-w6-31226c-kserve-858d95fd76-bkf7c" event={"ID":"f3fc11c6-03c9-4314-99c3-f857b149415c","Type":"ContainerStarted","Data":"54d0af1252aa238c5e891f1c8dd85f8deb17d528e68f7710a3942f0999a4be22"} Oct 29 22:53:58.800157 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:58.800110 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-31226c/distinct-w6-31226c-kserve-858d95fd76-bkf7c" podStartSLOduration=1.800092896 podStartE2EDuration="1.800092896s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:53:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:53:58.798282338 +0000 UTC m=+1219.581495763" watchObservedRunningTime="2026-10-29 22:53:58.800092896 +0000 UTC m=+1219.583306323" Oct 29 22:53:59.131200 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:59.131128 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-31226c/distinct-w6-31226c-kserve-858d95fd76-bkf7c" Oct 29 22:53:59.171642 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:53:59.171614 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:54:00.693189 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:00.693154 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:54:02.170743 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:02.170709 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:54:03.709386 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:03.707628 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:54:03.764490 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:03.764458 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-08e15d/distinct-w0-08e15d-kserve-85695c78bb-p67gk" Oct 29 22:54:03.777698 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:03.777663 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-08e15d/distinct-w0-08e15d-kserve-85695c78bb-p67gk" Oct 29 22:54:03.813395 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:03.813359 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-08e15d/distinct-w0-08e15d-kserve-85695c78bb-p67gk" Oct 29 22:54:05.266712 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:05.266673 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:54:05.747462 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:05.747432 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:54:05.853765 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:05.853730 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:54:06.696672 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:06.696633 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:54:07.299594 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:07.299515 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v2tfm"] Oct 29 22:54:08.131145 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:08.131115 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-31226c/distinct-w6-31226c-kserve-858d95fd76-bkf7c" Oct 29 22:54:08.144493 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:08.144469 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-31226c/distinct-w6-31226c-kserve-858d95fd76-bkf7c" Oct 29 22:54:08.195898 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:08.195860 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:54:08.766004 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:08.765969 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:54:08.855720 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:08.855685 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-31226c/distinct-w6-31226c-kserve-858d95fd76-bkf7c" Oct 29 22:54:08.892880 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:08.892843 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:54:09.011321 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:09.011281 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:54:09.017899 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:09.017822 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-9nnj5"] Oct 29 22:54:09.024953 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:09.024927 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-9nnj5" Oct 29 22:54:09.040883 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:09.040864 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-17394a-openshift-default-dockercfg-x6t7f\"" Oct 29 22:54:09.048300 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:09.048274 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-9nnj5"] Oct 29 22:54:09.128512 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:09.128466 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c237fe02-4087-448d-b7b1-c7e03f1872c3-istio-podinfo\") pod \"e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-9nnj5\" (UID: \"c237fe02-4087-448d-b7b1-c7e03f1872c3\") " pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-9nnj5" Oct 29 22:54:09.128817 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:09.128610 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c237fe02-4087-448d-b7b1-c7e03f1872c3-istio-data\") pod \"e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-9nnj5\" (UID: \"c237fe02-4087-448d-b7b1-c7e03f1872c3\") " pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-9nnj5" Oct 29 22:54:09.128817 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:09.128640 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c237fe02-4087-448d-b7b1-c7e03f1872c3-istio-envoy\") pod \"e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-9nnj5\" (UID: \"c237fe02-4087-448d-b7b1-c7e03f1872c3\") " pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-9nnj5" Oct 29 22:54:09.128817 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:09.128755 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c237fe02-4087-448d-b7b1-c7e03f1872c3-istio-token\") pod \"e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-9nnj5\" (UID: \"c237fe02-4087-448d-b7b1-c7e03f1872c3\") " pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-9nnj5" Oct 29 22:54:09.129019 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:09.128826 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c237fe02-4087-448d-b7b1-c7e03f1872c3-credential-socket\") pod \"e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-9nnj5\" (UID: \"c237fe02-4087-448d-b7b1-c7e03f1872c3\") " pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-9nnj5" Oct 29 22:54:09.129019 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:09.128914 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c237fe02-4087-448d-b7b1-c7e03f1872c3-workload-socket\") pod \"e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-9nnj5\" (UID: \"c237fe02-4087-448d-b7b1-c7e03f1872c3\") " pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-9nnj5" Oct 29 22:54:09.129019 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:09.129010 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dfkbf\" (UniqueName: \"kubernetes.io/projected/c237fe02-4087-448d-b7b1-c7e03f1872c3-kube-api-access-dfkbf\") pod \"e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-9nnj5\" (UID: \"c237fe02-4087-448d-b7b1-c7e03f1872c3\") " pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-9nnj5" Oct 29 22:54:09.129188 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:09.129045 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c237fe02-4087-448d-b7b1-c7e03f1872c3-workload-certs\") pod \"e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-9nnj5\" (UID: \"c237fe02-4087-448d-b7b1-c7e03f1872c3\") " pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-9nnj5" Oct 29 22:54:09.129188 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:09.129079 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c237fe02-4087-448d-b7b1-c7e03f1872c3-istiod-ca-cert\") pod \"e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-9nnj5\" (UID: \"c237fe02-4087-448d-b7b1-c7e03f1872c3\") " pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-9nnj5" Oct 29 22:54:09.230845 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:09.230806 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c237fe02-4087-448d-b7b1-c7e03f1872c3-credential-socket\") pod \"e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-9nnj5\" (UID: \"c237fe02-4087-448d-b7b1-c7e03f1872c3\") " pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-9nnj5" Oct 29 22:54:09.230845 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:09.230852 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c237fe02-4087-448d-b7b1-c7e03f1872c3-workload-socket\") pod \"e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-9nnj5\" (UID: \"c237fe02-4087-448d-b7b1-c7e03f1872c3\") " pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-9nnj5" Oct 29 22:54:09.231405 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:09.230900 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfkbf\" (UniqueName: \"kubernetes.io/projected/c237fe02-4087-448d-b7b1-c7e03f1872c3-kube-api-access-dfkbf\") pod \"e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-9nnj5\" (UID: \"c237fe02-4087-448d-b7b1-c7e03f1872c3\") " pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-9nnj5" Oct 29 22:54:09.231405 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:09.230921 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c237fe02-4087-448d-b7b1-c7e03f1872c3-workload-certs\") pod \"e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-9nnj5\" (UID: \"c237fe02-4087-448d-b7b1-c7e03f1872c3\") " pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-9nnj5" Oct 29 22:54:09.231405 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:09.230941 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c237fe02-4087-448d-b7b1-c7e03f1872c3-istiod-ca-cert\") pod \"e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-9nnj5\" (UID: \"c237fe02-4087-448d-b7b1-c7e03f1872c3\") " pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-9nnj5" Oct 29 22:54:09.231405 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:09.231031 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c237fe02-4087-448d-b7b1-c7e03f1872c3-istio-podinfo\") pod \"e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-9nnj5\" (UID: \"c237fe02-4087-448d-b7b1-c7e03f1872c3\") " pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-9nnj5" Oct 29 22:54:09.231405 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:09.231136 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c237fe02-4087-448d-b7b1-c7e03f1872c3-istio-data\") pod \"e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-9nnj5\" (UID: \"c237fe02-4087-448d-b7b1-c7e03f1872c3\") " pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-9nnj5" Oct 29 22:54:09.231405 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:09.231169 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c237fe02-4087-448d-b7b1-c7e03f1872c3-istio-envoy\") pod \"e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-9nnj5\" (UID: \"c237fe02-4087-448d-b7b1-c7e03f1872c3\") " pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-9nnj5" Oct 29 22:54:09.231405 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:09.231233 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c237fe02-4087-448d-b7b1-c7e03f1872c3-istio-token\") pod \"e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-9nnj5\" (UID: \"c237fe02-4087-448d-b7b1-c7e03f1872c3\") " pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-9nnj5" Oct 29 22:54:09.231405 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:09.231376 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c237fe02-4087-448d-b7b1-c7e03f1872c3-workload-certs\") pod \"e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-9nnj5\" (UID: \"c237fe02-4087-448d-b7b1-c7e03f1872c3\") " pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-9nnj5" Oct 29 22:54:09.231405 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:09.231395 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c237fe02-4087-448d-b7b1-c7e03f1872c3-workload-socket\") pod \"e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-9nnj5\" (UID: \"c237fe02-4087-448d-b7b1-c7e03f1872c3\") " pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-9nnj5" Oct 29 22:54:09.231852 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:09.231586 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c237fe02-4087-448d-b7b1-c7e03f1872c3-istio-data\") pod \"e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-9nnj5\" (UID: \"c237fe02-4087-448d-b7b1-c7e03f1872c3\") " pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-9nnj5" Oct 29 22:54:09.231852 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:09.231714 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c237fe02-4087-448d-b7b1-c7e03f1872c3-istiod-ca-cert\") pod \"e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-9nnj5\" (UID: \"c237fe02-4087-448d-b7b1-c7e03f1872c3\") " pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-9nnj5" Oct 29 22:54:09.232024 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:09.232004 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c237fe02-4087-448d-b7b1-c7e03f1872c3-credential-socket\") pod \"e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-9nnj5\" (UID: \"c237fe02-4087-448d-b7b1-c7e03f1872c3\") " pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-9nnj5" Oct 29 22:54:09.233911 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:09.233866 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c237fe02-4087-448d-b7b1-c7e03f1872c3-istio-podinfo\") pod \"e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-9nnj5\" (UID: \"c237fe02-4087-448d-b7b1-c7e03f1872c3\") " pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-9nnj5" Oct 29 22:54:09.234161 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:09.234137 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c237fe02-4087-448d-b7b1-c7e03f1872c3-istio-envoy\") pod \"e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-9nnj5\" (UID: \"c237fe02-4087-448d-b7b1-c7e03f1872c3\") " pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-9nnj5" Oct 29 22:54:09.252756 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:09.252734 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c237fe02-4087-448d-b7b1-c7e03f1872c3-istio-token\") pod \"e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-9nnj5\" (UID: \"c237fe02-4087-448d-b7b1-c7e03f1872c3\") " pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-9nnj5" Oct 29 22:54:09.252913 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:09.252812 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfkbf\" (UniqueName: \"kubernetes.io/projected/c237fe02-4087-448d-b7b1-c7e03f1872c3-kube-api-access-dfkbf\") pod \"e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-9nnj5\" (UID: \"c237fe02-4087-448d-b7b1-c7e03f1872c3\") " pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-9nnj5" Oct 29 22:54:09.345301 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:09.345236 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-9nnj5" Oct 29 22:54:09.666052 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:09.666030 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-9nnj5"] Oct 29 22:54:09.667574 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:54:09.667551 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc237fe02_4087_448d_b7b1_c7e03f1872c3.slice/crio-ebc6668326b38613ad5c69a83cf5bc841a0f894a3f66d79c770822c3d53da637 WatchSource:0}: Error finding container ebc6668326b38613ad5c69a83cf5bc841a0f894a3f66d79c770822c3d53da637: Status 404 returned error can't find the container with id ebc6668326b38613ad5c69a83cf5bc841a0f894a3f66d79c770822c3d53da637 Oct 29 22:54:09.670093 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:09.670061 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:54:09.670178 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:09.670124 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:54:09.670178 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:09.670154 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:54:09.846088 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:09.846051 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-9nnj5" event={"ID":"c237fe02-4087-448d-b7b1-c7e03f1872c3","Type":"ContainerStarted","Data":"a365358bd2744c25f8081adcaeaf4f38c39c9b3d8fda8100cd1a6497beec85e5"} Oct 29 22:54:09.846254 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:09.846121 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-9nnj5" event={"ID":"c237fe02-4087-448d-b7b1-c7e03f1872c3","Type":"ContainerStarted","Data":"ebc6668326b38613ad5c69a83cf5bc841a0f894a3f66d79c770822c3d53da637"} Oct 29 22:54:09.888787 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:09.888728 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-9nnj5" podStartSLOduration=1.888709101 podStartE2EDuration="1.888709101s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:54:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:54:09.887850996 +0000 UTC m=+1230.671064434" watchObservedRunningTime="2026-10-29 22:54:09.888709101 +0000 UTC m=+1230.671922532" Oct 29 22:54:09.992007 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:09.991892 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:54:10.345828 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:10.345744 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-9nnj5" Oct 29 22:54:10.347457 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:10.347433 5057 patch_prober.go:29] interesting pod/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-9nnj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Oct 29 22:54:10.347571 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:10.347480 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-9nnj5" podUID="c237fe02-4087-448d-b7b1-c7e03f1872c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Oct 29 22:54:11.346339 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:11.346293 5057 patch_prober.go:29] interesting pod/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-9nnj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Oct 29 22:54:11.346805 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:11.346366 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-9nnj5" podUID="c237fe02-4087-448d-b7b1-c7e03f1872c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Oct 29 22:54:11.605429 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:11.604132 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:54:12.346267 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:12.346225 5057 patch_prober.go:29] interesting pod/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-9nnj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Oct 29 22:54:12.346463 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:12.346293 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-9nnj5" podUID="c237fe02-4087-448d-b7b1-c7e03f1872c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Oct 29 22:54:13.187783 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:13.187711 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:54:13.346396 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:13.346361 5057 patch_prober.go:29] interesting pod/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-9nnj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Oct 29 22:54:13.346536 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:13.346421 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-9nnj5" podUID="c237fe02-4087-448d-b7b1-c7e03f1872c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Oct 29 22:54:14.346538 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:14.346501 5057 patch_prober.go:29] interesting pod/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-9nnj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Oct 29 22:54:14.347056 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:14.347016 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-9nnj5" podUID="c237fe02-4087-448d-b7b1-c7e03f1872c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Oct 29 22:54:14.674829 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:14.674749 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:54:14.815077 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:14.815043 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-d6kwq"] Oct 29 22:54:14.831960 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:14.821862 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-d6kwq" Oct 29 22:54:14.832524 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:14.832497 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-d6kwq"] Oct 29 22:54:14.932862 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:14.932775 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73a11a50-d1bd-4eb5-8685-13e724462b24-workload-certs\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-d6kwq\" (UID: \"73a11a50-d1bd-4eb5-8685-13e724462b24\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-d6kwq" Oct 29 22:54:14.932862 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:14.932852 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4rll\" (UniqueName: \"kubernetes.io/projected/73a11a50-d1bd-4eb5-8685-13e724462b24-kube-api-access-g4rll\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-d6kwq\" (UID: \"73a11a50-d1bd-4eb5-8685-13e724462b24\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-d6kwq" Oct 29 22:54:14.933062 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:14.932912 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73a11a50-d1bd-4eb5-8685-13e724462b24-istiod-ca-cert\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-d6kwq\" (UID: \"73a11a50-d1bd-4eb5-8685-13e724462b24\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-d6kwq" Oct 29 22:54:14.933062 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:14.932947 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73a11a50-d1bd-4eb5-8685-13e724462b24-workload-socket\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-d6kwq\" (UID: \"73a11a50-d1bd-4eb5-8685-13e724462b24\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-d6kwq" Oct 29 22:54:14.933062 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:14.932963 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73a11a50-d1bd-4eb5-8685-13e724462b24-istio-envoy\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-d6kwq\" (UID: \"73a11a50-d1bd-4eb5-8685-13e724462b24\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-d6kwq" Oct 29 22:54:14.933062 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:14.932987 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73a11a50-d1bd-4eb5-8685-13e724462b24-istio-podinfo\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-d6kwq\" (UID: \"73a11a50-d1bd-4eb5-8685-13e724462b24\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-d6kwq" Oct 29 22:54:14.933196 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:14.933182 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73a11a50-d1bd-4eb5-8685-13e724462b24-istio-token\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-d6kwq\" (UID: \"73a11a50-d1bd-4eb5-8685-13e724462b24\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-d6kwq" Oct 29 22:54:14.933230 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:14.933201 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73a11a50-d1bd-4eb5-8685-13e724462b24-credential-socket\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-d6kwq\" (UID: \"73a11a50-d1bd-4eb5-8685-13e724462b24\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-d6kwq" Oct 29 22:54:14.933342 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:14.933318 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73a11a50-d1bd-4eb5-8685-13e724462b24-istio-data\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-d6kwq\" (UID: \"73a11a50-d1bd-4eb5-8685-13e724462b24\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-d6kwq" Oct 29 22:54:15.034178 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:15.034145 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73a11a50-d1bd-4eb5-8685-13e724462b24-istio-data\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-d6kwq\" (UID: \"73a11a50-d1bd-4eb5-8685-13e724462b24\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-d6kwq" Oct 29 22:54:15.034364 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:15.034189 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73a11a50-d1bd-4eb5-8685-13e724462b24-workload-certs\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-d6kwq\" (UID: \"73a11a50-d1bd-4eb5-8685-13e724462b24\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-d6kwq" Oct 29 22:54:15.034364 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:15.034232 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4rll\" (UniqueName: \"kubernetes.io/projected/73a11a50-d1bd-4eb5-8685-13e724462b24-kube-api-access-g4rll\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-d6kwq\" (UID: \"73a11a50-d1bd-4eb5-8685-13e724462b24\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-d6kwq" Oct 29 22:54:15.034364 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:15.034262 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73a11a50-d1bd-4eb5-8685-13e724462b24-istiod-ca-cert\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-d6kwq\" (UID: \"73a11a50-d1bd-4eb5-8685-13e724462b24\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-d6kwq" Oct 29 22:54:15.034364 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:15.034281 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73a11a50-d1bd-4eb5-8685-13e724462b24-workload-socket\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-d6kwq\" (UID: \"73a11a50-d1bd-4eb5-8685-13e724462b24\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-d6kwq" Oct 29 22:54:15.034364 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:15.034298 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73a11a50-d1bd-4eb5-8685-13e724462b24-istio-envoy\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-d6kwq\" (UID: \"73a11a50-d1bd-4eb5-8685-13e724462b24\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-d6kwq" Oct 29 22:54:15.034364 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:15.034323 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73a11a50-d1bd-4eb5-8685-13e724462b24-istio-podinfo\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-d6kwq\" (UID: \"73a11a50-d1bd-4eb5-8685-13e724462b24\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-d6kwq" Oct 29 22:54:15.034676 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:15.034492 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73a11a50-d1bd-4eb5-8685-13e724462b24-istio-token\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-d6kwq\" (UID: \"73a11a50-d1bd-4eb5-8685-13e724462b24\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-d6kwq" Oct 29 22:54:15.034676 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:15.034520 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73a11a50-d1bd-4eb5-8685-13e724462b24-credential-socket\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-d6kwq\" (UID: \"73a11a50-d1bd-4eb5-8685-13e724462b24\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-d6kwq" Oct 29 22:54:15.034676 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:15.034548 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73a11a50-d1bd-4eb5-8685-13e724462b24-istio-data\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-d6kwq\" (UID: \"73a11a50-d1bd-4eb5-8685-13e724462b24\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-d6kwq" Oct 29 22:54:15.034676 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:15.034612 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73a11a50-d1bd-4eb5-8685-13e724462b24-workload-certs\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-d6kwq\" (UID: \"73a11a50-d1bd-4eb5-8685-13e724462b24\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-d6kwq" Oct 29 22:54:15.034900 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:15.034854 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73a11a50-d1bd-4eb5-8685-13e724462b24-credential-socket\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-d6kwq\" (UID: \"73a11a50-d1bd-4eb5-8685-13e724462b24\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-d6kwq" Oct 29 22:54:15.034958 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:15.034944 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73a11a50-d1bd-4eb5-8685-13e724462b24-workload-socket\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-d6kwq\" (UID: \"73a11a50-d1bd-4eb5-8685-13e724462b24\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-d6kwq" Oct 29 22:54:15.035107 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:15.035089 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73a11a50-d1bd-4eb5-8685-13e724462b24-istiod-ca-cert\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-d6kwq\" (UID: \"73a11a50-d1bd-4eb5-8685-13e724462b24\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-d6kwq" Oct 29 22:54:15.036794 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:15.036769 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73a11a50-d1bd-4eb5-8685-13e724462b24-istio-envoy\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-d6kwq\" (UID: \"73a11a50-d1bd-4eb5-8685-13e724462b24\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-d6kwq" Oct 29 22:54:15.037048 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:15.037030 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73a11a50-d1bd-4eb5-8685-13e724462b24-istio-podinfo\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-d6kwq\" (UID: \"73a11a50-d1bd-4eb5-8685-13e724462b24\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-d6kwq" Oct 29 22:54:15.055923 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:15.055900 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73a11a50-d1bd-4eb5-8685-13e724462b24-istio-token\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-d6kwq\" (UID: \"73a11a50-d1bd-4eb5-8685-13e724462b24\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-d6kwq" Oct 29 22:54:15.056233 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:15.056208 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4rll\" (UniqueName: \"kubernetes.io/projected/73a11a50-d1bd-4eb5-8685-13e724462b24-kube-api-access-g4rll\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-d6kwq\" (UID: \"73a11a50-d1bd-4eb5-8685-13e724462b24\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-d6kwq" Oct 29 22:54:15.159411 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:15.159379 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-d6kwq" Oct 29 22:54:15.346575 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:15.346536 5057 patch_prober.go:29] interesting pod/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-9nnj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Oct 29 22:54:15.346934 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:15.346595 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-9nnj5" podUID="c237fe02-4087-448d-b7b1-c7e03f1872c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Oct 29 22:54:15.527898 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:15.527793 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-d6kwq"] Oct 29 22:54:15.529710 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:54:15.529681 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73a11a50_d1bd_4eb5_8685_13e724462b24.slice/crio-3b155c9810fd51aa18d796a6172489415fd9795d42875892b87865685aa277d1 WatchSource:0}: Error finding container 3b155c9810fd51aa18d796a6172489415fd9795d42875892b87865685aa277d1: Status 404 returned error can't find the container with id 3b155c9810fd51aa18d796a6172489415fd9795d42875892b87865685aa277d1 Oct 29 22:54:15.532265 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:15.532227 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:54:15.532385 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:15.532294 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:54:15.532385 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:15.532366 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:54:15.923703 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:15.923612 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-d6kwq" event={"ID":"73a11a50-d1bd-4eb5-8685-13e724462b24","Type":"ContainerStarted","Data":"a811b8b4a0b8f411df2de3f13a231f5e31f54595ff9fd7f756502cb58143437b"} Oct 29 22:54:15.923703 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:15.923684 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-d6kwq" event={"ID":"73a11a50-d1bd-4eb5-8685-13e724462b24","Type":"ContainerStarted","Data":"3b155c9810fd51aa18d796a6172489415fd9795d42875892b87865685aa277d1"} Oct 29 22:54:15.989748 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:15.989680 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-d6kwq" podStartSLOduration=1.9896616740000002 podStartE2EDuration="1.989661674s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:54:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:54:15.974892635 +0000 UTC m=+1236.758106061" watchObservedRunningTime="2026-10-29 22:54:15.989661674 +0000 UTC m=+1236.772875101" Oct 29 22:54:16.098551 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:16.098516 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:54:16.160486 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:16.160452 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-d6kwq" Oct 29 22:54:16.162093 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:16.162061 5057 patch_prober.go:29] interesting pod/e2e-worker-w1-e131af-openshift-default-c476c699c-d6kwq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Oct 29 22:54:16.162225 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:16.162117 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-d6kwq" podUID="73a11a50-d1bd-4eb5-8685-13e724462b24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Oct 29 22:54:16.444495 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:16.444278 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:54:16.757040 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:16.757003 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-9nnj5" Oct 29 22:54:16.757403 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:16.757374 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-9nnj5" Oct 29 22:54:16.758529 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:16.758508 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-9nnj5" Oct 29 22:54:17.171402 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:17.170857 5057 patch_prober.go:29] interesting pod/e2e-worker-w1-e131af-openshift-default-c476c699c-d6kwq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Oct 29 22:54:17.171402 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:17.170913 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-d6kwq" podUID="73a11a50-d1bd-4eb5-8685-13e724462b24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Oct 29 22:54:17.817483 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:17.817423 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:54:18.160443 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:18.160361 5057 patch_prober.go:29] interesting pod/e2e-worker-w1-e131af-openshift-default-c476c699c-d6kwq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Oct 29 22:54:18.160443 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:18.160421 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-d6kwq" podUID="73a11a50-d1bd-4eb5-8685-13e724462b24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Oct 29 22:54:19.036865 ip-10-0-140-74 kubenswrapper[5057]: E1029 22:54:19.036825 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d2dnn" containerName="extractor" Oct 29 22:54:19.160036 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:19.160000 5057 patch_prober.go:29] interesting pod/e2e-worker-w1-e131af-openshift-default-c476c699c-d6kwq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Oct 29 22:54:19.160216 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:19.160084 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-d6kwq" podUID="73a11a50-d1bd-4eb5-8685-13e724462b24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Oct 29 22:54:19.432790 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:19.432605 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:54:20.159825 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:20.159788 5057 patch_prober.go:29] interesting pod/e2e-worker-w1-e131af-openshift-default-c476c699c-d6kwq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Oct 29 22:54:20.160238 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:20.159850 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-d6kwq" podUID="73a11a50-d1bd-4eb5-8685-13e724462b24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Oct 29 22:54:21.024376 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:21.024316 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:54:21.159981 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:21.159946 5057 patch_prober.go:29] interesting pod/e2e-worker-w1-e131af-openshift-default-c476c699c-d6kwq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Oct 29 22:54:21.160468 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:21.160006 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-d6kwq" podUID="73a11a50-d1bd-4eb5-8685-13e724462b24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Oct 29 22:54:22.160606 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:22.160569 5057 patch_prober.go:29] interesting pod/e2e-worker-w1-e131af-openshift-default-c476c699c-d6kwq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Oct 29 22:54:22.161015 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:22.160645 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-d6kwq" podUID="73a11a50-d1bd-4eb5-8685-13e724462b24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Oct 29 22:54:22.699982 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:22.699944 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:54:23.363661 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:23.363627 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-d6kwq" Oct 29 22:54:23.365011 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:23.364988 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-d6kwq" Oct 29 22:54:23.366251 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:23.366226 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-d6kwq" Oct 29 22:54:24.277951 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:24.277916 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:54:24.831415 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:24.831383 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-17394a/distinct-2-w3-17394a-kserve-6545c766c4-h4chf"] Oct 29 22:54:24.837824 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:24.837798 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-17394a/distinct-2-w3-17394a-kserve-6545c766c4-h4chf" Oct 29 22:54:24.861761 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:24.861734 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-17394a/distinct-2-w3-17394a-kserve-6545c766c4-h4chf"] Oct 29 22:54:24.861947 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:24.861928 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-17394a\"/\"openshift-service-ca.crt\"" Oct 29 22:54:24.862124 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:24.862107 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-17394a\"/\"kube-root-ca.crt\"" Oct 29 22:54:24.862218 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:24.862205 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-17394a\"/\"default-dockercfg-mtks9\"" Oct 29 22:54:24.862289 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:24.862277 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-17394a\"/\"distinct-2-w3-17394a-kserve-self-signed-certs\"" Oct 29 22:54:24.902842 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:24.902806 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:54:24.966526 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:24.966491 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f29e8b65-b7dd-45d4-86b0-f13a29ef8e66-tmp-dir\") pod \"distinct-2-w3-17394a-kserve-6545c766c4-h4chf\" (UID: \"f29e8b65-b7dd-45d4-86b0-f13a29ef8e66\") " pod="e2e-models-e2e-worker-w3-17394a/distinct-2-w3-17394a-kserve-6545c766c4-h4chf" Oct 29 22:54:24.966698 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:24.966674 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86s6t\" (UniqueName: \"kubernetes.io/projected/f29e8b65-b7dd-45d4-86b0-f13a29ef8e66-kube-api-access-86s6t\") pod \"distinct-2-w3-17394a-kserve-6545c766c4-h4chf\" (UID: \"f29e8b65-b7dd-45d4-86b0-f13a29ef8e66\") " pod="e2e-models-e2e-worker-w3-17394a/distinct-2-w3-17394a-kserve-6545c766c4-h4chf" Oct 29 22:54:24.966761 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:24.966724 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f29e8b65-b7dd-45d4-86b0-f13a29ef8e66-home\") pod \"distinct-2-w3-17394a-kserve-6545c766c4-h4chf\" (UID: \"f29e8b65-b7dd-45d4-86b0-f13a29ef8e66\") " pod="e2e-models-e2e-worker-w3-17394a/distinct-2-w3-17394a-kserve-6545c766c4-h4chf" Oct 29 22:54:24.966813 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:24.966794 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f29e8b65-b7dd-45d4-86b0-f13a29ef8e66-model-cache\") pod \"distinct-2-w3-17394a-kserve-6545c766c4-h4chf\" (UID: \"f29e8b65-b7dd-45d4-86b0-f13a29ef8e66\") " pod="e2e-models-e2e-worker-w3-17394a/distinct-2-w3-17394a-kserve-6545c766c4-h4chf" Oct 29 22:54:24.967037 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:24.967008 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f29e8b65-b7dd-45d4-86b0-f13a29ef8e66-tls-certs\") pod \"distinct-2-w3-17394a-kserve-6545c766c4-h4chf\" (UID: \"f29e8b65-b7dd-45d4-86b0-f13a29ef8e66\") " pod="e2e-models-e2e-worker-w3-17394a/distinct-2-w3-17394a-kserve-6545c766c4-h4chf" Oct 29 22:54:24.967149 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:24.967055 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f29e8b65-b7dd-45d4-86b0-f13a29ef8e66-dshm\") pod \"distinct-2-w3-17394a-kserve-6545c766c4-h4chf\" (UID: \"f29e8b65-b7dd-45d4-86b0-f13a29ef8e66\") " pod="e2e-models-e2e-worker-w3-17394a/distinct-2-w3-17394a-kserve-6545c766c4-h4chf" Oct 29 22:54:25.068436 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:25.068373 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f29e8b65-b7dd-45d4-86b0-f13a29ef8e66-tls-certs\") pod \"distinct-2-w3-17394a-kserve-6545c766c4-h4chf\" (UID: \"f29e8b65-b7dd-45d4-86b0-f13a29ef8e66\") " pod="e2e-models-e2e-worker-w3-17394a/distinct-2-w3-17394a-kserve-6545c766c4-h4chf" Oct 29 22:54:25.068436 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:25.068431 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f29e8b65-b7dd-45d4-86b0-f13a29ef8e66-dshm\") pod \"distinct-2-w3-17394a-kserve-6545c766c4-h4chf\" (UID: \"f29e8b65-b7dd-45d4-86b0-f13a29ef8e66\") " pod="e2e-models-e2e-worker-w3-17394a/distinct-2-w3-17394a-kserve-6545c766c4-h4chf" Oct 29 22:54:25.068722 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:25.068507 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f29e8b65-b7dd-45d4-86b0-f13a29ef8e66-tmp-dir\") pod \"distinct-2-w3-17394a-kserve-6545c766c4-h4chf\" (UID: \"f29e8b65-b7dd-45d4-86b0-f13a29ef8e66\") " pod="e2e-models-e2e-worker-w3-17394a/distinct-2-w3-17394a-kserve-6545c766c4-h4chf" Oct 29 22:54:25.068722 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:25.068604 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86s6t\" (UniqueName: \"kubernetes.io/projected/f29e8b65-b7dd-45d4-86b0-f13a29ef8e66-kube-api-access-86s6t\") pod \"distinct-2-w3-17394a-kserve-6545c766c4-h4chf\" (UID: \"f29e8b65-b7dd-45d4-86b0-f13a29ef8e66\") " pod="e2e-models-e2e-worker-w3-17394a/distinct-2-w3-17394a-kserve-6545c766c4-h4chf" Oct 29 22:54:25.068722 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:25.068644 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f29e8b65-b7dd-45d4-86b0-f13a29ef8e66-home\") pod \"distinct-2-w3-17394a-kserve-6545c766c4-h4chf\" (UID: \"f29e8b65-b7dd-45d4-86b0-f13a29ef8e66\") " pod="e2e-models-e2e-worker-w3-17394a/distinct-2-w3-17394a-kserve-6545c766c4-h4chf" Oct 29 22:54:25.068722 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:25.068671 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f29e8b65-b7dd-45d4-86b0-f13a29ef8e66-model-cache\") pod \"distinct-2-w3-17394a-kserve-6545c766c4-h4chf\" (UID: \"f29e8b65-b7dd-45d4-86b0-f13a29ef8e66\") " pod="e2e-models-e2e-worker-w3-17394a/distinct-2-w3-17394a-kserve-6545c766c4-h4chf" Oct 29 22:54:25.069195 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:25.069170 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f29e8b65-b7dd-45d4-86b0-f13a29ef8e66-model-cache\") pod \"distinct-2-w3-17394a-kserve-6545c766c4-h4chf\" (UID: \"f29e8b65-b7dd-45d4-86b0-f13a29ef8e66\") " pod="e2e-models-e2e-worker-w3-17394a/distinct-2-w3-17394a-kserve-6545c766c4-h4chf" Oct 29 22:54:25.069551 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:25.069526 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f29e8b65-b7dd-45d4-86b0-f13a29ef8e66-tmp-dir\") pod \"distinct-2-w3-17394a-kserve-6545c766c4-h4chf\" (UID: \"f29e8b65-b7dd-45d4-86b0-f13a29ef8e66\") " pod="e2e-models-e2e-worker-w3-17394a/distinct-2-w3-17394a-kserve-6545c766c4-h4chf" Oct 29 22:54:25.069816 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:25.069797 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f29e8b65-b7dd-45d4-86b0-f13a29ef8e66-home\") pod \"distinct-2-w3-17394a-kserve-6545c766c4-h4chf\" (UID: \"f29e8b65-b7dd-45d4-86b0-f13a29ef8e66\") " pod="e2e-models-e2e-worker-w3-17394a/distinct-2-w3-17394a-kserve-6545c766c4-h4chf" Oct 29 22:54:25.071499 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:25.071475 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f29e8b65-b7dd-45d4-86b0-f13a29ef8e66-tls-certs\") pod \"distinct-2-w3-17394a-kserve-6545c766c4-h4chf\" (UID: \"f29e8b65-b7dd-45d4-86b0-f13a29ef8e66\") " pod="e2e-models-e2e-worker-w3-17394a/distinct-2-w3-17394a-kserve-6545c766c4-h4chf" Oct 29 22:54:25.081651 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:25.081587 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f29e8b65-b7dd-45d4-86b0-f13a29ef8e66-dshm\") pod \"distinct-2-w3-17394a-kserve-6545c766c4-h4chf\" (UID: \"f29e8b65-b7dd-45d4-86b0-f13a29ef8e66\") " pod="e2e-models-e2e-worker-w3-17394a/distinct-2-w3-17394a-kserve-6545c766c4-h4chf" Oct 29 22:54:25.093524 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:25.093490 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-86s6t\" (UniqueName: \"kubernetes.io/projected/f29e8b65-b7dd-45d4-86b0-f13a29ef8e66-kube-api-access-86s6t\") pod \"distinct-2-w3-17394a-kserve-6545c766c4-h4chf\" (UID: \"f29e8b65-b7dd-45d4-86b0-f13a29ef8e66\") " pod="e2e-models-e2e-worker-w3-17394a/distinct-2-w3-17394a-kserve-6545c766c4-h4chf" Oct 29 22:54:25.172112 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:25.172083 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-17394a/distinct-2-w3-17394a-kserve-6545c766c4-h4chf" Oct 29 22:54:25.484159 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:25.484129 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-17394a/distinct-2-w3-17394a-kserve-6545c766c4-h4chf"] Oct 29 22:54:25.484788 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:54:25.484762 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf29e8b65_b7dd_45d4_86b0_f13a29ef8e66.slice/crio-11f5dd07350fdc84aaa620b23f82eb4f62c3264b38de76fad9c0a79fd9e4207c WatchSource:0}: Error finding container 11f5dd07350fdc84aaa620b23f82eb4f62c3264b38de76fad9c0a79fd9e4207c: Status 404 returned error can't find the container with id 11f5dd07350fdc84aaa620b23f82eb4f62c3264b38de76fad9c0a79fd9e4207c Oct 29 22:54:25.791759 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:25.791680 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:54:26.007907 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:26.007866 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-17394a/distinct-2-w3-17394a-kserve-6545c766c4-h4chf" event={"ID":"f29e8b65-b7dd-45d4-86b0-f13a29ef8e66","Type":"ContainerStarted","Data":"534283814de5a5f1bc34e8b2c5b3f11ed7f463f3e605be39cc7734627e01f942"} Oct 29 22:54:26.008370 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:26.007916 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-17394a/distinct-2-w3-17394a-kserve-6545c766c4-h4chf" event={"ID":"f29e8b65-b7dd-45d4-86b0-f13a29ef8e66","Type":"ContainerStarted","Data":"11f5dd07350fdc84aaa620b23f82eb4f62c3264b38de76fad9c0a79fd9e4207c"} Oct 29 22:54:26.047765 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:26.047655 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-17394a/distinct-2-w3-17394a-kserve-6545c766c4-h4chf" podStartSLOduration=2.047642103 podStartE2EDuration="2.047642103s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:54:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:54:26.045267548 +0000 UTC m=+1246.828480977" watchObservedRunningTime="2026-10-29 22:54:26.047642103 +0000 UTC m=+1246.830855531" Oct 29 22:54:26.172480 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:26.172447 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-17394a/distinct-2-w3-17394a-kserve-6545c766c4-h4chf" Oct 29 22:54:27.293727 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:27.293689 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:54:28.840278 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:28.840242 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:54:30.319715 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:30.319655 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:54:30.449919 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:30.449884 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:54:31.335587 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:31.335549 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:54:31.895664 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:31.895626 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:54:32.024080 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:32.024034 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:54:33.496076 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:33.496041 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:54:33.997746 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:33.997715 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v2tfm"] Oct 29 22:54:35.172506 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:35.172454 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-17394a/distinct-2-w3-17394a-kserve-6545c766c4-h4chf" Oct 29 22:54:35.184810 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:35.184784 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-17394a/distinct-2-w3-17394a-kserve-6545c766c4-h4chf" Oct 29 22:54:35.316135 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:35.316093 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:54:36.097314 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:36.097258 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-17394a/distinct-2-w3-17394a-kserve-6545c766c4-h4chf" Oct 29 22:54:36.284437 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:36.284399 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:54:36.396440 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:36.396367 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:54:36.990137 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:36.990105 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:54:38.532122 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:38.532089 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:54:38.681083 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:38.681047 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:54:40.154769 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:40.154732 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:54:41.839867 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:41.839831 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:54:43.369450 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:43.369411 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:54:44.891635 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:44.891556 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:54:46.342944 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:46.342905 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:54:47.880625 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:47.880589 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:54:49.029215 ip-10-0-140-74 kubenswrapper[5057]: E1029 22:54:49.029178 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d2dnn" containerName="extractor" Oct 29 22:54:49.337612 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:49.337529 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:54:49.357171 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:49.357141 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-nz2fk"] Oct 29 22:54:49.363004 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:49.362983 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-nz2fk" Oct 29 22:54:49.394935 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:49.394879 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-nz2fk"] Oct 29 22:54:49.395312 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:49.395294 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-575d59-openshift-default-dockercfg-gzz6n\"" Oct 29 22:54:49.410402 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:49.409009 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03a26d04-61f6-4e70-b52d-f6263220a95d-istio-token\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-nz2fk\" (UID: \"03a26d04-61f6-4e70-b52d-f6263220a95d\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-nz2fk" Oct 29 22:54:49.410402 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:49.409135 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03a26d04-61f6-4e70-b52d-f6263220a95d-workload-certs\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-nz2fk\" (UID: \"03a26d04-61f6-4e70-b52d-f6263220a95d\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-nz2fk" Oct 29 22:54:49.410402 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:49.409232 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03a26d04-61f6-4e70-b52d-f6263220a95d-istio-podinfo\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-nz2fk\" (UID: \"03a26d04-61f6-4e70-b52d-f6263220a95d\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-nz2fk" Oct 29 22:54:49.410402 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:49.409303 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03a26d04-61f6-4e70-b52d-f6263220a95d-credential-socket\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-nz2fk\" (UID: \"03a26d04-61f6-4e70-b52d-f6263220a95d\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-nz2fk" Oct 29 22:54:49.410402 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:49.409345 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03a26d04-61f6-4e70-b52d-f6263220a95d-istio-envoy\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-nz2fk\" (UID: \"03a26d04-61f6-4e70-b52d-f6263220a95d\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-nz2fk" Oct 29 22:54:49.410402 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:49.409369 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03a26d04-61f6-4e70-b52d-f6263220a95d-istiod-ca-cert\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-nz2fk\" (UID: \"03a26d04-61f6-4e70-b52d-f6263220a95d\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-nz2fk" Oct 29 22:54:49.410402 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:49.409396 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03a26d04-61f6-4e70-b52d-f6263220a95d-istio-data\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-nz2fk\" (UID: \"03a26d04-61f6-4e70-b52d-f6263220a95d\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-nz2fk" Oct 29 22:54:49.410402 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:49.409509 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03a26d04-61f6-4e70-b52d-f6263220a95d-workload-socket\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-nz2fk\" (UID: \"03a26d04-61f6-4e70-b52d-f6263220a95d\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-nz2fk" Oct 29 22:54:49.410402 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:49.409553 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8x2bd\" (UniqueName: \"kubernetes.io/projected/03a26d04-61f6-4e70-b52d-f6263220a95d-kube-api-access-8x2bd\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-nz2fk\" (UID: \"03a26d04-61f6-4e70-b52d-f6263220a95d\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-nz2fk" Oct 29 22:54:49.511269 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:49.511232 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03a26d04-61f6-4e70-b52d-f6263220a95d-istio-token\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-nz2fk\" (UID: \"03a26d04-61f6-4e70-b52d-f6263220a95d\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-nz2fk" Oct 29 22:54:49.511456 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:49.511361 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03a26d04-61f6-4e70-b52d-f6263220a95d-workload-certs\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-nz2fk\" (UID: \"03a26d04-61f6-4e70-b52d-f6263220a95d\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-nz2fk" Oct 29 22:54:49.511456 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:49.511414 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03a26d04-61f6-4e70-b52d-f6263220a95d-istio-podinfo\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-nz2fk\" (UID: \"03a26d04-61f6-4e70-b52d-f6263220a95d\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-nz2fk" Oct 29 22:54:49.511562 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:49.511476 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03a26d04-61f6-4e70-b52d-f6263220a95d-credential-socket\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-nz2fk\" (UID: \"03a26d04-61f6-4e70-b52d-f6263220a95d\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-nz2fk" Oct 29 22:54:49.511562 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:49.511499 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03a26d04-61f6-4e70-b52d-f6263220a95d-istio-envoy\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-nz2fk\" (UID: \"03a26d04-61f6-4e70-b52d-f6263220a95d\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-nz2fk" Oct 29 22:54:49.511562 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:49.511522 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03a26d04-61f6-4e70-b52d-f6263220a95d-istiod-ca-cert\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-nz2fk\" (UID: \"03a26d04-61f6-4e70-b52d-f6263220a95d\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-nz2fk" Oct 29 22:54:49.511562 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:49.511548 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03a26d04-61f6-4e70-b52d-f6263220a95d-istio-data\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-nz2fk\" (UID: \"03a26d04-61f6-4e70-b52d-f6263220a95d\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-nz2fk" Oct 29 22:54:49.511788 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:49.511636 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03a26d04-61f6-4e70-b52d-f6263220a95d-workload-socket\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-nz2fk\" (UID: \"03a26d04-61f6-4e70-b52d-f6263220a95d\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-nz2fk" Oct 29 22:54:49.511788 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:49.511681 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8x2bd\" (UniqueName: \"kubernetes.io/projected/03a26d04-61f6-4e70-b52d-f6263220a95d-kube-api-access-8x2bd\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-nz2fk\" (UID: \"03a26d04-61f6-4e70-b52d-f6263220a95d\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-nz2fk" Oct 29 22:54:49.511949 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:49.511924 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03a26d04-61f6-4e70-b52d-f6263220a95d-credential-socket\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-nz2fk\" (UID: \"03a26d04-61f6-4e70-b52d-f6263220a95d\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-nz2fk" Oct 29 22:54:49.512041 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:49.512018 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03a26d04-61f6-4e70-b52d-f6263220a95d-istio-data\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-nz2fk\" (UID: \"03a26d04-61f6-4e70-b52d-f6263220a95d\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-nz2fk" Oct 29 22:54:49.512104 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:49.512034 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03a26d04-61f6-4e70-b52d-f6263220a95d-workload-certs\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-nz2fk\" (UID: \"03a26d04-61f6-4e70-b52d-f6263220a95d\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-nz2fk" Oct 29 22:54:49.512260 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:49.512239 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03a26d04-61f6-4e70-b52d-f6263220a95d-workload-socket\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-nz2fk\" (UID: \"03a26d04-61f6-4e70-b52d-f6263220a95d\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-nz2fk" Oct 29 22:54:49.512305 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:49.512264 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03a26d04-61f6-4e70-b52d-f6263220a95d-istiod-ca-cert\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-nz2fk\" (UID: \"03a26d04-61f6-4e70-b52d-f6263220a95d\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-nz2fk" Oct 29 22:54:49.513858 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:49.513838 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03a26d04-61f6-4e70-b52d-f6263220a95d-istio-envoy\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-nz2fk\" (UID: \"03a26d04-61f6-4e70-b52d-f6263220a95d\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-nz2fk" Oct 29 22:54:49.514293 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:49.514274 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03a26d04-61f6-4e70-b52d-f6263220a95d-istio-podinfo\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-nz2fk\" (UID: \"03a26d04-61f6-4e70-b52d-f6263220a95d\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-nz2fk" Oct 29 22:54:49.532984 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:49.532960 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03a26d04-61f6-4e70-b52d-f6263220a95d-istio-token\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-nz2fk\" (UID: \"03a26d04-61f6-4e70-b52d-f6263220a95d\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-nz2fk" Oct 29 22:54:49.533233 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:49.533211 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8x2bd\" (UniqueName: \"kubernetes.io/projected/03a26d04-61f6-4e70-b52d-f6263220a95d-kube-api-access-8x2bd\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-nz2fk\" (UID: \"03a26d04-61f6-4e70-b52d-f6263220a95d\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-nz2fk" Oct 29 22:54:49.678940 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:49.678859 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-nz2fk" Oct 29 22:54:50.004886 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:54:50.004853 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03a26d04_61f6_4e70_b52d_f6263220a95d.slice/crio-ebe25b768396567f330019db6bf23634aa7d9f09a375d8764159eebe3faa7b89 WatchSource:0}: Error finding container ebe25b768396567f330019db6bf23634aa7d9f09a375d8764159eebe3faa7b89: Status 404 returned error can't find the container with id ebe25b768396567f330019db6bf23634aa7d9f09a375d8764159eebe3faa7b89 Oct 29 22:54:50.005895 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:50.005857 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-nz2fk"] Oct 29 22:54:50.007640 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:50.007597 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:54:50.007737 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:50.007691 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:54:50.007803 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:50.007734 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:54:50.183234 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:50.183200 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-nz2fk" event={"ID":"03a26d04-61f6-4e70-b52d-f6263220a95d","Type":"ContainerStarted","Data":"b8576a0b55136d72e23dc28a90605488773503d9d1eb5adf4ed25f1f89274634"} Oct 29 22:54:50.183643 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:50.183257 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-nz2fk" event={"ID":"03a26d04-61f6-4e70-b52d-f6263220a95d","Type":"ContainerStarted","Data":"ebe25b768396567f330019db6bf23634aa7d9f09a375d8764159eebe3faa7b89"} Oct 29 22:54:50.224353 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:50.224278 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-nz2fk" podStartSLOduration=1.224260637 podStartE2EDuration="1.224260637s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:54:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:54:50.22376416 +0000 UTC m=+1271.006977599" watchObservedRunningTime="2026-10-29 22:54:50.224260637 +0000 UTC m=+1271.007474060" Oct 29 22:54:50.689355 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:50.684585 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-nz2fk" Oct 29 22:54:50.754948 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:50.754903 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:54:51.424030 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:51.423991 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-r8g84"] Oct 29 22:54:51.430167 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:51.430121 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-r8g84" Oct 29 22:54:51.449928 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:51.449894 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-r8g84"] Oct 29 22:54:51.539294 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:51.539258 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6c2cdc5-4b87-43a0-a1a0-258e925d849c-istio-envoy\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-r8g84\" (UID: \"f6c2cdc5-4b87-43a0-a1a0-258e925d849c\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-r8g84" Oct 29 22:54:51.539294 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:51.539295 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6c2cdc5-4b87-43a0-a1a0-258e925d849c-istio-token\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-r8g84\" (UID: \"f6c2cdc5-4b87-43a0-a1a0-258e925d849c\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-r8g84" Oct 29 22:54:51.539580 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:51.539312 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2bv8f\" (UniqueName: \"kubernetes.io/projected/f6c2cdc5-4b87-43a0-a1a0-258e925d849c-kube-api-access-2bv8f\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-r8g84\" (UID: \"f6c2cdc5-4b87-43a0-a1a0-258e925d849c\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-r8g84" Oct 29 22:54:51.539580 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:51.539354 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6c2cdc5-4b87-43a0-a1a0-258e925d849c-istio-data\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-r8g84\" (UID: \"f6c2cdc5-4b87-43a0-a1a0-258e925d849c\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-r8g84" Oct 29 22:54:51.539580 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:51.539525 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6c2cdc5-4b87-43a0-a1a0-258e925d849c-workload-socket\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-r8g84\" (UID: \"f6c2cdc5-4b87-43a0-a1a0-258e925d849c\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-r8g84" Oct 29 22:54:51.539739 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:51.539612 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6c2cdc5-4b87-43a0-a1a0-258e925d849c-workload-certs\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-r8g84\" (UID: \"f6c2cdc5-4b87-43a0-a1a0-258e925d849c\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-r8g84" Oct 29 22:54:51.539800 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:51.539777 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6c2cdc5-4b87-43a0-a1a0-258e925d849c-istiod-ca-cert\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-r8g84\" (UID: \"f6c2cdc5-4b87-43a0-a1a0-258e925d849c\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-r8g84" Oct 29 22:54:51.539879 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:51.539857 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6c2cdc5-4b87-43a0-a1a0-258e925d849c-credential-socket\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-r8g84\" (UID: \"f6c2cdc5-4b87-43a0-a1a0-258e925d849c\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-r8g84" Oct 29 22:54:51.539935 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:51.539920 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6c2cdc5-4b87-43a0-a1a0-258e925d849c-istio-podinfo\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-r8g84\" (UID: \"f6c2cdc5-4b87-43a0-a1a0-258e925d849c\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-r8g84" Oct 29 22:54:51.641482 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:51.641445 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6c2cdc5-4b87-43a0-a1a0-258e925d849c-istiod-ca-cert\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-r8g84\" (UID: \"f6c2cdc5-4b87-43a0-a1a0-258e925d849c\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-r8g84" Oct 29 22:54:51.641675 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:51.641521 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6c2cdc5-4b87-43a0-a1a0-258e925d849c-credential-socket\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-r8g84\" (UID: \"f6c2cdc5-4b87-43a0-a1a0-258e925d849c\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-r8g84" Oct 29 22:54:51.641675 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:51.641573 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6c2cdc5-4b87-43a0-a1a0-258e925d849c-istio-podinfo\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-r8g84\" (UID: \"f6c2cdc5-4b87-43a0-a1a0-258e925d849c\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-r8g84" Oct 29 22:54:51.641675 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:51.641662 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6c2cdc5-4b87-43a0-a1a0-258e925d849c-istio-envoy\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-r8g84\" (UID: \"f6c2cdc5-4b87-43a0-a1a0-258e925d849c\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-r8g84" Oct 29 22:54:51.641848 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:51.641688 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6c2cdc5-4b87-43a0-a1a0-258e925d849c-istio-token\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-r8g84\" (UID: \"f6c2cdc5-4b87-43a0-a1a0-258e925d849c\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-r8g84" Oct 29 22:54:51.641848 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:51.641712 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bv8f\" (UniqueName: \"kubernetes.io/projected/f6c2cdc5-4b87-43a0-a1a0-258e925d849c-kube-api-access-2bv8f\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-r8g84\" (UID: \"f6c2cdc5-4b87-43a0-a1a0-258e925d849c\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-r8g84" Oct 29 22:54:51.641848 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:51.641747 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6c2cdc5-4b87-43a0-a1a0-258e925d849c-istio-data\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-r8g84\" (UID: \"f6c2cdc5-4b87-43a0-a1a0-258e925d849c\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-r8g84" Oct 29 22:54:51.641848 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:51.641789 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6c2cdc5-4b87-43a0-a1a0-258e925d849c-workload-socket\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-r8g84\" (UID: \"f6c2cdc5-4b87-43a0-a1a0-258e925d849c\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-r8g84" Oct 29 22:54:51.642049 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:51.641921 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6c2cdc5-4b87-43a0-a1a0-258e925d849c-workload-certs\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-r8g84\" (UID: \"f6c2cdc5-4b87-43a0-a1a0-258e925d849c\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-r8g84" Oct 29 22:54:51.642049 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:51.641981 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6c2cdc5-4b87-43a0-a1a0-258e925d849c-credential-socket\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-r8g84\" (UID: \"f6c2cdc5-4b87-43a0-a1a0-258e925d849c\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-r8g84" Oct 29 22:54:51.642316 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:51.642291 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6c2cdc5-4b87-43a0-a1a0-258e925d849c-istiod-ca-cert\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-r8g84\" (UID: \"f6c2cdc5-4b87-43a0-a1a0-258e925d849c\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-r8g84" Oct 29 22:54:51.642438 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:51.642318 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6c2cdc5-4b87-43a0-a1a0-258e925d849c-workload-certs\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-r8g84\" (UID: \"f6c2cdc5-4b87-43a0-a1a0-258e925d849c\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-r8g84" Oct 29 22:54:51.642438 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:51.642364 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6c2cdc5-4b87-43a0-a1a0-258e925d849c-istio-data\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-r8g84\" (UID: \"f6c2cdc5-4b87-43a0-a1a0-258e925d849c\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-r8g84" Oct 29 22:54:51.642526 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:51.642456 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6c2cdc5-4b87-43a0-a1a0-258e925d849c-workload-socket\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-r8g84\" (UID: \"f6c2cdc5-4b87-43a0-a1a0-258e925d849c\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-r8g84" Oct 29 22:54:51.644119 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:51.644094 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6c2cdc5-4b87-43a0-a1a0-258e925d849c-istio-envoy\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-r8g84\" (UID: \"f6c2cdc5-4b87-43a0-a1a0-258e925d849c\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-r8g84" Oct 29 22:54:51.644561 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:51.644544 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6c2cdc5-4b87-43a0-a1a0-258e925d849c-istio-podinfo\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-r8g84\" (UID: \"f6c2cdc5-4b87-43a0-a1a0-258e925d849c\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-r8g84" Oct 29 22:54:51.663122 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:51.663101 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6c2cdc5-4b87-43a0-a1a0-258e925d849c-istio-token\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-r8g84\" (UID: \"f6c2cdc5-4b87-43a0-a1a0-258e925d849c\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-r8g84" Oct 29 22:54:51.663271 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:51.663253 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2bv8f\" (UniqueName: \"kubernetes.io/projected/f6c2cdc5-4b87-43a0-a1a0-258e925d849c-kube-api-access-2bv8f\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-r8g84\" (UID: \"f6c2cdc5-4b87-43a0-a1a0-258e925d849c\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-r8g84" Oct 29 22:54:51.702352 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:51.700166 5057 patch_prober.go:29] interesting pod/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-nz2fk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Oct 29 22:54:51.702352 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:51.700242 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-nz2fk" podUID="03a26d04-61f6-4e70-b52d-f6263220a95d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Oct 29 22:54:51.751382 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:51.751318 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-r8g84" Oct 29 22:54:52.114639 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:52.114142 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-r8g84"] Oct 29 22:54:52.117246 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:54:52.117215 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6c2cdc5_4b87_43a0_a1a0_258e925d849c.slice/crio-628cfdc341f9d3aa564fff85404c12c8215f9da56c6a17b1338bf627903e28b4 WatchSource:0}: Error finding container 628cfdc341f9d3aa564fff85404c12c8215f9da56c6a17b1338bf627903e28b4: Status 404 returned error can't find the container with id 628cfdc341f9d3aa564fff85404c12c8215f9da56c6a17b1338bf627903e28b4 Oct 29 22:54:52.121710 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:52.121670 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:54:52.121797 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:52.121752 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:54:52.121841 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:52.121795 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:54:52.217380 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:52.217159 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:54:52.222275 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:52.222247 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-r8g84" event={"ID":"f6c2cdc5-4b87-43a0-a1a0-258e925d849c","Type":"ContainerStarted","Data":"628cfdc341f9d3aa564fff85404c12c8215f9da56c6a17b1338bf627903e28b4"} Oct 29 22:54:52.680705 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:52.680670 5057 patch_prober.go:29] interesting pod/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-nz2fk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Oct 29 22:54:52.681263 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:52.681227 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-nz2fk" podUID="03a26d04-61f6-4e70-b52d-f6263220a95d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Oct 29 22:54:53.243915 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:53.243880 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-r8g84" event={"ID":"f6c2cdc5-4b87-43a0-a1a0-258e925d849c","Type":"ContainerStarted","Data":"076e34911feb1ca961414be82bbb939094b236e50500069ffa6215f2a5e68dd1"} Oct 29 22:54:53.353833 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:53.353762 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-r8g84" podStartSLOduration=2.35374199 podStartE2EDuration="2.35374199s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:54:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:54:53.300088073 +0000 UTC m=+1274.083301513" watchObservedRunningTime="2026-10-29 22:54:53.35374199 +0000 UTC m=+1274.136955416" Oct 29 22:54:53.684435 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:53.680698 5057 patch_prober.go:29] interesting pod/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-nz2fk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Oct 29 22:54:53.684435 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:53.680759 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-nz2fk" podUID="03a26d04-61f6-4e70-b52d-f6263220a95d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Oct 29 22:54:53.752352 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:53.752267 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-r8g84" Oct 29 22:54:53.754233 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:53.754205 5057 patch_prober.go:29] interesting pod/e2e-worker-w4-063b46-openshift-default-799d8d64cf-r8g84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Oct 29 22:54:53.754380 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:53.754255 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-r8g84" podUID="f6c2cdc5-4b87-43a0-a1a0-258e925d849c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Oct 29 22:54:53.896030 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:53.895992 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:54:54.698357 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:54.695908 5057 patch_prober.go:29] interesting pod/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-nz2fk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Oct 29 22:54:54.698357 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:54.695973 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-nz2fk" podUID="03a26d04-61f6-4e70-b52d-f6263220a95d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Oct 29 22:54:54.752412 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:54.752372 5057 patch_prober.go:29] interesting pod/e2e-worker-w4-063b46-openshift-default-799d8d64cf-r8g84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Oct 29 22:54:54.752570 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:54.752423 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-r8g84" podUID="f6c2cdc5-4b87-43a0-a1a0-258e925d849c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Oct 29 22:54:55.688356 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:55.680666 5057 patch_prober.go:29] interesting pod/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-nz2fk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Oct 29 22:54:55.688356 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:55.680726 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-nz2fk" podUID="03a26d04-61f6-4e70-b52d-f6263220a95d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Oct 29 22:54:55.696056 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:55.693227 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:54:55.752099 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:55.752064 5057 patch_prober.go:29] interesting pod/e2e-worker-w4-063b46-openshift-default-799d8d64cf-r8g84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Oct 29 22:54:55.752514 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:55.752407 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-r8g84" podUID="f6c2cdc5-4b87-43a0-a1a0-258e925d849c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Oct 29 22:54:56.434468 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:56.434434 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:54:56.683589 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:56.683107 5057 patch_prober.go:29] interesting pod/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-nz2fk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Oct 29 22:54:56.683589 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:56.683175 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-nz2fk" podUID="03a26d04-61f6-4e70-b52d-f6263220a95d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Oct 29 22:54:56.752662 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:56.752625 5057 patch_prober.go:29] interesting pod/e2e-worker-w4-063b46-openshift-default-799d8d64cf-r8g84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Oct 29 22:54:56.753142 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:56.752691 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-r8g84" podUID="f6c2cdc5-4b87-43a0-a1a0-258e925d849c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Oct 29 22:54:57.310269 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:57.310230 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:54:57.700365 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:57.700311 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-nz2fk" Oct 29 22:54:57.701965 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:57.701946 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-nz2fk" Oct 29 22:54:57.707630 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:57.707608 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-nz2fk" Oct 29 22:54:57.751975 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:57.751946 5057 patch_prober.go:29] interesting pod/e2e-worker-w4-063b46-openshift-default-799d8d64cf-r8g84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Oct 29 22:54:57.752069 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:57.751987 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-r8g84" podUID="f6c2cdc5-4b87-43a0-a1a0-258e925d849c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Oct 29 22:54:58.752129 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:58.752094 5057 patch_prober.go:29] interesting pod/e2e-worker-w4-063b46-openshift-default-799d8d64cf-r8g84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Oct 29 22:54:58.752540 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:58.752156 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-r8g84" podUID="f6c2cdc5-4b87-43a0-a1a0-258e925d849c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Oct 29 22:54:58.833524 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:58.833485 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:54:59.767360 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:59.767299 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-r8g84" Oct 29 22:54:59.767874 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:59.767826 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-r8g84" Oct 29 22:54:59.768928 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:54:59.768904 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-r8g84" Oct 29 22:55:00.438741 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:00.438707 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:55:02.967077 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:02.967045 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:55:04.699216 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:04.699176 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:55:06.212309 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:06.212268 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:55:07.558051 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:07.558012 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:55:07.892766 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:07.892684 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:55:09.571125 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:09.571090 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:55:11.124658 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:11.124622 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:55:12.671453 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:12.671413 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:55:14.297635 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:14.297596 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:55:16.010564 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:16.010527 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:55:16.035245 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:16.035212 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-575d59/distinct-2-w2-575d59-kserve-d4565b76c-hdgbf"] Oct 29 22:55:16.042411 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:16.042388 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-575d59/distinct-2-w2-575d59-kserve-d4565b76c-hdgbf" Oct 29 22:55:16.058795 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:16.058761 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-575d59/distinct-2-w2-575d59-kserve-d4565b76c-hdgbf"] Oct 29 22:55:16.067413 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:16.067377 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-575d59\"/\"default-dockercfg-stszs\"" Oct 29 22:55:16.067413 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:16.067406 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-575d59\"/\"kube-root-ca.crt\"" Oct 29 22:55:16.067569 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:16.067418 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-575d59\"/\"distinct-2-w2-575d59-kserve-self-signed-certs\"" Oct 29 22:55:16.067569 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:16.067407 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-575d59\"/\"openshift-service-ca.crt\"" Oct 29 22:55:16.125888 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:16.125845 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:55:16.136483 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:16.136455 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06bb975e-d825-426f-b030-17bd1f99bb65-tls-certs\") pod \"distinct-2-w2-575d59-kserve-d4565b76c-hdgbf\" (UID: \"06bb975e-d825-426f-b030-17bd1f99bb65\") " pod="e2e-models-e2e-worker-w2-575d59/distinct-2-w2-575d59-kserve-d4565b76c-hdgbf" Oct 29 22:55:16.136591 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:16.136489 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06bb975e-d825-426f-b030-17bd1f99bb65-model-cache\") pod \"distinct-2-w2-575d59-kserve-d4565b76c-hdgbf\" (UID: \"06bb975e-d825-426f-b030-17bd1f99bb65\") " pod="e2e-models-e2e-worker-w2-575d59/distinct-2-w2-575d59-kserve-d4565b76c-hdgbf" Oct 29 22:55:16.136633 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:16.136607 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hchnc\" (UniqueName: \"kubernetes.io/projected/06bb975e-d825-426f-b030-17bd1f99bb65-kube-api-access-hchnc\") pod \"distinct-2-w2-575d59-kserve-d4565b76c-hdgbf\" (UID: \"06bb975e-d825-426f-b030-17bd1f99bb65\") " pod="e2e-models-e2e-worker-w2-575d59/distinct-2-w2-575d59-kserve-d4565b76c-hdgbf" Oct 29 22:55:16.136633 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:16.136627 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06bb975e-d825-426f-b030-17bd1f99bb65-tmp-dir\") pod \"distinct-2-w2-575d59-kserve-d4565b76c-hdgbf\" (UID: \"06bb975e-d825-426f-b030-17bd1f99bb65\") " pod="e2e-models-e2e-worker-w2-575d59/distinct-2-w2-575d59-kserve-d4565b76c-hdgbf" Oct 29 22:55:16.136703 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:16.136668 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06bb975e-d825-426f-b030-17bd1f99bb65-home\") pod \"distinct-2-w2-575d59-kserve-d4565b76c-hdgbf\" (UID: \"06bb975e-d825-426f-b030-17bd1f99bb65\") " pod="e2e-models-e2e-worker-w2-575d59/distinct-2-w2-575d59-kserve-d4565b76c-hdgbf" Oct 29 22:55:16.136762 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:16.136749 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06bb975e-d825-426f-b030-17bd1f99bb65-dshm\") pod \"distinct-2-w2-575d59-kserve-d4565b76c-hdgbf\" (UID: \"06bb975e-d825-426f-b030-17bd1f99bb65\") " pod="e2e-models-e2e-worker-w2-575d59/distinct-2-w2-575d59-kserve-d4565b76c-hdgbf" Oct 29 22:55:16.238054 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:16.238021 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06bb975e-d825-426f-b030-17bd1f99bb65-tls-certs\") pod \"distinct-2-w2-575d59-kserve-d4565b76c-hdgbf\" (UID: \"06bb975e-d825-426f-b030-17bd1f99bb65\") " pod="e2e-models-e2e-worker-w2-575d59/distinct-2-w2-575d59-kserve-d4565b76c-hdgbf" Oct 29 22:55:16.238207 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:16.238065 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06bb975e-d825-426f-b030-17bd1f99bb65-model-cache\") pod \"distinct-2-w2-575d59-kserve-d4565b76c-hdgbf\" (UID: \"06bb975e-d825-426f-b030-17bd1f99bb65\") " pod="e2e-models-e2e-worker-w2-575d59/distinct-2-w2-575d59-kserve-d4565b76c-hdgbf" Oct 29 22:55:16.238207 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:16.238161 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hchnc\" (UniqueName: \"kubernetes.io/projected/06bb975e-d825-426f-b030-17bd1f99bb65-kube-api-access-hchnc\") pod \"distinct-2-w2-575d59-kserve-d4565b76c-hdgbf\" (UID: \"06bb975e-d825-426f-b030-17bd1f99bb65\") " pod="e2e-models-e2e-worker-w2-575d59/distinct-2-w2-575d59-kserve-d4565b76c-hdgbf" Oct 29 22:55:16.238207 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:16.238187 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06bb975e-d825-426f-b030-17bd1f99bb65-tmp-dir\") pod \"distinct-2-w2-575d59-kserve-d4565b76c-hdgbf\" (UID: \"06bb975e-d825-426f-b030-17bd1f99bb65\") " pod="e2e-models-e2e-worker-w2-575d59/distinct-2-w2-575d59-kserve-d4565b76c-hdgbf" Oct 29 22:55:16.238361 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:16.238255 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06bb975e-d825-426f-b030-17bd1f99bb65-home\") pod \"distinct-2-w2-575d59-kserve-d4565b76c-hdgbf\" (UID: \"06bb975e-d825-426f-b030-17bd1f99bb65\") " pod="e2e-models-e2e-worker-w2-575d59/distinct-2-w2-575d59-kserve-d4565b76c-hdgbf" Oct 29 22:55:16.238361 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:16.238281 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06bb975e-d825-426f-b030-17bd1f99bb65-dshm\") pod \"distinct-2-w2-575d59-kserve-d4565b76c-hdgbf\" (UID: \"06bb975e-d825-426f-b030-17bd1f99bb65\") " pod="e2e-models-e2e-worker-w2-575d59/distinct-2-w2-575d59-kserve-d4565b76c-hdgbf" Oct 29 22:55:16.239153 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:16.239125 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06bb975e-d825-426f-b030-17bd1f99bb65-model-cache\") pod \"distinct-2-w2-575d59-kserve-d4565b76c-hdgbf\" (UID: \"06bb975e-d825-426f-b030-17bd1f99bb65\") " pod="e2e-models-e2e-worker-w2-575d59/distinct-2-w2-575d59-kserve-d4565b76c-hdgbf" Oct 29 22:55:16.239405 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:16.239384 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06bb975e-d825-426f-b030-17bd1f99bb65-home\") pod \"distinct-2-w2-575d59-kserve-d4565b76c-hdgbf\" (UID: \"06bb975e-d825-426f-b030-17bd1f99bb65\") " pod="e2e-models-e2e-worker-w2-575d59/distinct-2-w2-575d59-kserve-d4565b76c-hdgbf" Oct 29 22:55:16.239540 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:16.239519 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06bb975e-d825-426f-b030-17bd1f99bb65-tmp-dir\") pod \"distinct-2-w2-575d59-kserve-d4565b76c-hdgbf\" (UID: \"06bb975e-d825-426f-b030-17bd1f99bb65\") " pod="e2e-models-e2e-worker-w2-575d59/distinct-2-w2-575d59-kserve-d4565b76c-hdgbf" Oct 29 22:55:16.241546 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:16.241525 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06bb975e-d825-426f-b030-17bd1f99bb65-dshm\") pod \"distinct-2-w2-575d59-kserve-d4565b76c-hdgbf\" (UID: \"06bb975e-d825-426f-b030-17bd1f99bb65\") " pod="e2e-models-e2e-worker-w2-575d59/distinct-2-w2-575d59-kserve-d4565b76c-hdgbf" Oct 29 22:55:16.242790 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:16.242767 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06bb975e-d825-426f-b030-17bd1f99bb65-tls-certs\") pod \"distinct-2-w2-575d59-kserve-d4565b76c-hdgbf\" (UID: \"06bb975e-d825-426f-b030-17bd1f99bb65\") " pod="e2e-models-e2e-worker-w2-575d59/distinct-2-w2-575d59-kserve-d4565b76c-hdgbf" Oct 29 22:55:16.265202 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:16.265085 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hchnc\" (UniqueName: \"kubernetes.io/projected/06bb975e-d825-426f-b030-17bd1f99bb65-kube-api-access-hchnc\") pod \"distinct-2-w2-575d59-kserve-d4565b76c-hdgbf\" (UID: \"06bb975e-d825-426f-b030-17bd1f99bb65\") " pod="e2e-models-e2e-worker-w2-575d59/distinct-2-w2-575d59-kserve-d4565b76c-hdgbf" Oct 29 22:55:16.361640 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:16.361606 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-575d59/distinct-2-w2-575d59-kserve-d4565b76c-hdgbf" Oct 29 22:55:16.672894 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:16.672865 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-575d59/distinct-2-w2-575d59-kserve-d4565b76c-hdgbf"] Oct 29 22:55:16.676402 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:55:16.676371 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod06bb975e_d825_426f_b030_17bd1f99bb65.slice/crio-9cde8c313d2c28b3d25a153f7c5954231196de2ed7b6f9f267d9dcb2fcf04fe7 WatchSource:0}: Error finding container 9cde8c313d2c28b3d25a153f7c5954231196de2ed7b6f9f267d9dcb2fcf04fe7: Status 404 returned error can't find the container with id 9cde8c313d2c28b3d25a153f7c5954231196de2ed7b6f9f267d9dcb2fcf04fe7 Oct 29 22:55:17.468025 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:17.467985 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-575d59/distinct-2-w2-575d59-kserve-d4565b76c-hdgbf" event={"ID":"06bb975e-d825-426f-b030-17bd1f99bb65","Type":"ContainerStarted","Data":"e704c8a8830b7916eee00fae54a8c9ad8a79e8eb469659a79fab08e751af81a1"} Oct 29 22:55:17.468570 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:17.468035 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-575d59/distinct-2-w2-575d59-kserve-d4565b76c-hdgbf" event={"ID":"06bb975e-d825-426f-b030-17bd1f99bb65","Type":"ContainerStarted","Data":"9cde8c313d2c28b3d25a153f7c5954231196de2ed7b6f9f267d9dcb2fcf04fe7"} Oct 29 22:55:17.514174 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:17.514122 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-575d59/distinct-2-w2-575d59-kserve-d4565b76c-hdgbf" podStartSLOduration=1.514107692 podStartE2EDuration="1.514107692s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:55:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:55:17.511763727 +0000 UTC m=+1298.294977155" watchObservedRunningTime="2026-10-29 22:55:17.514107692 +0000 UTC m=+1298.297321117" Oct 29 22:55:17.608837 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:17.608796 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:55:18.361971 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:18.361933 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-575d59/distinct-2-w2-575d59-kserve-d4565b76c-hdgbf" Oct 29 22:55:19.031930 ip-10-0-140-74 kubenswrapper[5057]: E1029 22:55:19.031895 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d2dnn" containerName="extractor" Oct 29 22:55:19.081304 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:19.081268 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:55:20.691906 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:20.691820 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:55:22.298822 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:22.298785 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:55:23.021860 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:23.021818 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-7smjh"] Oct 29 22:55:23.027806 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:23.027779 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-7smjh" Oct 29 22:55:23.062242 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:23.062209 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-7smjh"] Oct 29 22:55:23.135659 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:23.135629 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-08e15d-openshift-default-dockercfg-kjhd8\"" Oct 29 22:55:23.136088 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:23.136066 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bc2103e-768a-4393-9522-68b0fc370219-credential-socket\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-7smjh\" (UID: \"7bc2103e-768a-4393-9522-68b0fc370219\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-7smjh" Oct 29 22:55:23.136133 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:23.136099 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7bc2103e-768a-4393-9522-68b0fc370219-istiod-ca-cert\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-7smjh\" (UID: \"7bc2103e-768a-4393-9522-68b0fc370219\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-7smjh" Oct 29 22:55:23.136133 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:23.136123 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njw8k\" (UniqueName: \"kubernetes.io/projected/7bc2103e-768a-4393-9522-68b0fc370219-kube-api-access-njw8k\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-7smjh\" (UID: \"7bc2103e-768a-4393-9522-68b0fc370219\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-7smjh" Oct 29 22:55:23.136271 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:23.136258 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7bc2103e-768a-4393-9522-68b0fc370219-istio-envoy\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-7smjh\" (UID: \"7bc2103e-768a-4393-9522-68b0fc370219\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-7smjh" Oct 29 22:55:23.136373 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:23.136327 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7bc2103e-768a-4393-9522-68b0fc370219-istio-podinfo\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-7smjh\" (UID: \"7bc2103e-768a-4393-9522-68b0fc370219\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-7smjh" Oct 29 22:55:23.136427 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:23.136410 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7bc2103e-768a-4393-9522-68b0fc370219-istio-token\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-7smjh\" (UID: \"7bc2103e-768a-4393-9522-68b0fc370219\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-7smjh" Oct 29 22:55:23.136501 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:23.136489 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7bc2103e-768a-4393-9522-68b0fc370219-istio-data\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-7smjh\" (UID: \"7bc2103e-768a-4393-9522-68b0fc370219\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-7smjh" Oct 29 22:55:23.136539 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:23.136530 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7bc2103e-768a-4393-9522-68b0fc370219-workload-certs\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-7smjh\" (UID: \"7bc2103e-768a-4393-9522-68b0fc370219\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-7smjh" Oct 29 22:55:23.136573 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:23.136563 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bc2103e-768a-4393-9522-68b0fc370219-workload-socket\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-7smjh\" (UID: \"7bc2103e-768a-4393-9522-68b0fc370219\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-7smjh" Oct 29 22:55:23.237378 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:23.237324 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7bc2103e-768a-4393-9522-68b0fc370219-istio-data\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-7smjh\" (UID: \"7bc2103e-768a-4393-9522-68b0fc370219\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-7smjh" Oct 29 22:55:23.237378 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:23.237382 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7bc2103e-768a-4393-9522-68b0fc370219-workload-certs\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-7smjh\" (UID: \"7bc2103e-768a-4393-9522-68b0fc370219\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-7smjh" Oct 29 22:55:23.237621 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:23.237418 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bc2103e-768a-4393-9522-68b0fc370219-workload-socket\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-7smjh\" (UID: \"7bc2103e-768a-4393-9522-68b0fc370219\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-7smjh" Oct 29 22:55:23.237621 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:23.237463 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bc2103e-768a-4393-9522-68b0fc370219-credential-socket\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-7smjh\" (UID: \"7bc2103e-768a-4393-9522-68b0fc370219\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-7smjh" Oct 29 22:55:23.237621 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:23.237480 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7bc2103e-768a-4393-9522-68b0fc370219-istiod-ca-cert\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-7smjh\" (UID: \"7bc2103e-768a-4393-9522-68b0fc370219\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-7smjh" Oct 29 22:55:23.237621 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:23.237495 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njw8k\" (UniqueName: \"kubernetes.io/projected/7bc2103e-768a-4393-9522-68b0fc370219-kube-api-access-njw8k\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-7smjh\" (UID: \"7bc2103e-768a-4393-9522-68b0fc370219\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-7smjh" Oct 29 22:55:23.237859 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:23.237629 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7bc2103e-768a-4393-9522-68b0fc370219-istio-envoy\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-7smjh\" (UID: \"7bc2103e-768a-4393-9522-68b0fc370219\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-7smjh" Oct 29 22:55:23.237859 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:23.237766 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7bc2103e-768a-4393-9522-68b0fc370219-istio-podinfo\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-7smjh\" (UID: \"7bc2103e-768a-4393-9522-68b0fc370219\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-7smjh" Oct 29 22:55:23.237859 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:23.237820 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7bc2103e-768a-4393-9522-68b0fc370219-istio-token\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-7smjh\" (UID: \"7bc2103e-768a-4393-9522-68b0fc370219\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-7smjh" Oct 29 22:55:23.237859 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:23.237818 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7bc2103e-768a-4393-9522-68b0fc370219-istio-data\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-7smjh\" (UID: \"7bc2103e-768a-4393-9522-68b0fc370219\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-7smjh" Oct 29 22:55:23.238023 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:23.237929 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bc2103e-768a-4393-9522-68b0fc370219-credential-socket\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-7smjh\" (UID: \"7bc2103e-768a-4393-9522-68b0fc370219\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-7smjh" Oct 29 22:55:23.238023 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:23.237954 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7bc2103e-768a-4393-9522-68b0fc370219-workload-certs\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-7smjh\" (UID: \"7bc2103e-768a-4393-9522-68b0fc370219\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-7smjh" Oct 29 22:55:23.238183 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:23.238166 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bc2103e-768a-4393-9522-68b0fc370219-workload-socket\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-7smjh\" (UID: \"7bc2103e-768a-4393-9522-68b0fc370219\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-7smjh" Oct 29 22:55:23.238406 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:23.238386 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7bc2103e-768a-4393-9522-68b0fc370219-istiod-ca-cert\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-7smjh\" (UID: \"7bc2103e-768a-4393-9522-68b0fc370219\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-7smjh" Oct 29 22:55:23.239814 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:23.239779 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7bc2103e-768a-4393-9522-68b0fc370219-istio-envoy\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-7smjh\" (UID: \"7bc2103e-768a-4393-9522-68b0fc370219\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-7smjh" Oct 29 22:55:23.240257 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:23.240229 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7bc2103e-768a-4393-9522-68b0fc370219-istio-podinfo\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-7smjh\" (UID: \"7bc2103e-768a-4393-9522-68b0fc370219\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-7smjh" Oct 29 22:55:23.259021 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:23.258998 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7bc2103e-768a-4393-9522-68b0fc370219-istio-token\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-7smjh\" (UID: \"7bc2103e-768a-4393-9522-68b0fc370219\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-7smjh" Oct 29 22:55:23.259119 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:23.259022 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-njw8k\" (UniqueName: \"kubernetes.io/projected/7bc2103e-768a-4393-9522-68b0fc370219-kube-api-access-njw8k\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-7smjh\" (UID: \"7bc2103e-768a-4393-9522-68b0fc370219\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-7smjh" Oct 29 22:55:23.344605 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:23.344530 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-7smjh" Oct 29 22:55:23.672345 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:23.672303 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-7smjh"] Oct 29 22:55:23.675092 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:55:23.675062 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bc2103e_768a_4393_9522_68b0fc370219.slice/crio-c460d10527202accda73cf505984dd3c983ec9d0a7c327d7decab3a6dfd6c1dd WatchSource:0}: Error finding container c460d10527202accda73cf505984dd3c983ec9d0a7c327d7decab3a6dfd6c1dd: Status 404 returned error can't find the container with id c460d10527202accda73cf505984dd3c983ec9d0a7c327d7decab3a6dfd6c1dd Oct 29 22:55:23.677908 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:23.677871 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:55:23.678013 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:23.677937 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:55:23.678013 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:23.677967 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:55:24.000979 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:24.000931 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:55:24.528811 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:24.528774 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-7smjh" event={"ID":"7bc2103e-768a-4393-9522-68b0fc370219","Type":"ContainerStarted","Data":"151c14950da1cf77656122259bc0c192ec6e36821c5b2a64e2e6dc54a4d6d3ba"} Oct 29 22:55:24.529197 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:24.528836 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-7smjh" event={"ID":"7bc2103e-768a-4393-9522-68b0fc370219","Type":"ContainerStarted","Data":"c460d10527202accda73cf505984dd3c983ec9d0a7c327d7decab3a6dfd6c1dd"} Oct 29 22:55:24.617409 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:24.610055 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-7smjh" podStartSLOduration=2.605325528 podStartE2EDuration="2.605325528s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:55:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:55:24.587589821 +0000 UTC m=+1305.370803250" watchObservedRunningTime="2026-10-29 22:55:24.605325528 +0000 UTC m=+1305.388538956" Oct 29 22:55:24.913402 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:24.909446 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v2tfm"] Oct 29 22:55:25.345567 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:25.345528 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-7smjh" Oct 29 22:55:25.347195 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:25.347168 5057 patch_prober.go:29] interesting pod/e2e-worker-w0-08e15d-openshift-default-d579fdc75-7smjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Oct 29 22:55:25.347410 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:25.347221 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-7smjh" podUID="7bc2103e-768a-4393-9522-68b0fc370219" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Oct 29 22:55:25.648020 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:25.647918 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:55:26.345828 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:26.345781 5057 patch_prober.go:29] interesting pod/e2e-worker-w0-08e15d-openshift-default-d579fdc75-7smjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Oct 29 22:55:26.346046 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:26.345857 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-7smjh" podUID="7bc2103e-768a-4393-9522-68b0fc370219" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Oct 29 22:55:26.362166 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:26.362128 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-575d59/distinct-2-w2-575d59-kserve-d4565b76c-hdgbf" Oct 29 22:55:26.415355 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:26.411900 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-575d59/distinct-2-w2-575d59-kserve-d4565b76c-hdgbf" Oct 29 22:55:26.565080 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:26.564814 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-575d59/distinct-2-w2-575d59-kserve-d4565b76c-hdgbf" Oct 29 22:55:27.352145 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:27.352108 5057 patch_prober.go:29] interesting pod/e2e-worker-w0-08e15d-openshift-default-d579fdc75-7smjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Oct 29 22:55:27.352748 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:27.352715 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-7smjh" podUID="7bc2103e-768a-4393-9522-68b0fc370219" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Oct 29 22:55:27.861373 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:27.852463 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-31226c/distinct-w6-31226c-kserve-858d95fd76-bkf7c" podUID="f3fc11c6-03c9-4314-99c3-f857b149415c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:55:28.349756 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:28.349720 5057 patch_prober.go:29] interesting pod/e2e-worker-w0-08e15d-openshift-default-d579fdc75-7smjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Oct 29 22:55:28.350050 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:28.350020 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-7smjh" podUID="7bc2103e-768a-4393-9522-68b0fc370219" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Oct 29 22:55:28.839974 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:28.839939 5057 patch_prober.go:29] interesting pod/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-nz2fk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 22:55:28.852221 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:28.840550 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-nz2fk" podUID="03a26d04-61f6-4e70-b52d-f6263220a95d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:55:29.081365 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:29.073143 5057 patch_prober.go:29] interesting pod/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-pk92s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 22:55:29.081365 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:29.073204 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-pk92s" podUID="9033e6ee-4fd2-41a2-8a7d-132b6ca20128" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:55:29.135209 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:29.135115 5057 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-31226c/distinct-w6-31226c-kserve-858d95fd76-bkf7c" podUID="f3fc11c6-03c9-4314-99c3-f857b149415c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:55:29.154467 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:29.154423 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17394a/distinct-2-w3-17394a-kserve-6545c766c4-h4chf" podUID="f29e8b65-b7dd-45d4-86b0-f13a29ef8e66" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:55:29.346468 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:29.346432 5057 patch_prober.go:29] interesting pod/e2e-worker-w0-08e15d-openshift-default-d579fdc75-7smjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Oct 29 22:55:29.486523 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:29.486479 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-7smjh" podUID="7bc2103e-768a-4393-9522-68b0fc370219" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Oct 29 22:55:29.812521 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:29.812421 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-08e15d/distinct-w0-08e15d-kserve-85695c78bb-p67gk" podUID="e664f012-3b80-4a8a-ac19-d767d92c0a33" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:55:30.232387 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:30.219628 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17394a/distinct-2-w3-17394a-kserve-6545c766c4-h4chf" podUID="f29e8b65-b7dd-45d4-86b0-f13a29ef8e66" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:55:30.232387 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:30.221065 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-17394a/distinct-2-w3-17394a-kserve-6545c766c4-h4chf" Oct 29 22:55:30.349732 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:30.344833 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-17394a/distinct-2-w3-17394a-kserve-6545c766c4-h4chf" Oct 29 22:55:30.349732 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:30.346033 5057 patch_prober.go:29] interesting pod/e2e-worker-w0-08e15d-openshift-default-d579fdc75-7smjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Oct 29 22:55:30.350446 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:30.350413 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-7smjh" podUID="7bc2103e-768a-4393-9522-68b0fc370219" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Oct 29 22:55:30.650661 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:30.650579 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:55:31.345150 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:31.345094 5057 patch_prober.go:29] interesting pod/e2e-worker-w0-08e15d-openshift-default-d579fdc75-7smjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Oct 29 22:55:31.345640 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:31.345169 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-7smjh" podUID="7bc2103e-768a-4393-9522-68b0fc370219" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Oct 29 22:55:31.480954 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:31.480919 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:55:32.345783 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:32.345747 5057 patch_prober.go:29] interesting pod/e2e-worker-w0-08e15d-openshift-default-d579fdc75-7smjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Oct 29 22:55:32.346153 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:32.345797 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-7smjh" podUID="7bc2103e-768a-4393-9522-68b0fc370219" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Oct 29 22:55:32.813120 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:32.813074 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:55:33.349980 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:33.349946 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-7smjh" Oct 29 22:55:33.350446 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:33.350366 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-7smjh" Oct 29 22:55:33.351164 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:33.351145 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-7smjh" Oct 29 22:55:34.415994 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:34.415962 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:55:35.974527 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:35.974495 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:55:37.404626 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:37.404584 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:55:38.025295 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:38.025261 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-vmj4r"] Oct 29 22:55:38.030301 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:38.030282 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-vmj4r" Oct 29 22:55:38.062163 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:38.062133 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-vmj4r"] Oct 29 22:55:38.133549 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:38.133515 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5954f1f5-c478-4f49-877e-9d3300568c2d-istio-podinfo\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-vmj4r\" (UID: \"5954f1f5-c478-4f49-877e-9d3300568c2d\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-vmj4r" Oct 29 22:55:38.133712 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:38.133674 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5954f1f5-c478-4f49-877e-9d3300568c2d-istio-data\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-vmj4r\" (UID: \"5954f1f5-c478-4f49-877e-9d3300568c2d\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-vmj4r" Oct 29 22:55:38.133760 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:38.133739 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5954f1f5-c478-4f49-877e-9d3300568c2d-istio-envoy\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-vmj4r\" (UID: \"5954f1f5-c478-4f49-877e-9d3300568c2d\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-vmj4r" Oct 29 22:55:38.133858 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:38.133833 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5954f1f5-c478-4f49-877e-9d3300568c2d-credential-socket\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-vmj4r\" (UID: \"5954f1f5-c478-4f49-877e-9d3300568c2d\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-vmj4r" Oct 29 22:55:38.133927 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:38.133911 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5954f1f5-c478-4f49-877e-9d3300568c2d-workload-socket\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-vmj4r\" (UID: \"5954f1f5-c478-4f49-877e-9d3300568c2d\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-vmj4r" Oct 29 22:55:38.133985 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:38.133970 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5954f1f5-c478-4f49-877e-9d3300568c2d-istiod-ca-cert\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-vmj4r\" (UID: \"5954f1f5-c478-4f49-877e-9d3300568c2d\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-vmj4r" Oct 29 22:55:38.134046 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:38.134034 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5954f1f5-c478-4f49-877e-9d3300568c2d-istio-token\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-vmj4r\" (UID: \"5954f1f5-c478-4f49-877e-9d3300568c2d\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-vmj4r" Oct 29 22:55:38.134184 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:38.134168 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxwzp\" (UniqueName: \"kubernetes.io/projected/5954f1f5-c478-4f49-877e-9d3300568c2d-kube-api-access-rxwzp\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-vmj4r\" (UID: \"5954f1f5-c478-4f49-877e-9d3300568c2d\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-vmj4r" Oct 29 22:55:38.134218 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:38.134206 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5954f1f5-c478-4f49-877e-9d3300568c2d-workload-certs\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-vmj4r\" (UID: \"5954f1f5-c478-4f49-877e-9d3300568c2d\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-vmj4r" Oct 29 22:55:38.235229 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:38.235171 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxwzp\" (UniqueName: \"kubernetes.io/projected/5954f1f5-c478-4f49-877e-9d3300568c2d-kube-api-access-rxwzp\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-vmj4r\" (UID: \"5954f1f5-c478-4f49-877e-9d3300568c2d\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-vmj4r" Oct 29 22:55:38.235229 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:38.235220 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5954f1f5-c478-4f49-877e-9d3300568c2d-workload-certs\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-vmj4r\" (UID: \"5954f1f5-c478-4f49-877e-9d3300568c2d\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-vmj4r" Oct 29 22:55:38.235519 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:38.235284 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5954f1f5-c478-4f49-877e-9d3300568c2d-istio-podinfo\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-vmj4r\" (UID: \"5954f1f5-c478-4f49-877e-9d3300568c2d\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-vmj4r" Oct 29 22:55:38.235519 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:38.235429 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5954f1f5-c478-4f49-877e-9d3300568c2d-istio-data\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-vmj4r\" (UID: \"5954f1f5-c478-4f49-877e-9d3300568c2d\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-vmj4r" Oct 29 22:55:38.235519 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:38.235485 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5954f1f5-c478-4f49-877e-9d3300568c2d-istio-envoy\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-vmj4r\" (UID: \"5954f1f5-c478-4f49-877e-9d3300568c2d\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-vmj4r" Oct 29 22:55:38.235519 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:38.235517 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5954f1f5-c478-4f49-877e-9d3300568c2d-credential-socket\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-vmj4r\" (UID: \"5954f1f5-c478-4f49-877e-9d3300568c2d\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-vmj4r" Oct 29 22:55:38.235743 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:38.235565 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5954f1f5-c478-4f49-877e-9d3300568c2d-workload-socket\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-vmj4r\" (UID: \"5954f1f5-c478-4f49-877e-9d3300568c2d\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-vmj4r" Oct 29 22:55:38.235743 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:38.235606 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5954f1f5-c478-4f49-877e-9d3300568c2d-istiod-ca-cert\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-vmj4r\" (UID: \"5954f1f5-c478-4f49-877e-9d3300568c2d\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-vmj4r" Oct 29 22:55:38.235743 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:38.235681 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5954f1f5-c478-4f49-877e-9d3300568c2d-istio-token\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-vmj4r\" (UID: \"5954f1f5-c478-4f49-877e-9d3300568c2d\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-vmj4r" Oct 29 22:55:38.235743 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:38.235725 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5954f1f5-c478-4f49-877e-9d3300568c2d-workload-certs\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-vmj4r\" (UID: \"5954f1f5-c478-4f49-877e-9d3300568c2d\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-vmj4r" Oct 29 22:55:38.235938 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:38.235767 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5954f1f5-c478-4f49-877e-9d3300568c2d-istio-data\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-vmj4r\" (UID: \"5954f1f5-c478-4f49-877e-9d3300568c2d\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-vmj4r" Oct 29 22:55:38.235938 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:38.235858 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5954f1f5-c478-4f49-877e-9d3300568c2d-credential-socket\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-vmj4r\" (UID: \"5954f1f5-c478-4f49-877e-9d3300568c2d\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-vmj4r" Oct 29 22:55:38.236050 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:38.236009 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5954f1f5-c478-4f49-877e-9d3300568c2d-workload-socket\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-vmj4r\" (UID: \"5954f1f5-c478-4f49-877e-9d3300568c2d\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-vmj4r" Oct 29 22:55:38.236580 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:38.236557 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5954f1f5-c478-4f49-877e-9d3300568c2d-istiod-ca-cert\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-vmj4r\" (UID: \"5954f1f5-c478-4f49-877e-9d3300568c2d\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-vmj4r" Oct 29 22:55:38.237743 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:38.237721 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5954f1f5-c478-4f49-877e-9d3300568c2d-istio-envoy\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-vmj4r\" (UID: \"5954f1f5-c478-4f49-877e-9d3300568c2d\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-vmj4r" Oct 29 22:55:38.237924 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:38.237906 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5954f1f5-c478-4f49-877e-9d3300568c2d-istio-podinfo\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-vmj4r\" (UID: \"5954f1f5-c478-4f49-877e-9d3300568c2d\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-vmj4r" Oct 29 22:55:38.256282 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:38.256259 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5954f1f5-c478-4f49-877e-9d3300568c2d-istio-token\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-vmj4r\" (UID: \"5954f1f5-c478-4f49-877e-9d3300568c2d\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-vmj4r" Oct 29 22:55:38.256700 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:38.256677 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxwzp\" (UniqueName: \"kubernetes.io/projected/5954f1f5-c478-4f49-877e-9d3300568c2d-kube-api-access-rxwzp\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-vmj4r\" (UID: \"5954f1f5-c478-4f49-877e-9d3300568c2d\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-vmj4r" Oct 29 22:55:38.348802 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:38.348725 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-vmj4r" Oct 29 22:55:38.683282 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:38.683257 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-vmj4r"] Oct 29 22:55:38.684253 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:55:38.684223 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5954f1f5_c478_4f49_877e_9d3300568c2d.slice/crio-438b58ac6bf6ef644c08c6d89e00527781bd576c4079e4d1e7ee8a109f980186 WatchSource:0}: Error finding container 438b58ac6bf6ef644c08c6d89e00527781bd576c4079e4d1e7ee8a109f980186: Status 404 returned error can't find the container with id 438b58ac6bf6ef644c08c6d89e00527781bd576c4079e4d1e7ee8a109f980186 Oct 29 22:55:38.686785 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:38.686753 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:55:38.686874 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:38.686820 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:55:38.686874 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:38.686849 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:55:38.766399 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:38.766367 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-vmj4r" event={"ID":"5954f1f5-c478-4f49-877e-9d3300568c2d","Type":"ContainerStarted","Data":"438b58ac6bf6ef644c08c6d89e00527781bd576c4079e4d1e7ee8a109f980186"} Oct 29 22:55:38.887753 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:38.887718 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:55:39.800354 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:39.795525 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-vmj4r" event={"ID":"5954f1f5-c478-4f49-877e-9d3300568c2d","Type":"ContainerStarted","Data":"90020358042d221b88ea87fbb4cccb904f0836c0747ad5df003478138a7ec83a"} Oct 29 22:55:39.850203 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:39.850149 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-vmj4r" podStartSLOduration=2.850134469 podStartE2EDuration="2.850134469s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:55:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:55:39.84907796 +0000 UTC m=+1320.632291407" watchObservedRunningTime="2026-10-29 22:55:39.850134469 +0000 UTC m=+1320.633347906" Oct 29 22:55:40.349263 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:40.349229 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-vmj4r" Oct 29 22:55:40.350500 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:40.350474 5057 patch_prober.go:29] interesting pod/e2e-worker-w0-08e15d-openshift-default-d579fdc75-vmj4r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Oct 29 22:55:40.350592 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:40.350531 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-vmj4r" podUID="5954f1f5-c478-4f49-877e-9d3300568c2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Oct 29 22:55:40.510648 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:40.510615 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:55:41.350014 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:41.349983 5057 patch_prober.go:29] interesting pod/e2e-worker-w0-08e15d-openshift-default-d579fdc75-vmj4r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Oct 29 22:55:41.350420 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:41.350032 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-vmj4r" podUID="5954f1f5-c478-4f49-877e-9d3300568c2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Oct 29 22:55:41.987302 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:41.987206 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:55:42.349873 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:42.349781 5057 patch_prober.go:29] interesting pod/e2e-worker-w0-08e15d-openshift-default-d579fdc75-vmj4r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Oct 29 22:55:42.349873 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:42.349844 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-vmj4r" podUID="5954f1f5-c478-4f49-877e-9d3300568c2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Oct 29 22:55:43.268215 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:43.268175 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:55:43.349204 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:43.349171 5057 patch_prober.go:29] interesting pod/e2e-worker-w0-08e15d-openshift-default-d579fdc75-vmj4r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Oct 29 22:55:43.349371 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:43.349228 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-vmj4r" podUID="5954f1f5-c478-4f49-877e-9d3300568c2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Oct 29 22:55:43.591960 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:43.591876 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:55:43.734686 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:43.734639 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:55:43.854971 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:43.854881 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:55:44.349599 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:44.349568 5057 patch_prober.go:29] interesting pod/e2e-worker-w0-08e15d-openshift-default-d579fdc75-vmj4r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Oct 29 22:55:44.349788 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:44.349619 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-vmj4r" podUID="5954f1f5-c478-4f49-877e-9d3300568c2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Oct 29 22:55:45.290995 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:45.290955 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:55:45.349316 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:45.349283 5057 patch_prober.go:29] interesting pod/e2e-worker-w0-08e15d-openshift-default-d579fdc75-vmj4r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Oct 29 22:55:45.349592 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:45.349556 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-vmj4r" podUID="5954f1f5-c478-4f49-877e-9d3300568c2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Oct 29 22:55:45.437183 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:45.437123 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:55:45.557981 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:45.557891 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:55:45.705473 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:45.685254 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:55:46.353973 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:46.353937 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-vmj4r" Oct 29 22:55:46.354461 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:46.354238 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-vmj4r" Oct 29 22:55:46.355027 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:46.355007 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-vmj4r" Oct 29 22:55:46.642857 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:46.642773 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:55:46.756715 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:46.756681 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:55:46.872073 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:46.872035 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:55:46.990013 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:46.989974 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:55:47.326278 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:47.326198 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:55:47.765651 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:47.765615 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:55:47.877683 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:47.877645 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:55:48.879452 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:48.879409 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:55:49.050377 ip-10-0-140-74 kubenswrapper[5057]: E1029 22:55:49.050324 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d2dnn" containerName="extractor" Oct 29 22:55:49.466936 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:49.466901 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:55:49.540722 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:49.540688 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-k8ws5"] Oct 29 22:55:49.546984 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:49.546963 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-k8ws5" Oct 29 22:55:49.569057 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:49.569025 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-k8ws5"] Oct 29 22:55:49.596127 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:49.596090 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:55:49.646659 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:49.646627 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-f3092a4d-7745bbd6ff-vsstw"] Oct 29 22:55:49.651264 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:49.651244 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-f3092a4d-7745bbd6ff-vsstw" Oct 29 22:55:49.671410 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:49.671383 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-f3092a4d\"" Oct 29 22:55:49.672427 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:49.672402 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-f3092a4d-7745bbd6ff-vsstw"] Oct 29 22:55:49.676146 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:49.676121 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f53c1835-d1af-41c6-b8b9-e749ed7ef210-istio-podinfo\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-k8ws5\" (UID: \"f53c1835-d1af-41c6-b8b9-e749ed7ef210\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-k8ws5" Oct 29 22:55:49.676278 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:49.676156 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f53c1835-d1af-41c6-b8b9-e749ed7ef210-istio-data\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-k8ws5\" (UID: \"f53c1835-d1af-41c6-b8b9-e749ed7ef210\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-k8ws5" Oct 29 22:55:49.676278 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:49.676187 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f53c1835-d1af-41c6-b8b9-e749ed7ef210-workload-certs\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-k8ws5\" (UID: \"f53c1835-d1af-41c6-b8b9-e749ed7ef210\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-k8ws5" Oct 29 22:55:49.676278 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:49.676231 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87scs\" (UniqueName: \"kubernetes.io/projected/f53c1835-d1af-41c6-b8b9-e749ed7ef210-kube-api-access-87scs\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-k8ws5\" (UID: \"f53c1835-d1af-41c6-b8b9-e749ed7ef210\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-k8ws5" Oct 29 22:55:49.676478 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:49.676426 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f53c1835-d1af-41c6-b8b9-e749ed7ef210-workload-socket\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-k8ws5\" (UID: \"f53c1835-d1af-41c6-b8b9-e749ed7ef210\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-k8ws5" Oct 29 22:55:49.676478 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:49.676450 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f53c1835-d1af-41c6-b8b9-e749ed7ef210-istio-envoy\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-k8ws5\" (UID: \"f53c1835-d1af-41c6-b8b9-e749ed7ef210\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-k8ws5" Oct 29 22:55:49.676617 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:49.676600 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f53c1835-d1af-41c6-b8b9-e749ed7ef210-istiod-ca-cert\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-k8ws5\" (UID: \"f53c1835-d1af-41c6-b8b9-e749ed7ef210\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-k8ws5" Oct 29 22:55:49.676834 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:49.676815 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f53c1835-d1af-41c6-b8b9-e749ed7ef210-credential-socket\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-k8ws5\" (UID: \"f53c1835-d1af-41c6-b8b9-e749ed7ef210\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-k8ws5" Oct 29 22:55:49.676902 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:49.676857 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f53c1835-d1af-41c6-b8b9-e749ed7ef210-istio-token\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-k8ws5\" (UID: \"f53c1835-d1af-41c6-b8b9-e749ed7ef210\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-k8ws5" Oct 29 22:55:49.696809 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:49.696772 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-f3092a4d-dockercfg-v9j6d\"" Oct 29 22:55:49.739851 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:49.739785 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f3092a4d-7f9bb6f4fd-2kck8"] Oct 29 22:55:49.744476 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:49.744453 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-f3092a4d-7f9bb6f4fd-2kck8" Oct 29 22:55:49.764014 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:49.763982 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f3092a4d-7f9bb6f4fd-2kck8"] Oct 29 22:55:49.801108 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:49.798183 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pv76g\" (UniqueName: \"kubernetes.io/projected/ccfbe3ea-306d-4c9f-be5c-19317c09847d-kube-api-access-pv76g\") pod \"payload-pre-processing-e2e-mt-f3092a4d-7745bbd6ff-vsstw\" (UID: \"ccfbe3ea-306d-4c9f-be5c-19317c09847d\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-f3092a4d-7745bbd6ff-vsstw" Oct 29 22:55:49.801108 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:49.798281 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f53c1835-d1af-41c6-b8b9-e749ed7ef210-workload-socket\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-k8ws5\" (UID: \"f53c1835-d1af-41c6-b8b9-e749ed7ef210\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-k8ws5" Oct 29 22:55:49.801108 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:49.798308 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f53c1835-d1af-41c6-b8b9-e749ed7ef210-istio-envoy\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-k8ws5\" (UID: \"f53c1835-d1af-41c6-b8b9-e749ed7ef210\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-k8ws5" Oct 29 22:55:49.801108 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:49.798361 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4b2f080-3a05-48e3-b667-ba1e4f93c5d5-plugins-config-volume\") pod \"payload-processing-e2e-mt-f3092a4d-7f9bb6f4fd-2kck8\" (UID: \"e4b2f080-3a05-48e3-b667-ba1e4f93c5d5\") " pod="openshift-ingress/payload-processing-e2e-mt-f3092a4d-7f9bb6f4fd-2kck8" Oct 29 22:55:49.801108 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:49.798409 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ccfbe3ea-306d-4c9f-be5c-19317c09847d-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-f3092a4d-7745bbd6ff-vsstw\" (UID: \"ccfbe3ea-306d-4c9f-be5c-19317c09847d\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-f3092a4d-7745bbd6ff-vsstw" Oct 29 22:55:49.801108 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:49.798487 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5klq\" (UniqueName: \"kubernetes.io/projected/e4b2f080-3a05-48e3-b667-ba1e4f93c5d5-kube-api-access-t5klq\") pod \"payload-processing-e2e-mt-f3092a4d-7f9bb6f4fd-2kck8\" (UID: \"e4b2f080-3a05-48e3-b667-ba1e4f93c5d5\") " pod="openshift-ingress/payload-processing-e2e-mt-f3092a4d-7f9bb6f4fd-2kck8" Oct 29 22:55:49.801108 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:49.798535 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f53c1835-d1af-41c6-b8b9-e749ed7ef210-istiod-ca-cert\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-k8ws5\" (UID: \"f53c1835-d1af-41c6-b8b9-e749ed7ef210\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-k8ws5" Oct 29 22:55:49.801108 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:49.798578 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f53c1835-d1af-41c6-b8b9-e749ed7ef210-credential-socket\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-k8ws5\" (UID: \"f53c1835-d1af-41c6-b8b9-e749ed7ef210\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-k8ws5" Oct 29 22:55:49.801108 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:49.798616 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f53c1835-d1af-41c6-b8b9-e749ed7ef210-istio-token\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-k8ws5\" (UID: \"f53c1835-d1af-41c6-b8b9-e749ed7ef210\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-k8ws5" Oct 29 22:55:49.801108 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:49.798681 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f53c1835-d1af-41c6-b8b9-e749ed7ef210-istio-podinfo\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-k8ws5\" (UID: \"f53c1835-d1af-41c6-b8b9-e749ed7ef210\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-k8ws5" Oct 29 22:55:49.801108 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:49.798714 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f53c1835-d1af-41c6-b8b9-e749ed7ef210-istio-data\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-k8ws5\" (UID: \"f53c1835-d1af-41c6-b8b9-e749ed7ef210\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-k8ws5" Oct 29 22:55:49.801108 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:49.798753 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f53c1835-d1af-41c6-b8b9-e749ed7ef210-workload-certs\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-k8ws5\" (UID: \"f53c1835-d1af-41c6-b8b9-e749ed7ef210\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-k8ws5" Oct 29 22:55:49.801108 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:49.798774 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87scs\" (UniqueName: \"kubernetes.io/projected/f53c1835-d1af-41c6-b8b9-e749ed7ef210-kube-api-access-87scs\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-k8ws5\" (UID: \"f53c1835-d1af-41c6-b8b9-e749ed7ef210\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-k8ws5" Oct 29 22:55:49.801108 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:49.799577 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f53c1835-d1af-41c6-b8b9-e749ed7ef210-workload-socket\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-k8ws5\" (UID: \"f53c1835-d1af-41c6-b8b9-e749ed7ef210\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-k8ws5" Oct 29 22:55:49.801108 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:49.800201 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f53c1835-d1af-41c6-b8b9-e749ed7ef210-credential-socket\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-k8ws5\" (UID: \"f53c1835-d1af-41c6-b8b9-e749ed7ef210\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-k8ws5" Oct 29 22:55:49.801108 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:49.800915 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f53c1835-d1af-41c6-b8b9-e749ed7ef210-istiod-ca-cert\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-k8ws5\" (UID: \"f53c1835-d1af-41c6-b8b9-e749ed7ef210\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-k8ws5" Oct 29 22:55:49.804498 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:49.802083 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f53c1835-d1af-41c6-b8b9-e749ed7ef210-workload-certs\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-k8ws5\" (UID: \"f53c1835-d1af-41c6-b8b9-e749ed7ef210\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-k8ws5" Oct 29 22:55:49.812732 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:49.812703 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f53c1835-d1af-41c6-b8b9-e749ed7ef210-istio-data\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-k8ws5\" (UID: \"f53c1835-d1af-41c6-b8b9-e749ed7ef210\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-k8ws5" Oct 29 22:55:49.844453 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:49.839268 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f53c1835-d1af-41c6-b8b9-e749ed7ef210-istio-token\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-k8ws5\" (UID: \"f53c1835-d1af-41c6-b8b9-e749ed7ef210\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-k8ws5" Oct 29 22:55:49.848298 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:49.848268 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-87scs\" (UniqueName: \"kubernetes.io/projected/f53c1835-d1af-41c6-b8b9-e749ed7ef210-kube-api-access-87scs\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-k8ws5\" (UID: \"f53c1835-d1af-41c6-b8b9-e749ed7ef210\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-k8ws5" Oct 29 22:55:49.853437 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:49.853400 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f53c1835-d1af-41c6-b8b9-e749ed7ef210-istio-envoy\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-k8ws5\" (UID: \"f53c1835-d1af-41c6-b8b9-e749ed7ef210\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-k8ws5" Oct 29 22:55:49.855915 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:49.855888 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f53c1835-d1af-41c6-b8b9-e749ed7ef210-istio-podinfo\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-k8ws5\" (UID: \"f53c1835-d1af-41c6-b8b9-e749ed7ef210\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-k8ws5" Oct 29 22:55:49.871355 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:49.869685 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-k8ws5" Oct 29 22:55:49.900247 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:49.900211 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ccfbe3ea-306d-4c9f-be5c-19317c09847d-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-f3092a4d-7745bbd6ff-vsstw\" (UID: \"ccfbe3ea-306d-4c9f-be5c-19317c09847d\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-f3092a4d-7745bbd6ff-vsstw" Oct 29 22:55:49.900696 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:49.900318 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5klq\" (UniqueName: \"kubernetes.io/projected/e4b2f080-3a05-48e3-b667-ba1e4f93c5d5-kube-api-access-t5klq\") pod \"payload-processing-e2e-mt-f3092a4d-7f9bb6f4fd-2kck8\" (UID: \"e4b2f080-3a05-48e3-b667-ba1e4f93c5d5\") " pod="openshift-ingress/payload-processing-e2e-mt-f3092a4d-7f9bb6f4fd-2kck8" Oct 29 22:55:49.900696 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:49.900533 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pv76g\" (UniqueName: \"kubernetes.io/projected/ccfbe3ea-306d-4c9f-be5c-19317c09847d-kube-api-access-pv76g\") pod \"payload-pre-processing-e2e-mt-f3092a4d-7745bbd6ff-vsstw\" (UID: \"ccfbe3ea-306d-4c9f-be5c-19317c09847d\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-f3092a4d-7745bbd6ff-vsstw" Oct 29 22:55:49.900696 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:49.900606 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4b2f080-3a05-48e3-b667-ba1e4f93c5d5-plugins-config-volume\") pod \"payload-processing-e2e-mt-f3092a4d-7f9bb6f4fd-2kck8\" (UID: \"e4b2f080-3a05-48e3-b667-ba1e4f93c5d5\") " pod="openshift-ingress/payload-processing-e2e-mt-f3092a4d-7f9bb6f4fd-2kck8" Oct 29 22:55:49.901501 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:49.901478 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4b2f080-3a05-48e3-b667-ba1e4f93c5d5-plugins-config-volume\") pod \"payload-processing-e2e-mt-f3092a4d-7f9bb6f4fd-2kck8\" (UID: \"e4b2f080-3a05-48e3-b667-ba1e4f93c5d5\") " pod="openshift-ingress/payload-processing-e2e-mt-f3092a4d-7f9bb6f4fd-2kck8" Oct 29 22:55:49.902217 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:49.902192 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ccfbe3ea-306d-4c9f-be5c-19317c09847d-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-f3092a4d-7745bbd6ff-vsstw\" (UID: \"ccfbe3ea-306d-4c9f-be5c-19317c09847d\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-f3092a4d-7745bbd6ff-vsstw" Oct 29 22:55:49.928493 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:49.926030 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pv76g\" (UniqueName: \"kubernetes.io/projected/ccfbe3ea-306d-4c9f-be5c-19317c09847d-kube-api-access-pv76g\") pod \"payload-pre-processing-e2e-mt-f3092a4d-7745bbd6ff-vsstw\" (UID: \"ccfbe3ea-306d-4c9f-be5c-19317c09847d\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-f3092a4d-7745bbd6ff-vsstw" Oct 29 22:55:49.930736 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:49.930664 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5klq\" (UniqueName: \"kubernetes.io/projected/e4b2f080-3a05-48e3-b667-ba1e4f93c5d5-kube-api-access-t5klq\") pod \"payload-processing-e2e-mt-f3092a4d-7f9bb6f4fd-2kck8\" (UID: \"e4b2f080-3a05-48e3-b667-ba1e4f93c5d5\") " pod="openshift-ingress/payload-processing-e2e-mt-f3092a4d-7f9bb6f4fd-2kck8" Oct 29 22:55:49.979477 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:49.967045 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-f3092a4d-7745bbd6ff-vsstw" Oct 29 22:55:50.063072 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:50.062699 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-f3092a4d-7f9bb6f4fd-2kck8" Oct 29 22:55:50.246641 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:55:50.246609 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf53c1835_d1af_41c6_b8b9_e749ed7ef210.slice/crio-f90015118699f9baa22bde68e99fb77de0bb2cd118777d21ce62d47dc1d7ba8b WatchSource:0}: Error finding container f90015118699f9baa22bde68e99fb77de0bb2cd118777d21ce62d47dc1d7ba8b: Status 404 returned error can't find the container with id f90015118699f9baa22bde68e99fb77de0bb2cd118777d21ce62d47dc1d7ba8b Oct 29 22:55:50.248675 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:50.248613 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-k8ws5"] Oct 29 22:55:50.250265 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:50.250232 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:55:50.250384 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:50.250292 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:55:50.250384 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:50.250324 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:55:50.321979 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:50.321909 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-f3092a4d-7745bbd6ff-vsstw"] Oct 29 22:55:50.434883 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:50.434820 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f3092a4d-7f9bb6f4fd-2kck8"] Oct 29 22:55:50.436605 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:55:50.436576 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4b2f080_3a05_48e3_b667_ba1e4f93c5d5.slice/crio-8bf71b7181d8beb5c24d50342a1d54f33397c351ec77f86cb8c22a9f65f15ffe WatchSource:0}: Error finding container 8bf71b7181d8beb5c24d50342a1d54f33397c351ec77f86cb8c22a9f65f15ffe: Status 404 returned error can't find the container with id 8bf71b7181d8beb5c24d50342a1d54f33397c351ec77f86cb8c22a9f65f15ffe Oct 29 22:55:50.569109 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:50.569078 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:55:50.894577 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:50.894448 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-k8ws5" event={"ID":"f53c1835-d1af-41c6-b8b9-e749ed7ef210","Type":"ContainerStarted","Data":"7351d791569edd66770f39ec0b0b3b6a545aac98c0cfed385b0bbb3a82f07022"} Oct 29 22:55:50.894577 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:50.894517 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-k8ws5" event={"ID":"f53c1835-d1af-41c6-b8b9-e749ed7ef210","Type":"ContainerStarted","Data":"f90015118699f9baa22bde68e99fb77de0bb2cd118777d21ce62d47dc1d7ba8b"} Oct 29 22:55:50.896409 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:50.896375 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-f3092a4d-7745bbd6ff-vsstw" event={"ID":"ccfbe3ea-306d-4c9f-be5c-19317c09847d","Type":"ContainerStarted","Data":"72f34310c3ae1e29e98daeaef74a23e95182776d384b62684be8959a6e57e98f"} Oct 29 22:55:50.896563 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:50.896433 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-f3092a4d-7745bbd6ff-vsstw" event={"ID":"ccfbe3ea-306d-4c9f-be5c-19317c09847d","Type":"ContainerStarted","Data":"724c9a2b57615acabfc5a3c76958418f0e898b48a2e17caa5d87b8da958d22cd"} Oct 29 22:55:50.896563 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:50.896466 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-f3092a4d-7745bbd6ff-vsstw" Oct 29 22:55:50.898933 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:50.898908 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-f3092a4d-7f9bb6f4fd-2kck8" event={"ID":"e4b2f080-3a05-48e3-b667-ba1e4f93c5d5","Type":"ContainerStarted","Data":"e5078bda077aeb4401c31ac98653e12ce108bfd69d78ca65bb93b3f8e1f0dd7c"} Oct 29 22:55:50.899036 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:50.898959 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-f3092a4d-7f9bb6f4fd-2kck8" event={"ID":"e4b2f080-3a05-48e3-b667-ba1e4f93c5d5","Type":"ContainerStarted","Data":"8bf71b7181d8beb5c24d50342a1d54f33397c351ec77f86cb8c22a9f65f15ffe"} Oct 29 22:55:50.899036 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:50.898988 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-f3092a4d-7f9bb6f4fd-2kck8" Oct 29 22:55:50.939259 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:50.939209 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-k8ws5" podStartSLOduration=1.939195059 podStartE2EDuration="1.939195059s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:55:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:55:50.936496135 +0000 UTC m=+1331.719709561" watchObservedRunningTime="2026-10-29 22:55:50.939195059 +0000 UTC m=+1331.722408489" Oct 29 22:55:50.975034 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:50.974963 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-f3092a4d-7f9bb6f4fd-2kck8" podStartSLOduration=1.974950891 podStartE2EDuration="1.974950891s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:55:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:55:50.973153469 +0000 UTC m=+1331.756366905" watchObservedRunningTime="2026-10-29 22:55:50.974950891 +0000 UTC m=+1331.758164310" Oct 29 22:55:51.013210 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:51.013161 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-f3092a4d-7745bbd6ff-vsstw" podStartSLOduration=2.013146857 podStartE2EDuration="2.013146857s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:55:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:55:51.009307817 +0000 UTC m=+1331.792521243" watchObservedRunningTime="2026-10-29 22:55:51.013146857 +0000 UTC m=+1331.796360320" Oct 29 22:55:51.097321 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:51.097263 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-bclh8"] Oct 29 22:55:51.110608 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:51.108226 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-bclh8" Oct 29 22:55:51.140586 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:51.140501 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-bclh8"] Oct 29 22:55:51.240567 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:51.240520 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b510ea66-642c-49ae-a600-fab375005cb6-workload-socket\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-bclh8\" (UID: \"b510ea66-642c-49ae-a600-fab375005cb6\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-bclh8" Oct 29 22:55:51.240754 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:51.240631 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xr25\" (UniqueName: \"kubernetes.io/projected/b510ea66-642c-49ae-a600-fab375005cb6-kube-api-access-9xr25\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-bclh8\" (UID: \"b510ea66-642c-49ae-a600-fab375005cb6\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-bclh8" Oct 29 22:55:51.240754 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:51.240729 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b510ea66-642c-49ae-a600-fab375005cb6-istio-envoy\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-bclh8\" (UID: \"b510ea66-642c-49ae-a600-fab375005cb6\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-bclh8" Oct 29 22:55:51.240872 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:51.240772 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b510ea66-642c-49ae-a600-fab375005cb6-istiod-ca-cert\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-bclh8\" (UID: \"b510ea66-642c-49ae-a600-fab375005cb6\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-bclh8" Oct 29 22:55:51.240925 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:51.240898 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b510ea66-642c-49ae-a600-fab375005cb6-istio-data\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-bclh8\" (UID: \"b510ea66-642c-49ae-a600-fab375005cb6\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-bclh8" Oct 29 22:55:51.241011 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:51.240993 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b510ea66-642c-49ae-a600-fab375005cb6-workload-certs\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-bclh8\" (UID: \"b510ea66-642c-49ae-a600-fab375005cb6\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-bclh8" Oct 29 22:55:51.241077 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:51.241029 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b510ea66-642c-49ae-a600-fab375005cb6-istio-token\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-bclh8\" (UID: \"b510ea66-642c-49ae-a600-fab375005cb6\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-bclh8" Oct 29 22:55:51.241136 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:51.241078 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b510ea66-642c-49ae-a600-fab375005cb6-credential-socket\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-bclh8\" (UID: \"b510ea66-642c-49ae-a600-fab375005cb6\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-bclh8" Oct 29 22:55:51.241136 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:51.241124 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b510ea66-642c-49ae-a600-fab375005cb6-istio-podinfo\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-bclh8\" (UID: \"b510ea66-642c-49ae-a600-fab375005cb6\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-bclh8" Oct 29 22:55:51.342353 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:51.342310 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b510ea66-642c-49ae-a600-fab375005cb6-istio-data\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-bclh8\" (UID: \"b510ea66-642c-49ae-a600-fab375005cb6\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-bclh8" Oct 29 22:55:51.342539 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:51.342424 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b510ea66-642c-49ae-a600-fab375005cb6-workload-certs\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-bclh8\" (UID: \"b510ea66-642c-49ae-a600-fab375005cb6\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-bclh8" Oct 29 22:55:51.342539 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:51.342448 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b510ea66-642c-49ae-a600-fab375005cb6-istio-token\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-bclh8\" (UID: \"b510ea66-642c-49ae-a600-fab375005cb6\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-bclh8" Oct 29 22:55:51.342539 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:51.342479 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b510ea66-642c-49ae-a600-fab375005cb6-credential-socket\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-bclh8\" (UID: \"b510ea66-642c-49ae-a600-fab375005cb6\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-bclh8" Oct 29 22:55:51.342723 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:51.342612 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b510ea66-642c-49ae-a600-fab375005cb6-istio-podinfo\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-bclh8\" (UID: \"b510ea66-642c-49ae-a600-fab375005cb6\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-bclh8" Oct 29 22:55:51.342723 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:51.342664 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b510ea66-642c-49ae-a600-fab375005cb6-workload-socket\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-bclh8\" (UID: \"b510ea66-642c-49ae-a600-fab375005cb6\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-bclh8" Oct 29 22:55:51.342831 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:51.342725 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xr25\" (UniqueName: \"kubernetes.io/projected/b510ea66-642c-49ae-a600-fab375005cb6-kube-api-access-9xr25\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-bclh8\" (UID: \"b510ea66-642c-49ae-a600-fab375005cb6\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-bclh8" Oct 29 22:55:51.342831 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:51.342779 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b510ea66-642c-49ae-a600-fab375005cb6-istio-data\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-bclh8\" (UID: \"b510ea66-642c-49ae-a600-fab375005cb6\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-bclh8" Oct 29 22:55:51.342831 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:51.342793 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b510ea66-642c-49ae-a600-fab375005cb6-credential-socket\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-bclh8\" (UID: \"b510ea66-642c-49ae-a600-fab375005cb6\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-bclh8" Oct 29 22:55:51.342831 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:51.342799 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b510ea66-642c-49ae-a600-fab375005cb6-istio-envoy\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-bclh8\" (UID: \"b510ea66-642c-49ae-a600-fab375005cb6\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-bclh8" Oct 29 22:55:51.343033 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:51.342892 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b510ea66-642c-49ae-a600-fab375005cb6-workload-certs\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-bclh8\" (UID: \"b510ea66-642c-49ae-a600-fab375005cb6\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-bclh8" Oct 29 22:55:51.343033 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:51.342986 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b510ea66-642c-49ae-a600-fab375005cb6-istiod-ca-cert\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-bclh8\" (UID: \"b510ea66-642c-49ae-a600-fab375005cb6\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-bclh8" Oct 29 22:55:51.343117 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:51.343060 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b510ea66-642c-49ae-a600-fab375005cb6-workload-socket\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-bclh8\" (UID: \"b510ea66-642c-49ae-a600-fab375005cb6\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-bclh8" Oct 29 22:55:51.343577 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:51.343555 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b510ea66-642c-49ae-a600-fab375005cb6-istiod-ca-cert\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-bclh8\" (UID: \"b510ea66-642c-49ae-a600-fab375005cb6\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-bclh8" Oct 29 22:55:51.345147 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:51.345102 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b510ea66-642c-49ae-a600-fab375005cb6-istio-envoy\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-bclh8\" (UID: \"b510ea66-642c-49ae-a600-fab375005cb6\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-bclh8" Oct 29 22:55:51.345280 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:51.345263 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b510ea66-642c-49ae-a600-fab375005cb6-istio-podinfo\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-bclh8\" (UID: \"b510ea66-642c-49ae-a600-fab375005cb6\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-bclh8" Oct 29 22:55:51.363967 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:51.363936 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b510ea66-642c-49ae-a600-fab375005cb6-istio-token\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-bclh8\" (UID: \"b510ea66-642c-49ae-a600-fab375005cb6\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-bclh8" Oct 29 22:55:51.364109 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:51.364090 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xr25\" (UniqueName: \"kubernetes.io/projected/b510ea66-642c-49ae-a600-fab375005cb6-kube-api-access-9xr25\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-bclh8\" (UID: \"b510ea66-642c-49ae-a600-fab375005cb6\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-bclh8" Oct 29 22:55:51.518145 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:51.517884 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-bclh8" Oct 29 22:55:51.614110 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:51.608394 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-kk86b"] Oct 29 22:55:51.636804 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:51.636775 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-kk86b" Oct 29 22:55:51.638846 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:51.638816 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-kk86b"] Oct 29 22:55:51.751574 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:51.751534 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79d262bd-3d59-44ec-b375-28e6c7a57c1c-workload-certs\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-kk86b\" (UID: \"79d262bd-3d59-44ec-b375-28e6c7a57c1c\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-kk86b" Oct 29 22:55:51.751747 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:51.751637 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79d262bd-3d59-44ec-b375-28e6c7a57c1c-credential-socket\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-kk86b\" (UID: \"79d262bd-3d59-44ec-b375-28e6c7a57c1c\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-kk86b" Oct 29 22:55:51.751747 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:51.751681 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79d262bd-3d59-44ec-b375-28e6c7a57c1c-istio-podinfo\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-kk86b\" (UID: \"79d262bd-3d59-44ec-b375-28e6c7a57c1c\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-kk86b" Oct 29 22:55:51.751747 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:51.751731 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79d262bd-3d59-44ec-b375-28e6c7a57c1c-istio-data\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-kk86b\" (UID: \"79d262bd-3d59-44ec-b375-28e6c7a57c1c\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-kk86b" Oct 29 22:55:51.751933 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:51.751756 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79d262bd-3d59-44ec-b375-28e6c7a57c1c-istiod-ca-cert\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-kk86b\" (UID: \"79d262bd-3d59-44ec-b375-28e6c7a57c1c\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-kk86b" Oct 29 22:55:51.751933 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:51.751789 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wnnr9\" (UniqueName: \"kubernetes.io/projected/79d262bd-3d59-44ec-b375-28e6c7a57c1c-kube-api-access-wnnr9\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-kk86b\" (UID: \"79d262bd-3d59-44ec-b375-28e6c7a57c1c\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-kk86b" Oct 29 22:55:51.752030 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:51.752003 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79d262bd-3d59-44ec-b375-28e6c7a57c1c-workload-socket\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-kk86b\" (UID: \"79d262bd-3d59-44ec-b375-28e6c7a57c1c\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-kk86b" Oct 29 22:55:51.752084 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:51.752031 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79d262bd-3d59-44ec-b375-28e6c7a57c1c-istio-token\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-kk86b\" (UID: \"79d262bd-3d59-44ec-b375-28e6c7a57c1c\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-kk86b" Oct 29 22:55:51.752138 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:51.752086 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79d262bd-3d59-44ec-b375-28e6c7a57c1c-istio-envoy\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-kk86b\" (UID: \"79d262bd-3d59-44ec-b375-28e6c7a57c1c\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-kk86b" Oct 29 22:55:51.853780 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:51.853708 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79d262bd-3d59-44ec-b375-28e6c7a57c1c-istio-data\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-kk86b\" (UID: \"79d262bd-3d59-44ec-b375-28e6c7a57c1c\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-kk86b" Oct 29 22:55:51.853780 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:51.853748 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79d262bd-3d59-44ec-b375-28e6c7a57c1c-istiod-ca-cert\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-kk86b\" (UID: \"79d262bd-3d59-44ec-b375-28e6c7a57c1c\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-kk86b" Oct 29 22:55:51.853780 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:51.853770 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wnnr9\" (UniqueName: \"kubernetes.io/projected/79d262bd-3d59-44ec-b375-28e6c7a57c1c-kube-api-access-wnnr9\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-kk86b\" (UID: \"79d262bd-3d59-44ec-b375-28e6c7a57c1c\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-kk86b" Oct 29 22:55:51.854071 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:51.853869 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79d262bd-3d59-44ec-b375-28e6c7a57c1c-workload-socket\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-kk86b\" (UID: \"79d262bd-3d59-44ec-b375-28e6c7a57c1c\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-kk86b" Oct 29 22:55:51.854071 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:51.853890 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79d262bd-3d59-44ec-b375-28e6c7a57c1c-istio-token\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-kk86b\" (UID: \"79d262bd-3d59-44ec-b375-28e6c7a57c1c\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-kk86b" Oct 29 22:55:51.854174 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:51.853933 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79d262bd-3d59-44ec-b375-28e6c7a57c1c-istio-envoy\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-kk86b\" (UID: \"79d262bd-3d59-44ec-b375-28e6c7a57c1c\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-kk86b" Oct 29 22:55:51.854227 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:51.854178 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79d262bd-3d59-44ec-b375-28e6c7a57c1c-istio-data\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-kk86b\" (UID: \"79d262bd-3d59-44ec-b375-28e6c7a57c1c\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-kk86b" Oct 29 22:55:51.854279 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:51.854258 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79d262bd-3d59-44ec-b375-28e6c7a57c1c-workload-socket\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-kk86b\" (UID: \"79d262bd-3d59-44ec-b375-28e6c7a57c1c\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-kk86b" Oct 29 22:55:51.854372 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:51.854324 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79d262bd-3d59-44ec-b375-28e6c7a57c1c-workload-certs\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-kk86b\" (UID: \"79d262bd-3d59-44ec-b375-28e6c7a57c1c\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-kk86b" Oct 29 22:55:51.854502 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:51.854484 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79d262bd-3d59-44ec-b375-28e6c7a57c1c-credential-socket\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-kk86b\" (UID: \"79d262bd-3d59-44ec-b375-28e6c7a57c1c\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-kk86b" Oct 29 22:55:51.854560 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:51.854528 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79d262bd-3d59-44ec-b375-28e6c7a57c1c-istio-podinfo\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-kk86b\" (UID: \"79d262bd-3d59-44ec-b375-28e6c7a57c1c\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-kk86b" Oct 29 22:55:51.854560 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:51.854545 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79d262bd-3d59-44ec-b375-28e6c7a57c1c-workload-certs\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-kk86b\" (UID: \"79d262bd-3d59-44ec-b375-28e6c7a57c1c\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-kk86b" Oct 29 22:55:51.854660 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:51.854575 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79d262bd-3d59-44ec-b375-28e6c7a57c1c-istiod-ca-cert\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-kk86b\" (UID: \"79d262bd-3d59-44ec-b375-28e6c7a57c1c\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-kk86b" Oct 29 22:55:51.854786 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:51.854768 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79d262bd-3d59-44ec-b375-28e6c7a57c1c-credential-socket\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-kk86b\" (UID: \"79d262bd-3d59-44ec-b375-28e6c7a57c1c\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-kk86b" Oct 29 22:55:51.856527 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:51.856496 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79d262bd-3d59-44ec-b375-28e6c7a57c1c-istio-envoy\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-kk86b\" (UID: \"79d262bd-3d59-44ec-b375-28e6c7a57c1c\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-kk86b" Oct 29 22:55:51.857005 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:51.856982 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79d262bd-3d59-44ec-b375-28e6c7a57c1c-istio-podinfo\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-kk86b\" (UID: \"79d262bd-3d59-44ec-b375-28e6c7a57c1c\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-kk86b" Oct 29 22:55:51.870796 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:51.870771 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-k8ws5" Oct 29 22:55:51.872416 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:51.872397 5057 patch_prober.go:29] interesting pod/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-k8ws5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Oct 29 22:55:51.872505 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:51.872437 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-k8ws5" podUID="f53c1835-d1af-41c6-b8b9-e749ed7ef210" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Oct 29 22:55:51.875376 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:51.875354 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wnnr9\" (UniqueName: \"kubernetes.io/projected/79d262bd-3d59-44ec-b375-28e6c7a57c1c-kube-api-access-wnnr9\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-kk86b\" (UID: \"79d262bd-3d59-44ec-b375-28e6c7a57c1c\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-kk86b" Oct 29 22:55:51.876036 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:51.876019 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79d262bd-3d59-44ec-b375-28e6c7a57c1c-istio-token\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-kk86b\" (UID: \"79d262bd-3d59-44ec-b375-28e6c7a57c1c\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-kk86b" Oct 29 22:55:51.919798 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:51.919762 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-bclh8"] Oct 29 22:55:51.920268 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:55:51.920240 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb510ea66_642c_49ae_a600_fab375005cb6.slice/crio-6c5a7f8eaa61c80ec547b019fc877bd764fea3599280aae13062b245b82c87e0 WatchSource:0}: Error finding container 6c5a7f8eaa61c80ec547b019fc877bd764fea3599280aae13062b245b82c87e0: Status 404 returned error can't find the container with id 6c5a7f8eaa61c80ec547b019fc877bd764fea3599280aae13062b245b82c87e0 Oct 29 22:55:51.922887 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:51.922845 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:55:51.922986 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:51.922935 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:55:51.923040 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:51.922980 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:55:51.961968 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:51.961945 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-kk86b" Oct 29 22:55:52.229631 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:52.229590 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:55:52.339633 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:52.339550 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-kk86b"] Oct 29 22:55:52.348194 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:52.347552 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:55:52.348194 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:52.347645 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:55:52.348194 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:52.347697 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:55:52.871921 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:52.871874 5057 patch_prober.go:29] interesting pod/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-k8ws5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Oct 29 22:55:52.873683 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:52.873554 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-k8ws5" podUID="f53c1835-d1af-41c6-b8b9-e749ed7ef210" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Oct 29 22:55:52.957471 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:52.957365 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-bclh8" event={"ID":"b510ea66-642c-49ae-a600-fab375005cb6","Type":"ContainerStarted","Data":"cdd38dbe12d687c21c3991a8a66a66291152d9ca48c363c07b83c4c1846ac9b1"} Oct 29 22:55:52.957471 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:52.957433 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-bclh8" event={"ID":"b510ea66-642c-49ae-a600-fab375005cb6","Type":"ContainerStarted","Data":"6c5a7f8eaa61c80ec547b019fc877bd764fea3599280aae13062b245b82c87e0"} Oct 29 22:55:52.958893 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:52.958861 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-kk86b" event={"ID":"79d262bd-3d59-44ec-b375-28e6c7a57c1c","Type":"ContainerStarted","Data":"48a12c1f45b59989923a819fd04006a6c7dec61a36e3b7f7579aeffe08f209a0"} Oct 29 22:55:52.959004 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:52.958919 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-kk86b" event={"ID":"79d262bd-3d59-44ec-b375-28e6c7a57c1c","Type":"ContainerStarted","Data":"27307c9c034ca20eab5c21fbbeae0d9c574c12944ade8299055e499741abbcc5"} Oct 29 22:55:52.962896 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:52.962875 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-kk86b" Oct 29 22:55:53.012374 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:53.000482 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-bclh8" podStartSLOduration=2.000455078 podStartE2EDuration="2.000455078s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:55:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:55:52.9978943 +0000 UTC m=+1333.781107732" watchObservedRunningTime="2026-10-29 22:55:53.000455078 +0000 UTC m=+1333.783668519" Oct 29 22:55:53.061464 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:53.060740 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-hfrrx"] Oct 29 22:55:53.065378 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:53.065307 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-kk86b" podStartSLOduration=2.065288182 podStartE2EDuration="2.065288182s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:55:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:55:53.052390969 +0000 UTC m=+1333.835604420" watchObservedRunningTime="2026-10-29 22:55:53.065288182 +0000 UTC m=+1333.848501612" Oct 29 22:55:53.070041 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:53.070019 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-hfrrx" Oct 29 22:55:53.162823 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:53.162791 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-hfrrx"] Oct 29 22:55:53.261202 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:53.261163 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43a9077f-2bb9-4801-abba-687de8a45076-istio-podinfo\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-hfrrx\" (UID: \"43a9077f-2bb9-4801-abba-687de8a45076\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-hfrrx" Oct 29 22:55:53.261396 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:53.261270 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xx56t\" (UniqueName: \"kubernetes.io/projected/43a9077f-2bb9-4801-abba-687de8a45076-kube-api-access-xx56t\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-hfrrx\" (UID: \"43a9077f-2bb9-4801-abba-687de8a45076\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-hfrrx" Oct 29 22:55:53.261531 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:53.261514 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43a9077f-2bb9-4801-abba-687de8a45076-istiod-ca-cert\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-hfrrx\" (UID: \"43a9077f-2bb9-4801-abba-687de8a45076\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-hfrrx" Oct 29 22:55:53.261804 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:53.261705 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43a9077f-2bb9-4801-abba-687de8a45076-workload-socket\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-hfrrx\" (UID: \"43a9077f-2bb9-4801-abba-687de8a45076\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-hfrrx" Oct 29 22:55:53.261929 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:53.261908 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43a9077f-2bb9-4801-abba-687de8a45076-istio-envoy\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-hfrrx\" (UID: \"43a9077f-2bb9-4801-abba-687de8a45076\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-hfrrx" Oct 29 22:55:53.262003 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:53.261955 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43a9077f-2bb9-4801-abba-687de8a45076-istio-token\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-hfrrx\" (UID: \"43a9077f-2bb9-4801-abba-687de8a45076\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-hfrrx" Oct 29 22:55:53.262043 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:53.262029 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43a9077f-2bb9-4801-abba-687de8a45076-workload-certs\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-hfrrx\" (UID: \"43a9077f-2bb9-4801-abba-687de8a45076\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-hfrrx" Oct 29 22:55:53.262077 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:53.262065 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43a9077f-2bb9-4801-abba-687de8a45076-istio-data\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-hfrrx\" (UID: \"43a9077f-2bb9-4801-abba-687de8a45076\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-hfrrx" Oct 29 22:55:53.262202 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:53.262173 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43a9077f-2bb9-4801-abba-687de8a45076-credential-socket\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-hfrrx\" (UID: \"43a9077f-2bb9-4801-abba-687de8a45076\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-hfrrx" Oct 29 22:55:53.363578 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:53.363540 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43a9077f-2bb9-4801-abba-687de8a45076-istiod-ca-cert\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-hfrrx\" (UID: \"43a9077f-2bb9-4801-abba-687de8a45076\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-hfrrx" Oct 29 22:55:53.363861 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:53.363840 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43a9077f-2bb9-4801-abba-687de8a45076-workload-socket\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-hfrrx\" (UID: \"43a9077f-2bb9-4801-abba-687de8a45076\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-hfrrx" Oct 29 22:55:53.364010 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:53.363998 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43a9077f-2bb9-4801-abba-687de8a45076-istio-envoy\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-hfrrx\" (UID: \"43a9077f-2bb9-4801-abba-687de8a45076\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-hfrrx" Oct 29 22:55:53.364108 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:53.364095 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43a9077f-2bb9-4801-abba-687de8a45076-istio-token\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-hfrrx\" (UID: \"43a9077f-2bb9-4801-abba-687de8a45076\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-hfrrx" Oct 29 22:55:53.364202 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:53.364189 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43a9077f-2bb9-4801-abba-687de8a45076-workload-certs\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-hfrrx\" (UID: \"43a9077f-2bb9-4801-abba-687de8a45076\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-hfrrx" Oct 29 22:55:53.366385 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:53.366360 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43a9077f-2bb9-4801-abba-687de8a45076-istio-data\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-hfrrx\" (UID: \"43a9077f-2bb9-4801-abba-687de8a45076\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-hfrrx" Oct 29 22:55:53.366625 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:53.366611 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43a9077f-2bb9-4801-abba-687de8a45076-credential-socket\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-hfrrx\" (UID: \"43a9077f-2bb9-4801-abba-687de8a45076\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-hfrrx" Oct 29 22:55:53.366807 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:53.366793 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43a9077f-2bb9-4801-abba-687de8a45076-istio-podinfo\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-hfrrx\" (UID: \"43a9077f-2bb9-4801-abba-687de8a45076\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-hfrrx" Oct 29 22:55:53.366963 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:53.366949 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xx56t\" (UniqueName: \"kubernetes.io/projected/43a9077f-2bb9-4801-abba-687de8a45076-kube-api-access-xx56t\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-hfrrx\" (UID: \"43a9077f-2bb9-4801-abba-687de8a45076\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-hfrrx" Oct 29 22:55:53.368034 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:53.368011 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43a9077f-2bb9-4801-abba-687de8a45076-workload-certs\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-hfrrx\" (UID: \"43a9077f-2bb9-4801-abba-687de8a45076\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-hfrrx" Oct 29 22:55:53.368171 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:53.364671 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43a9077f-2bb9-4801-abba-687de8a45076-workload-socket\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-hfrrx\" (UID: \"43a9077f-2bb9-4801-abba-687de8a45076\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-hfrrx" Oct 29 22:55:53.368507 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:53.368489 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43a9077f-2bb9-4801-abba-687de8a45076-istio-data\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-hfrrx\" (UID: \"43a9077f-2bb9-4801-abba-687de8a45076\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-hfrrx" Oct 29 22:55:53.368619 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:53.365287 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43a9077f-2bb9-4801-abba-687de8a45076-istiod-ca-cert\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-hfrrx\" (UID: \"43a9077f-2bb9-4801-abba-687de8a45076\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-hfrrx" Oct 29 22:55:53.368954 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:53.368935 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43a9077f-2bb9-4801-abba-687de8a45076-credential-socket\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-hfrrx\" (UID: \"43a9077f-2bb9-4801-abba-687de8a45076\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-hfrrx" Oct 29 22:55:53.394773 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:53.394727 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43a9077f-2bb9-4801-abba-687de8a45076-istio-envoy\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-hfrrx\" (UID: \"43a9077f-2bb9-4801-abba-687de8a45076\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-hfrrx" Oct 29 22:55:53.432418 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:53.408726 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43a9077f-2bb9-4801-abba-687de8a45076-istio-podinfo\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-hfrrx\" (UID: \"43a9077f-2bb9-4801-abba-687de8a45076\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-hfrrx" Oct 29 22:55:53.432418 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:53.416895 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43a9077f-2bb9-4801-abba-687de8a45076-istio-token\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-hfrrx\" (UID: \"43a9077f-2bb9-4801-abba-687de8a45076\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-hfrrx" Oct 29 22:55:53.460388 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:53.451956 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xx56t\" (UniqueName: \"kubernetes.io/projected/43a9077f-2bb9-4801-abba-687de8a45076-kube-api-access-xx56t\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-hfrrx\" (UID: \"43a9077f-2bb9-4801-abba-687de8a45076\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-hfrrx" Oct 29 22:55:53.497355 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:53.495940 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-hfrrx" Oct 29 22:55:53.521262 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:53.521226 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-bclh8" Oct 29 22:55:53.521494 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:53.521470 5057 patch_prober.go:29] interesting pod/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-bclh8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Oct 29 22:55:53.521697 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:53.521670 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-bclh8" podUID="b510ea66-642c-49ae-a600-fab375005cb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Oct 29 22:55:54.019423 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:54.003015 5057 patch_prober.go:29] interesting pod/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-k8ws5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Oct 29 22:55:54.045464 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:54.045411 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-k8ws5" podUID="f53c1835-d1af-41c6-b8b9-e749ed7ef210" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Oct 29 22:55:54.153279 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:54.153242 5057 patch_prober.go:29] interesting pod/e2e-worker-w4-063b46-openshift-default-799d8d64cf-kk86b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Oct 29 22:55:54.153821 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:54.153540 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-kk86b" podUID="79d262bd-3d59-44ec-b375-28e6c7a57c1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Oct 29 22:55:54.164364 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:54.162513 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:55:54.533434 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:54.533401 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-4mwxl"] Oct 29 22:55:54.595363 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:54.594754 5057 patch_prober.go:29] interesting pod/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-bclh8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Oct 29 22:55:54.595363 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:54.594816 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-bclh8" podUID="b510ea66-642c-49ae-a600-fab375005cb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Oct 29 22:55:54.626556 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:54.619364 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:55:54.626556 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:54.619446 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:55:54.626556 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:54.619488 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:55:54.700356 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:54.686683 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:55:54.863022 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:54.862933 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:55:54.874095 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:54.874065 5057 patch_prober.go:29] interesting pod/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-k8ws5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Oct 29 22:55:54.874319 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:54.874294 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-k8ws5" podUID="f53c1835-d1af-41c6-b8b9-e749ed7ef210" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Oct 29 22:55:54.975388 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:54.970479 5057 patch_prober.go:29] interesting pod/e2e-worker-w4-063b46-openshift-default-799d8d64cf-kk86b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Oct 29 22:55:54.975388 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:54.970536 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-kk86b" podUID="79d262bd-3d59-44ec-b375-28e6c7a57c1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Oct 29 22:55:54.975388 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:54.973115 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-hfrrx"] Oct 29 22:55:54.975388 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:54.973162 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-4mwxl"] Oct 29 22:55:54.975388 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:54.973258 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-4mwxl" Oct 29 22:55:55.100996 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:55.100963 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd32fc20-abda-413c-88c1-6b3b6156e777-istio-envoy\") pod \"e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-4mwxl\" (UID: \"bd32fc20-abda-413c-88c1-6b3b6156e777\") " pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-4mwxl" Oct 29 22:55:55.109250 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:55.109206 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd32fc20-abda-413c-88c1-6b3b6156e777-istiod-ca-cert\") pod \"e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-4mwxl\" (UID: \"bd32fc20-abda-413c-88c1-6b3b6156e777\") " pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-4mwxl" Oct 29 22:55:55.128921 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:55.109540 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd32fc20-abda-413c-88c1-6b3b6156e777-istio-data\") pod \"e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-4mwxl\" (UID: \"bd32fc20-abda-413c-88c1-6b3b6156e777\") " pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-4mwxl" Oct 29 22:55:55.129424 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:55.129384 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd32fc20-abda-413c-88c1-6b3b6156e777-workload-socket\") pod \"e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-4mwxl\" (UID: \"bd32fc20-abda-413c-88c1-6b3b6156e777\") " pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-4mwxl" Oct 29 22:55:55.129942 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:55.129922 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd32fc20-abda-413c-88c1-6b3b6156e777-workload-certs\") pod \"e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-4mwxl\" (UID: \"bd32fc20-abda-413c-88c1-6b3b6156e777\") " pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-4mwxl" Oct 29 22:55:55.130140 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:55.130112 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd32fc20-abda-413c-88c1-6b3b6156e777-istio-token\") pod \"e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-4mwxl\" (UID: \"bd32fc20-abda-413c-88c1-6b3b6156e777\") " pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-4mwxl" Oct 29 22:55:55.130425 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:55.130401 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd32fc20-abda-413c-88c1-6b3b6156e777-istio-podinfo\") pod \"e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-4mwxl\" (UID: \"bd32fc20-abda-413c-88c1-6b3b6156e777\") " pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-4mwxl" Oct 29 22:55:55.130629 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:55.130592 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd32fc20-abda-413c-88c1-6b3b6156e777-credential-socket\") pod \"e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-4mwxl\" (UID: \"bd32fc20-abda-413c-88c1-6b3b6156e777\") " pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-4mwxl" Oct 29 22:55:55.130845 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:55.130825 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bpfq\" (UniqueName: \"kubernetes.io/projected/bd32fc20-abda-413c-88c1-6b3b6156e777-kube-api-access-5bpfq\") pod \"e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-4mwxl\" (UID: \"bd32fc20-abda-413c-88c1-6b3b6156e777\") " pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-4mwxl" Oct 29 22:55:55.248554 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:55.239684 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd32fc20-abda-413c-88c1-6b3b6156e777-istio-envoy\") pod \"e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-4mwxl\" (UID: \"bd32fc20-abda-413c-88c1-6b3b6156e777\") " pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-4mwxl" Oct 29 22:55:55.248554 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:55.239918 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd32fc20-abda-413c-88c1-6b3b6156e777-istiod-ca-cert\") pod \"e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-4mwxl\" (UID: \"bd32fc20-abda-413c-88c1-6b3b6156e777\") " pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-4mwxl" Oct 29 22:55:55.248554 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:55.239961 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd32fc20-abda-413c-88c1-6b3b6156e777-istio-data\") pod \"e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-4mwxl\" (UID: \"bd32fc20-abda-413c-88c1-6b3b6156e777\") " pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-4mwxl" Oct 29 22:55:55.248554 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:55.240070 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd32fc20-abda-413c-88c1-6b3b6156e777-workload-socket\") pod \"e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-4mwxl\" (UID: \"bd32fc20-abda-413c-88c1-6b3b6156e777\") " pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-4mwxl" Oct 29 22:55:55.248554 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:55.240111 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd32fc20-abda-413c-88c1-6b3b6156e777-workload-certs\") pod \"e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-4mwxl\" (UID: \"bd32fc20-abda-413c-88c1-6b3b6156e777\") " pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-4mwxl" Oct 29 22:55:55.248554 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:55.240145 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd32fc20-abda-413c-88c1-6b3b6156e777-istio-token\") pod \"e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-4mwxl\" (UID: \"bd32fc20-abda-413c-88c1-6b3b6156e777\") " pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-4mwxl" Oct 29 22:55:55.248554 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:55.240209 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd32fc20-abda-413c-88c1-6b3b6156e777-istio-podinfo\") pod \"e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-4mwxl\" (UID: \"bd32fc20-abda-413c-88c1-6b3b6156e777\") " pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-4mwxl" Oct 29 22:55:55.248554 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:55.240251 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd32fc20-abda-413c-88c1-6b3b6156e777-credential-socket\") pod \"e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-4mwxl\" (UID: \"bd32fc20-abda-413c-88c1-6b3b6156e777\") " pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-4mwxl" Oct 29 22:55:55.248554 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:55.240285 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bpfq\" (UniqueName: \"kubernetes.io/projected/bd32fc20-abda-413c-88c1-6b3b6156e777-kube-api-access-5bpfq\") pod \"e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-4mwxl\" (UID: \"bd32fc20-abda-413c-88c1-6b3b6156e777\") " pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-4mwxl" Oct 29 22:55:55.271460 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:55.249636 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd32fc20-abda-413c-88c1-6b3b6156e777-credential-socket\") pod \"e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-4mwxl\" (UID: \"bd32fc20-abda-413c-88c1-6b3b6156e777\") " pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-4mwxl" Oct 29 22:55:55.271460 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:55.250207 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd32fc20-abda-413c-88c1-6b3b6156e777-istio-data\") pod \"e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-4mwxl\" (UID: \"bd32fc20-abda-413c-88c1-6b3b6156e777\") " pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-4mwxl" Oct 29 22:55:55.271460 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:55.250508 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd32fc20-abda-413c-88c1-6b3b6156e777-workload-socket\") pod \"e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-4mwxl\" (UID: \"bd32fc20-abda-413c-88c1-6b3b6156e777\") " pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-4mwxl" Oct 29 22:55:55.274843 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:55.274808 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd32fc20-abda-413c-88c1-6b3b6156e777-workload-certs\") pod \"e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-4mwxl\" (UID: \"bd32fc20-abda-413c-88c1-6b3b6156e777\") " pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-4mwxl" Oct 29 22:55:55.496355 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:55.483050 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bpfq\" (UniqueName: \"kubernetes.io/projected/bd32fc20-abda-413c-88c1-6b3b6156e777-kube-api-access-5bpfq\") pod \"e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-4mwxl\" (UID: \"bd32fc20-abda-413c-88c1-6b3b6156e777\") " pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-4mwxl" Oct 29 22:55:55.542432 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:55.500836 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd32fc20-abda-413c-88c1-6b3b6156e777-istio-podinfo\") pod \"e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-4mwxl\" (UID: \"bd32fc20-abda-413c-88c1-6b3b6156e777\") " pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-4mwxl" Oct 29 22:55:55.542653 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:55.535177 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd32fc20-abda-413c-88c1-6b3b6156e777-istio-token\") pod \"e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-4mwxl\" (UID: \"bd32fc20-abda-413c-88c1-6b3b6156e777\") " pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-4mwxl" Oct 29 22:55:55.542782 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:55.535411 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd32fc20-abda-413c-88c1-6b3b6156e777-istio-envoy\") pod \"e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-4mwxl\" (UID: \"bd32fc20-abda-413c-88c1-6b3b6156e777\") " pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-4mwxl" Oct 29 22:55:55.542886 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:55.537582 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd32fc20-abda-413c-88c1-6b3b6156e777-istiod-ca-cert\") pod \"e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-4mwxl\" (UID: \"bd32fc20-abda-413c-88c1-6b3b6156e777\") " pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-4mwxl" Oct 29 22:55:55.628770 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:55.628584 5057 patch_prober.go:29] interesting pod/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-bclh8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Oct 29 22:55:55.628770 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:55.628643 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-bclh8" podUID="b510ea66-642c-49ae-a600-fab375005cb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Oct 29 22:55:55.716373 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:55.715957 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-4mwxl" Oct 29 22:55:55.815440 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:55.811929 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-08e15d/distinct-w0-08e15d-kserve-85695c78bb-p67gk" podUID="e664f012-3b80-4a8a-ac19-d767d92c0a33" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:55:55.867376 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:55.849267 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-31226c/distinct-w6-31226c-kserve-858d95fd76-bkf7c" podUID="f3fc11c6-03c9-4314-99c3-f857b149415c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded" Oct 29 22:55:55.870831 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:55.870792 5057 patch_prober.go:29] interesting pod/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-k8ws5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Oct 29 22:55:55.870977 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:55.870860 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-k8ws5" podUID="f53c1835-d1af-41c6-b8b9-e749ed7ef210" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Oct 29 22:55:56.164283 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:56.102585 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:55:56.370939 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:56.223182 5057 patch_prober.go:29] interesting pod/e2e-worker-w4-063b46-openshift-default-799d8d64cf-kk86b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Oct 29 22:55:56.370939 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:56.367733 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-kk86b" podUID="79d262bd-3d59-44ec-b375-28e6c7a57c1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Oct 29 22:55:56.370939 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:56.286422 5057 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-17394a/distinct-2-w3-17394a-kserve-6545c766c4-h4chf" podUID="f29e8b65-b7dd-45d4-86b0-f13a29ef8e66" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 29 22:55:56.384442 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:56.381757 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-hfrrx" event={"ID":"43a9077f-2bb9-4801-abba-687de8a45076","Type":"ContainerStarted","Data":"a855f564baf16681264119ad39dc6b27c474133d10eee8ba886f2dd148ad58a3"} Oct 29 22:55:56.533722 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:56.533682 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:55:56.534423 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:56.534395 5057 patch_prober.go:29] interesting pod/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-bclh8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Oct 29 22:55:56.534605 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:56.534580 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-bclh8" podUID="b510ea66-642c-49ae-a600-fab375005cb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Oct 29 22:55:56.568460 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:56.568414 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17394a/distinct-2-w3-17394a-kserve-6545c766c4-h4chf" podUID="f29e8b65-b7dd-45d4-86b0-f13a29ef8e66" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:55:56.849015 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:56.848968 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-08e15d/distinct-w0-08e15d-kserve-85695c78bb-p67gk" podUID="e664f012-3b80-4a8a-ac19-d767d92c0a33" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": context deadline exceeded" Oct 29 22:55:56.875393 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:56.865823 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-08e15d/distinct-w0-08e15d-kserve-85695c78bb-p67gk" Oct 29 22:55:56.897142 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:56.897098 5057 patch_prober.go:29] interesting pod/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-k8ws5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Oct 29 22:55:56.897432 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:56.897400 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-k8ws5" podUID="f53c1835-d1af-41c6-b8b9-e749ed7ef210" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Oct 29 22:55:57.001783 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:56.977411 5057 patch_prober.go:29] interesting pod/e2e-worker-w4-063b46-openshift-default-799d8d64cf-kk86b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Oct 29 22:55:57.001783 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:56.998473 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-kk86b" podUID="79d262bd-3d59-44ec-b375-28e6c7a57c1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Oct 29 22:55:57.172549 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:57.168474 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:55:57.341425 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:57.341307 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:55:57.552472 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:57.552424 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/distinct-2-w2-575d59-kserve-d4565b76c-hdgbf" podUID="06bb975e-d825-426f-b030-17bd1f99bb65" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:55:57.593407 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:55:57.586343 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd32fc20_abda_413c_88c1_6b3b6156e777.slice/crio-0f241a5d13dfc20c4a09a4752e0c9a27a55835bdc447a7615a5f4e60d3ad183e WatchSource:0}: Error finding container 0f241a5d13dfc20c4a09a4752e0c9a27a55835bdc447a7615a5f4e60d3ad183e: Status 404 returned error can't find the container with id 0f241a5d13dfc20c4a09a4752e0c9a27a55835bdc447a7615a5f4e60d3ad183e Oct 29 22:55:57.628426 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:57.604899 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17394a/distinct-2-w3-17394a-kserve-6545c766c4-h4chf" podUID="f29e8b65-b7dd-45d4-86b0-f13a29ef8e66" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:55:57.710403 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:57.710367 5057 patch_prober.go:29] interesting pod/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-bclh8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Oct 29 22:55:57.710655 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:57.710631 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-bclh8" podUID="b510ea66-642c-49ae-a600-fab375005cb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Oct 29 22:55:57.747546 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:57.746498 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:55:57.747546 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:57.746615 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:55:57.747546 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:57.746664 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:55:57.772585 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:57.758896 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-17394a/distinct-2-w3-17394a-kserve-6545c766c4-h4chf" Oct 29 22:55:57.772585 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:57.758943 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-4mwxl"] Oct 29 22:55:57.860407 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:57.860030 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-08e15d/distinct-w0-08e15d-kserve-85695c78bb-p67gk" podUID="e664f012-3b80-4a8a-ac19-d767d92c0a33" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": context deadline exceeded" Oct 29 22:55:57.877355 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:57.875440 5057 patch_prober.go:29] interesting pod/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-k8ws5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Oct 29 22:55:57.877355 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:57.875510 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-k8ws5" podUID="f53c1835-d1af-41c6-b8b9-e749ed7ef210" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Oct 29 22:55:58.046903 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:58.014092 5057 patch_prober.go:29] interesting pod/e2e-worker-w4-063b46-openshift-default-799d8d64cf-kk86b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Oct 29 22:55:58.046903 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:58.014154 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-kk86b" podUID="79d262bd-3d59-44ec-b375-28e6c7a57c1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Oct 29 22:55:58.147360 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:58.147261 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:55:58.148619 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:58.148588 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-hfrrx" event={"ID":"43a9077f-2bb9-4801-abba-687de8a45076","Type":"ContainerStarted","Data":"cb4d8eb96cd3773d35f2f2bb1db5dc07f45ab4cfdec4e543c53e1253025cff5a"} Oct 29 22:55:58.373375 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:58.365646 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-hfrrx" podStartSLOduration=5.365621417 podStartE2EDuration="5.365621417s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:55:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:55:58.201640442 +0000 UTC m=+1338.984853893" watchObservedRunningTime="2026-10-29 22:55:58.365621417 +0000 UTC m=+1339.148834845" Oct 29 22:55:58.552357 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:58.531169 5057 patch_prober.go:29] interesting pod/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-bclh8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Oct 29 22:55:58.552357 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:58.531233 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-bclh8" podUID="b510ea66-642c-49ae-a600-fab375005cb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Oct 29 22:55:58.552357 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:58.531628 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-hfrrx" Oct 29 22:55:58.552357 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:58.531978 5057 patch_prober.go:29] interesting pod/e2e-worker-w5-b2921b-openshift-default-5547c88fff-hfrrx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Oct 29 22:55:58.552357 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:58.532016 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-hfrrx" podUID="43a9077f-2bb9-4801-abba-687de8a45076" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Oct 29 22:55:58.578428 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:58.578386 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/distinct-2-w2-575d59-kserve-d4565b76c-hdgbf" podUID="06bb975e-d825-426f-b030-17bd1f99bb65" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": context deadline exceeded" Oct 29 22:55:58.578999 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:58.578976 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-575d59/distinct-2-w2-575d59-kserve-d4565b76c-hdgbf" Oct 29 22:55:58.609509 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:58.609467 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17394a/distinct-2-w3-17394a-kserve-6545c766c4-h4chf" podUID="f29e8b65-b7dd-45d4-86b0-f13a29ef8e66" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:55:58.848504 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:58.848462 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-31226c/distinct-w6-31226c-kserve-858d95fd76-bkf7c" podUID="f3fc11c6-03c9-4314-99c3-f857b149415c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:55:58.861164 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:58.861027 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-08e15d/distinct-w0-08e15d-kserve-85695c78bb-p67gk" podUID="e664f012-3b80-4a8a-ac19-d767d92c0a33" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": context deadline exceeded" Oct 29 22:55:58.924194 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:58.922351 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:55:58.970398 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:58.970324 5057 patch_prober.go:29] interesting pod/e2e-worker-w4-063b46-openshift-default-799d8d64cf-kk86b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Oct 29 22:55:58.970602 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:58.970405 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-kk86b" podUID="79d262bd-3d59-44ec-b375-28e6c7a57c1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Oct 29 22:55:59.001656 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:59.001621 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-08e15d/distinct-w0-08e15d-kserve-85695c78bb-p67gk" Oct 29 22:55:59.172409 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:59.168835 5057 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-31226c/distinct-w6-31226c-kserve-858d95fd76-bkf7c" podUID="f3fc11c6-03c9-4314-99c3-f857b149415c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:55:59.508355 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:59.508128 5057 patch_prober.go:29] interesting pod/e2e-worker-w5-b2921b-openshift-default-5547c88fff-hfrrx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Oct 29 22:55:59.508355 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:59.508191 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-hfrrx" podUID="43a9077f-2bb9-4801-abba-687de8a45076" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Oct 29 22:55:59.658577 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:59.652212 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17394a/distinct-2-w3-17394a-kserve-6545c766c4-h4chf" podUID="f29e8b65-b7dd-45d4-86b0-f13a29ef8e66" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": context deadline exceeded" Oct 29 22:55:59.674376 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:59.659177 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/distinct-2-w2-575d59-kserve-d4565b76c-hdgbf" podUID="06bb975e-d825-426f-b030-17bd1f99bb65" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": context deadline exceeded" Oct 29 22:55:59.839792 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:59.816953 5057 patch_prober.go:29] interesting pod/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-bclh8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Oct 29 22:55:59.839792 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:59.817011 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-bclh8" podUID="b510ea66-642c-49ae-a600-fab375005cb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Oct 29 22:55:59.856355 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:59.855863 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-31226c/distinct-w6-31226c-kserve-858d95fd76-bkf7c" podUID="f3fc11c6-03c9-4314-99c3-f857b149415c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded" Oct 29 22:55:59.931364 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:55:59.878881 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-31226c/distinct-w6-31226c-kserve-858d95fd76-bkf7c" Oct 29 22:56:00.186585 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:00.177169 5057 patch_prober.go:29] interesting pod/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-k8ws5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 22:56:00.273356 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:00.235530 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-k8ws5" podUID="f53c1835-d1af-41c6-b8b9-e749ed7ef210" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:56:00.273356 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:00.205801 5057 patch_prober.go:29] interesting pod/e2e-worker-w4-063b46-openshift-default-799d8d64cf-kk86b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Oct 29 22:56:00.273356 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:00.235958 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-kk86b" podUID="79d262bd-3d59-44ec-b375-28e6c7a57c1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Oct 29 22:56:00.488409 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:00.447893 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-4mwxl" event={"ID":"bd32fc20-abda-413c-88c1-6b3b6156e777","Type":"ContainerStarted","Data":"8a177953428b4ad55d62c0e777cd59815b6feb1585f41222bc1c5cc94e76d9af"} Oct 29 22:56:00.488409 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:00.448157 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-17394a/distinct-2-w3-17394a-kserve-6545c766c4-h4chf" Oct 29 22:56:00.488409 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:00.448179 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-4mwxl" event={"ID":"bd32fc20-abda-413c-88c1-6b3b6156e777","Type":"ContainerStarted","Data":"0f241a5d13dfc20c4a09a4752e0c9a27a55835bdc447a7615a5f4e60d3ad183e"} Oct 29 22:56:00.886533 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:00.886434 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-31226c/distinct-w6-31226c-kserve-858d95fd76-bkf7c" podUID="f3fc11c6-03c9-4314-99c3-f857b149415c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:56:01.118405 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:00.927805 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/distinct-2-w2-575d59-kserve-d4565b76c-hdgbf" podUID="06bb975e-d825-426f-b030-17bd1f99bb65" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:56:01.120169 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:00.928019 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-4mwxl" podStartSLOduration=6.927997488 podStartE2EDuration="6.927997488s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:55:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:56:00.63598504 +0000 UTC m=+1341.419198465" watchObservedRunningTime="2026-10-29 22:56:00.927997488 +0000 UTC m=+1341.711210935" Oct 29 22:56:01.252827 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:01.031918 5057 patch_prober.go:29] interesting pod/e2e-worker-w4-063b46-openshift-default-799d8d64cf-kk86b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Oct 29 22:56:01.253190 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:01.253151 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-kk86b" podUID="79d262bd-3d59-44ec-b375-28e6c7a57c1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Oct 29 22:56:01.262088 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:01.180709 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-4mwxl" Oct 29 22:56:01.262396 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:01.159360 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-08e15d/distinct-w0-08e15d-kserve-85695c78bb-p67gk" podUID="e664f012-3b80-4a8a-ac19-d767d92c0a33" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:56:01.270697 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:01.120649 5057 patch_prober.go:29] interesting pod/e2e-worker-w5-b2921b-openshift-default-5547c88fff-hfrrx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Oct 29 22:56:01.273403 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:01.270954 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-hfrrx" podUID="43a9077f-2bb9-4801-abba-687de8a45076" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Oct 29 22:56:01.273626 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:01.131442 5057 patch_prober.go:29] interesting pod/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-bclh8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Oct 29 22:56:01.274393 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:01.273778 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-bclh8" podUID="b510ea66-642c-49ae-a600-fab375005cb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Oct 29 22:56:01.277439 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:01.188462 5057 patch_prober.go:29] interesting pod/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-4mwxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Oct 29 22:56:01.277657 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:01.277615 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-4mwxl" podUID="bd32fc20-abda-413c-88c1-6b3b6156e777" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Oct 29 22:56:01.605809 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:01.605713 5057 patch_prober.go:29] interesting pod/e2e-worker-w5-b2921b-openshift-default-5547c88fff-hfrrx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Oct 29 22:56:01.753417 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:01.753374 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-hfrrx" podUID="43a9077f-2bb9-4801-abba-687de8a45076" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Oct 29 22:56:01.753871 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:01.680524 5057 patch_prober.go:29] interesting pod/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-bclh8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Oct 29 22:56:01.754067 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:01.754043 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-bclh8" podUID="b510ea66-642c-49ae-a600-fab375005cb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Oct 29 22:56:01.754205 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:01.698449 5057 patch_prober.go:29] interesting pod/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-k8ws5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 22:56:01.754347 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:01.754312 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-k8ws5" podUID="f53c1835-d1af-41c6-b8b9-e749ed7ef210" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:56:01.797358 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:01.773070 5057 patch_prober.go:29] interesting pod/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-4mwxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Oct 29 22:56:01.920663 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:01.920574 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-4mwxl" podUID="bd32fc20-abda-413c-88c1-6b3b6156e777" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Oct 29 22:56:01.921196 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:01.901926 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17394a/distinct-2-w3-17394a-kserve-6545c766c4-h4chf" podUID="f29e8b65-b7dd-45d4-86b0-f13a29ef8e66" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:56:02.022800 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:02.022761 5057 patch_prober.go:29] interesting pod/e2e-worker-w4-063b46-openshift-default-799d8d64cf-kk86b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Oct 29 22:56:02.080204 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:02.023141 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-kk86b" podUID="79d262bd-3d59-44ec-b375-28e6c7a57c1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Oct 29 22:56:02.119011 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:02.118968 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/distinct-2-w2-575d59-kserve-d4565b76c-hdgbf" podUID="06bb975e-d825-426f-b030-17bd1f99bb65" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:56:02.119594 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:02.119564 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-31226c/distinct-w6-31226c-kserve-858d95fd76-bkf7c" podUID="f3fc11c6-03c9-4314-99c3-f857b149415c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded" Oct 29 22:56:02.199415 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:02.178530 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-f3092a4d-7f9bb6f4fd-2kck8" Oct 29 22:56:02.200177 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:02.200149 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-f3092a4d-7745bbd6ff-vsstw" Oct 29 22:56:02.306709 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:02.284695 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-08e15d/distinct-w0-08e15d-kserve-85695c78bb-p67gk" podUID="e664f012-3b80-4a8a-ac19-d767d92c0a33" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": context deadline exceeded" Oct 29 22:56:02.306709 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:02.285078 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-08e15d/distinct-w0-08e15d-kserve-85695c78bb-p67gk" Oct 29 22:56:02.528416 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:02.518558 5057 patch_prober.go:29] interesting pod/e2e-worker-w5-b2921b-openshift-default-5547c88fff-hfrrx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Oct 29 22:56:02.528716 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:02.528664 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-hfrrx" podUID="43a9077f-2bb9-4801-abba-687de8a45076" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Oct 29 22:56:02.550416 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:02.550377 5057 patch_prober.go:29] interesting pod/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-bclh8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Oct 29 22:56:02.554380 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:02.550667 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-bclh8" podUID="b510ea66-642c-49ae-a600-fab375005cb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Oct 29 22:56:02.705562 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:02.705511 5057 patch_prober.go:29] interesting pod/e2e-worker-w0-08e15d-openshift-default-d579fdc75-vmj4r container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 22:56:02.705847 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:02.705801 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-vmj4r" podUID="5954f1f5-c478-4f49-877e-9d3300568c2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:56:02.822713 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:02.822611 5057 patch_prober.go:29] interesting pod/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-k8ws5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 22:56:02.855357 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:02.822935 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-k8ws5" podUID="f53c1835-d1af-41c6-b8b9-e749ed7ef210" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:56:02.858824 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:02.858703 5057 patch_prober.go:29] interesting pod/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-4mwxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Oct 29 22:56:02.859249 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:02.859047 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-4mwxl" podUID="bd32fc20-abda-413c-88c1-6b3b6156e777" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Oct 29 22:56:02.925506 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:02.925462 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17394a/distinct-2-w3-17394a-kserve-6545c766c4-h4chf" podUID="f29e8b65-b7dd-45d4-86b0-f13a29ef8e66" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:56:02.944361 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:02.939906 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-17394a/distinct-2-w3-17394a-kserve-6545c766c4-h4chf" Oct 29 22:56:03.029648 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:03.029614 5057 patch_prober.go:29] interesting pod/e2e-worker-w4-063b46-openshift-default-799d8d64cf-kk86b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Oct 29 22:56:03.029899 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:03.029876 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-kk86b" podUID="79d262bd-3d59-44ec-b375-28e6c7a57c1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Oct 29 22:56:03.121935 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:03.121832 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-31226c/distinct-w6-31226c-kserve-858d95fd76-bkf7c" podUID="f3fc11c6-03c9-4314-99c3-f857b149415c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:56:03.170208 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:03.170166 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/distinct-2-w2-575d59-kserve-d4565b76c-hdgbf" podUID="06bb975e-d825-426f-b030-17bd1f99bb65" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 29 22:56:03.301596 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:03.301540 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-08e15d/distinct-w0-08e15d-kserve-85695c78bb-p67gk" podUID="e664f012-3b80-4a8a-ac19-d767d92c0a33" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:56:03.527475 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:03.515548 5057 patch_prober.go:29] interesting pod/e2e-worker-w5-b2921b-openshift-default-5547c88fff-hfrrx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Oct 29 22:56:03.527475 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:03.515608 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-hfrrx" podUID="43a9077f-2bb9-4801-abba-687de8a45076" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Oct 29 22:56:03.527475 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:03.518747 5057 patch_prober.go:29] interesting pod/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-bclh8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Oct 29 22:56:03.527475 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:03.518793 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-bclh8" podUID="b510ea66-642c-49ae-a600-fab375005cb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Oct 29 22:56:03.601496 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:03.561831 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:56:03.792413 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:03.790421 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:56:03.792413 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:03.790801 5057 patch_prober.go:29] interesting pod/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-4mwxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Oct 29 22:56:03.792413 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:03.790850 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-4mwxl" podUID="bd32fc20-abda-413c-88c1-6b3b6156e777" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Oct 29 22:56:03.934866 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:03.934807 5057 patch_prober.go:29] interesting pod/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-k8ws5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 22:56:03.935489 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:03.935439 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-k8ws5" podUID="f53c1835-d1af-41c6-b8b9-e749ed7ef210" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:56:04.010358 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:03.992109 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17394a/distinct-2-w3-17394a-kserve-6545c766c4-h4chf" podUID="f29e8b65-b7dd-45d4-86b0-f13a29ef8e66" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": context deadline exceeded" Oct 29 22:56:04.010358 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:04.005081 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:56:04.132517 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:04.132414 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-31226c/distinct-w6-31226c-kserve-858d95fd76-bkf7c" podUID="f3fc11c6-03c9-4314-99c3-f857b149415c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:56:04.177373 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:04.172404 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/distinct-2-w2-575d59-kserve-d4565b76c-hdgbf" podUID="06bb975e-d825-426f-b030-17bd1f99bb65" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:56:04.310504 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:04.310448 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-08e15d/distinct-w0-08e15d-kserve-85695c78bb-p67gk" podUID="e664f012-3b80-4a8a-ac19-d767d92c0a33" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:56:04.527296 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:04.526543 5057 patch_prober.go:29] interesting pod/e2e-worker-w5-b2921b-openshift-default-5547c88fff-hfrrx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Oct 29 22:56:04.527296 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:04.526601 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-hfrrx" podUID="43a9077f-2bb9-4801-abba-687de8a45076" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Oct 29 22:56:04.532717 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:04.532671 5057 patch_prober.go:29] interesting pod/e2e-worker-w0-08e15d-openshift-default-d579fdc75-7smjh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 22:56:04.532910 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:04.532869 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-7smjh" podUID="7bc2103e-768a-4393-9522-68b0fc370219" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:56:04.779519 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:04.779422 5057 patch_prober.go:29] interesting pod/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-4mwxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Oct 29 22:56:04.779519 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:04.779481 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-4mwxl" podUID="bd32fc20-abda-413c-88c1-6b3b6156e777" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Oct 29 22:56:04.882398 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:04.857718 5057 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-08e15d/distinct-w0-08e15d-kserve-85695c78bb-p67gk" podUID="e664f012-3b80-4a8a-ac19-d767d92c0a33" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:56:05.012655 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:04.906061 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-k8ws5" Oct 29 22:56:05.055855 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:05.023639 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17394a/distinct-2-w3-17394a-kserve-6545c766c4-h4chf" podUID="f29e8b65-b7dd-45d4-86b0-f13a29ef8e66" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": context deadline exceeded" Oct 29 22:56:05.066383 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:05.040471 5057 patch_prober.go:29] interesting pod/e2e-worker-w4-063b46-openshift-default-799d8d64cf-kk86b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 22:56:05.104677 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:05.104632 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-kk86b" podUID="79d262bd-3d59-44ec-b375-28e6c7a57c1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:56:05.105243 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:05.059438 5057 patch_prober.go:29] interesting pod/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-k8ws5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 22:56:05.105444 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:05.105418 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-k8ws5" podUID="f53c1835-d1af-41c6-b8b9-e749ed7ef210" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:56:05.168438 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:05.168387 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-31226c/distinct-w6-31226c-kserve-858d95fd76-bkf7c" podUID="f3fc11c6-03c9-4314-99c3-f857b149415c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:56:05.215356 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:05.185175 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/distinct-2-w2-575d59-kserve-d4565b76c-hdgbf" podUID="06bb975e-d825-426f-b030-17bd1f99bb65" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:56:05.380409 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:05.378566 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-08e15d/distinct-w0-08e15d-kserve-85695c78bb-p67gk" podUID="e664f012-3b80-4a8a-ac19-d767d92c0a33" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:56:05.589873 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:05.589831 5057 patch_prober.go:29] interesting pod/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-bclh8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 22:56:05.590178 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:05.590148 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-bclh8" podUID="b510ea66-642c-49ae-a600-fab375005cb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:56:05.594485 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:05.594459 5057 patch_prober.go:29] interesting pod/e2e-worker-w5-b2921b-openshift-default-5547c88fff-hfrrx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Oct 29 22:56:05.740733 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:05.740691 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-hfrrx" podUID="43a9077f-2bb9-4801-abba-687de8a45076" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Oct 29 22:56:05.891519 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:05.891482 5057 patch_prober.go:29] interesting pod/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-4mwxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Oct 29 22:56:05.891789 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:05.891756 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-4mwxl" podUID="bd32fc20-abda-413c-88c1-6b3b6156e777" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Oct 29 22:56:05.937432 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:05.932615 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-k8ws5" Oct 29 22:56:06.082408 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:06.060903 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17394a/distinct-2-w3-17394a-kserve-6545c766c4-h4chf" podUID="f29e8b65-b7dd-45d4-86b0-f13a29ef8e66" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": context deadline exceeded" Oct 29 22:56:06.082408 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:06.067871 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-k8ws5" Oct 29 22:56:06.110455 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:06.110416 5057 patch_prober.go:29] interesting pod/e2e-worker-w4-063b46-openshift-default-799d8d64cf-kk86b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 22:56:06.110720 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:06.110689 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-kk86b" podUID="79d262bd-3d59-44ec-b375-28e6c7a57c1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:56:06.196163 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:06.196113 5057 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-17394a/distinct-2-w3-17394a-kserve-6545c766c4-h4chf" podUID="f29e8b65-b7dd-45d4-86b0-f13a29ef8e66" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:56:06.221371 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:06.221308 5057 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-cm4wj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 22:56:06.221669 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:06.221628 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-cm4wj" podUID="4e861691-eac4-4f06-a6aa-b6553efdd358" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:56:06.223586 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:06.223541 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/distinct-2-w2-575d59-kserve-d4565b76c-hdgbf" podUID="06bb975e-d825-426f-b030-17bd1f99bb65" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:56:06.224435 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:06.224406 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-31226c/distinct-w6-31226c-kserve-858d95fd76-bkf7c" podUID="f3fc11c6-03c9-4314-99c3-f857b149415c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:56:06.431435 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:06.413737 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-kk86b" Oct 29 22:56:06.431435 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:06.415434 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-kk86b" Oct 29 22:56:06.448284 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:06.446527 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-08e15d/distinct-w0-08e15d-kserve-85695c78bb-p67gk" podUID="e664f012-3b80-4a8a-ac19-d767d92c0a33" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:56:06.484801 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:06.484767 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-kk86b" Oct 29 22:56:06.521701 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:06.519223 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-bclh8" Oct 29 22:56:06.548353 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:06.543556 5057 patch_prober.go:29] interesting pod/e2e-worker-w5-b2921b-openshift-default-5547c88fff-hfrrx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Oct 29 22:56:06.548353 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:06.543610 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-hfrrx" podUID="43a9077f-2bb9-4801-abba-687de8a45076" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Oct 29 22:56:06.596783 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:06.596748 5057 patch_prober.go:29] interesting pod/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-bclh8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 22:56:06.597047 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:06.597018 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-bclh8" podUID="b510ea66-642c-49ae-a600-fab375005cb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:56:06.809363 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:06.806766 5057 patch_prober.go:29] interesting pod/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-4mwxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Oct 29 22:56:06.809363 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:06.806826 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-4mwxl" podUID="bd32fc20-abda-413c-88c1-6b3b6156e777" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Oct 29 22:56:07.037357 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:07.011895 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:56:07.141815 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:07.069381 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17394a/distinct-2-w3-17394a-kserve-6545c766c4-h4chf" podUID="f29e8b65-b7dd-45d4-86b0-f13a29ef8e66" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": context deadline exceeded" Oct 29 22:56:07.227366 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:07.224487 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/distinct-2-w2-575d59-kserve-d4565b76c-hdgbf" podUID="06bb975e-d825-426f-b030-17bd1f99bb65" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:56:07.278677 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:07.278624 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-31226c/distinct-w6-31226c-kserve-858d95fd76-bkf7c" podUID="f3fc11c6-03c9-4314-99c3-f857b149415c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:56:07.578539 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:07.567245 5057 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-575d59/distinct-2-w2-575d59-kserve-d4565b76c-hdgbf" podUID="06bb975e-d825-426f-b030-17bd1f99bb65" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:56:07.578539 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:07.568360 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-08e15d/distinct-w0-08e15d-kserve-85695c78bb-p67gk" podUID="e664f012-3b80-4a8a-ac19-d767d92c0a33" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": context deadline exceeded" Oct 29 22:56:07.639360 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:07.634855 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:56:07.733424 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:07.717774 5057 patch_prober.go:29] interesting pod/e2e-worker-w5-b2921b-openshift-default-5547c88fff-hfrrx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Oct 29 22:56:07.733424 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:07.717834 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-hfrrx" podUID="43a9077f-2bb9-4801-abba-687de8a45076" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Oct 29 22:56:07.817269 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:07.794514 5057 patch_prober.go:29] interesting pod/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-4mwxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Oct 29 22:56:07.817269 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:07.794594 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-4mwxl" podUID="bd32fc20-abda-413c-88c1-6b3b6156e777" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Oct 29 22:56:07.817269 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:07.816809 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:56:08.001356 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:07.957019 5057 patch_prober.go:29] interesting pod/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-bclh8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 22:56:08.288469 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:08.284488 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-bclh8" podUID="b510ea66-642c-49ae-a600-fab375005cb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:56:08.431354 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:08.265807 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17394a/distinct-2-w3-17394a-kserve-6545c766c4-h4chf" podUID="f29e8b65-b7dd-45d4-86b0-f13a29ef8e66" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": context deadline exceeded" Oct 29 22:56:08.431354 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:08.294431 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-31226c/distinct-w6-31226c-kserve-858d95fd76-bkf7c" podUID="f3fc11c6-03c9-4314-99c3-f857b149415c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:56:08.458688 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:08.227013 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/distinct-2-w2-575d59-kserve-d4565b76c-hdgbf" podUID="06bb975e-d825-426f-b030-17bd1f99bb65" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": context deadline exceeded" Oct 29 22:56:08.615684 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:08.615583 5057 patch_prober.go:29] interesting pod/e2e-worker-w5-b2921b-openshift-default-5547c88fff-hfrrx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Oct 29 22:56:08.615970 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:08.615918 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-hfrrx" podUID="43a9077f-2bb9-4801-abba-687de8a45076" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Oct 29 22:56:08.884555 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:08.884425 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-08e15d/distinct-w0-08e15d-kserve-85695c78bb-p67gk" podUID="e664f012-3b80-4a8a-ac19-d767d92c0a33" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": context deadline exceeded" Oct 29 22:56:08.896846 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:08.896802 5057 patch_prober.go:29] interesting pod/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-4mwxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Oct 29 22:56:08.897166 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:08.897119 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-4mwxl" podUID="bd32fc20-abda-413c-88c1-6b3b6156e777" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Oct 29 22:56:08.975371 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:08.971713 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:56:08.988815 ip-10-0-140-74 kubenswrapper[5057]: E1029 22:56:08.988783 5057 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.664s" Oct 29 22:56:08.989060 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:08.989046 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-hfrrx" Oct 29 22:56:08.989295 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:08.989283 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-575d59/distinct-2-w2-575d59-kserve-d4565b76c-hdgbf" Oct 29 22:56:08.989455 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:08.989442 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f3092a4d-c79c57cf-wzjtm"] Oct 29 22:56:09.066497 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:09.063326 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f3092a4d-c79c57cf-wzjtm"] Oct 29 22:56:09.066497 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:09.064011 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f3092a4d-c79c57cf-wzjtm" Oct 29 22:56:09.106826 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:09.098425 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-f3092a4d-serving-cert\"" Oct 29 22:56:09.106826 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:09.103450 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:56:09.242355 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:09.198892 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/979cc0cb-b14a-4c0a-84b1-d4fe1cad4279-openshift-service-ca\") pod \"maas-api-e2e-mt-f3092a4d-c79c57cf-wzjtm\" (UID: \"979cc0cb-b14a-4c0a-84b1-d4fe1cad4279\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f3092a4d-c79c57cf-wzjtm" Oct 29 22:56:09.268383 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:09.261731 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/979cc0cb-b14a-4c0a-84b1-d4fe1cad4279-metrics-certs\") pod \"maas-api-e2e-mt-f3092a4d-c79c57cf-wzjtm\" (UID: \"979cc0cb-b14a-4c0a-84b1-d4fe1cad4279\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f3092a4d-c79c57cf-wzjtm" Oct 29 22:56:09.268383 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:09.262043 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/979cc0cb-b14a-4c0a-84b1-d4fe1cad4279-maas-api-tls\") pod \"maas-api-e2e-mt-f3092a4d-c79c57cf-wzjtm\" (UID: \"979cc0cb-b14a-4c0a-84b1-d4fe1cad4279\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f3092a4d-c79c57cf-wzjtm" Oct 29 22:56:09.268383 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:09.262147 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbhkb\" (UniqueName: \"kubernetes.io/projected/979cc0cb-b14a-4c0a-84b1-d4fe1cad4279-kube-api-access-fbhkb\") pod \"maas-api-e2e-mt-f3092a4d-c79c57cf-wzjtm\" (UID: \"979cc0cb-b14a-4c0a-84b1-d4fe1cad4279\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f3092a4d-c79c57cf-wzjtm" Oct 29 22:56:09.333358 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:09.278415 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-bclh8" Oct 29 22:56:09.347843 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:09.347694 5057 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-31226c/distinct-w6-31226c-kserve-858d95fd76-bkf7c" podUID="f3fc11c6-03c9-4314-99c3-f857b149415c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:56:09.350454 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:09.348853 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-08e15d/distinct-w0-08e15d-kserve-85695c78bb-p67gk" Oct 29 22:56:09.363710 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:09.363682 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/979cc0cb-b14a-4c0a-84b1-d4fe1cad4279-openshift-service-ca\") pod \"maas-api-e2e-mt-f3092a4d-c79c57cf-wzjtm\" (UID: \"979cc0cb-b14a-4c0a-84b1-d4fe1cad4279\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f3092a4d-c79c57cf-wzjtm" Oct 29 22:56:09.363996 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:09.363977 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/979cc0cb-b14a-4c0a-84b1-d4fe1cad4279-metrics-certs\") pod \"maas-api-e2e-mt-f3092a4d-c79c57cf-wzjtm\" (UID: \"979cc0cb-b14a-4c0a-84b1-d4fe1cad4279\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f3092a4d-c79c57cf-wzjtm" Oct 29 22:56:09.364310 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:09.364291 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/979cc0cb-b14a-4c0a-84b1-d4fe1cad4279-maas-api-tls\") pod \"maas-api-e2e-mt-f3092a4d-c79c57cf-wzjtm\" (UID: \"979cc0cb-b14a-4c0a-84b1-d4fe1cad4279\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f3092a4d-c79c57cf-wzjtm" Oct 29 22:56:09.364589 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:09.364557 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbhkb\" (UniqueName: \"kubernetes.io/projected/979cc0cb-b14a-4c0a-84b1-d4fe1cad4279-kube-api-access-fbhkb\") pod \"maas-api-e2e-mt-f3092a4d-c79c57cf-wzjtm\" (UID: \"979cc0cb-b14a-4c0a-84b1-d4fe1cad4279\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f3092a4d-c79c57cf-wzjtm" Oct 29 22:56:09.365181 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:09.365131 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/979cc0cb-b14a-4c0a-84b1-d4fe1cad4279-openshift-service-ca\") pod \"maas-api-e2e-mt-f3092a4d-c79c57cf-wzjtm\" (UID: \"979cc0cb-b14a-4c0a-84b1-d4fe1cad4279\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f3092a4d-c79c57cf-wzjtm" Oct 29 22:56:09.407907 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:09.404894 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/979cc0cb-b14a-4c0a-84b1-d4fe1cad4279-maas-api-tls\") pod \"maas-api-e2e-mt-f3092a4d-c79c57cf-wzjtm\" (UID: \"979cc0cb-b14a-4c0a-84b1-d4fe1cad4279\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f3092a4d-c79c57cf-wzjtm" Oct 29 22:56:09.446638 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:09.446560 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/979cc0cb-b14a-4c0a-84b1-d4fe1cad4279-metrics-certs\") pod \"maas-api-e2e-mt-f3092a4d-c79c57cf-wzjtm\" (UID: \"979cc0cb-b14a-4c0a-84b1-d4fe1cad4279\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f3092a4d-c79c57cf-wzjtm" Oct 29 22:56:09.460220 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:09.460190 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-bclh8" Oct 29 22:56:09.460402 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:09.460285 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-17394a/distinct-2-w3-17394a-kserve-6545c766c4-h4chf" Oct 29 22:56:09.460402 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:09.460346 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-31226c/distinct-w6-31226c-kserve-858d95fd76-bkf7c" Oct 29 22:56:09.463638 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:09.461190 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbhkb\" (UniqueName: \"kubernetes.io/projected/979cc0cb-b14a-4c0a-84b1-d4fe1cad4279-kube-api-access-fbhkb\") pod \"maas-api-e2e-mt-f3092a4d-c79c57cf-wzjtm\" (UID: \"979cc0cb-b14a-4c0a-84b1-d4fe1cad4279\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f3092a4d-c79c57cf-wzjtm" Oct 29 22:56:09.468183 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:09.468156 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:56:09.535173 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:09.534502 5057 patch_prober.go:29] interesting pod/e2e-worker-w5-b2921b-openshift-default-5547c88fff-hfrrx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Oct 29 22:56:09.539673 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:09.539632 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-hfrrx" podUID="43a9077f-2bb9-4801-abba-687de8a45076" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Oct 29 22:56:09.574493 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:09.574466 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f3092a4d-c79c57cf-wzjtm" Oct 29 22:56:09.766722 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:09.766689 5057 patch_prober.go:29] interesting pod/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-4mwxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Oct 29 22:56:09.766868 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:09.766739 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-4mwxl" podUID="bd32fc20-abda-413c-88c1-6b3b6156e777" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Oct 29 22:56:10.021743 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:10.021655 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f3092a4d-c79c57cf-wzjtm"] Oct 29 22:56:10.026914 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:56:10.026876 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod979cc0cb_b14a_4c0a_84b1_d4fe1cad4279.slice/crio-e3a2e1fb495542b831201f69c12ab945dac6abecddc98e628fb8204f7363b26b WatchSource:0}: Error finding container e3a2e1fb495542b831201f69c12ab945dac6abecddc98e628fb8204f7363b26b: Status 404 returned error can't find the container with id e3a2e1fb495542b831201f69c12ab945dac6abecddc98e628fb8204f7363b26b Oct 29 22:56:10.028425 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:10.028391 5057 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 29 22:56:10.707402 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:10.707299 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f3092a4d-c79c57cf-wzjtm" event={"ID":"979cc0cb-b14a-4c0a-84b1-d4fe1cad4279","Type":"ContainerStarted","Data":"590d522f203a0a01b1257aea23379cde3af0fea2bf8446b01135fb731a460798"} Oct 29 22:56:10.707402 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:10.707382 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f3092a4d-c79c57cf-wzjtm" event={"ID":"979cc0cb-b14a-4c0a-84b1-d4fe1cad4279","Type":"ContainerStarted","Data":"e3a2e1fb495542b831201f69c12ab945dac6abecddc98e628fb8204f7363b26b"} Oct 29 22:56:10.731449 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:10.708666 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f3092a4d-c79c57cf-wzjtm" Oct 29 22:56:10.751787 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:10.751729 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f3092a4d-c79c57cf-wzjtm" podStartSLOduration=3.324854535 podStartE2EDuration="3.751714402s" totalImagesPullingTime="426.859867ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:56:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:56:10.746161993 +0000 UTC m=+1351.529375419" watchObservedRunningTime="2026-10-29 22:56:10.751714402 +0000 UTC m=+1351.534927827" Oct 29 22:56:10.767455 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:10.767430 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-4mwxl" Oct 29 22:56:10.768767 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:10.768746 5057 patch_prober.go:29] interesting pod/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-4mwxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Oct 29 22:56:10.768835 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:10.768786 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-4mwxl" podUID="bd32fc20-abda-413c-88c1-6b3b6156e777" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Oct 29 22:56:11.188409 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:11.188009 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-hfrrx" Oct 29 22:56:11.222445 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:11.215619 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-hfrrx" Oct 29 22:56:11.253620 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:11.253580 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:56:11.767238 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:11.767199 5057 patch_prober.go:29] interesting pod/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-4mwxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Oct 29 22:56:11.767655 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:11.767264 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-4mwxl" podUID="bd32fc20-abda-413c-88c1-6b3b6156e777" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Oct 29 22:56:12.208019 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:12.207984 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-nqs5j"] Oct 29 22:56:12.214107 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:12.214082 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-nqs5j" Oct 29 22:56:12.248476 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:12.248444 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-nqs5j"] Oct 29 22:56:12.377774 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:12.377736 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e92e0a25-60cc-4250-a75d-537ed970fd75-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-nqs5j\" (UID: \"e92e0a25-60cc-4250-a75d-537ed970fd75\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-nqs5j" Oct 29 22:56:12.377983 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:12.377915 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8bck\" (UniqueName: \"kubernetes.io/projected/e92e0a25-60cc-4250-a75d-537ed970fd75-kube-api-access-x8bck\") pod \"maas-default-gateway-openshift-default-6db586dd6c-nqs5j\" (UID: \"e92e0a25-60cc-4250-a75d-537ed970fd75\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-nqs5j" Oct 29 22:56:12.378054 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:12.377983 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e92e0a25-60cc-4250-a75d-537ed970fd75-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-nqs5j\" (UID: \"e92e0a25-60cc-4250-a75d-537ed970fd75\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-nqs5j" Oct 29 22:56:12.378054 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:12.378037 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e92e0a25-60cc-4250-a75d-537ed970fd75-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-nqs5j\" (UID: \"e92e0a25-60cc-4250-a75d-537ed970fd75\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-nqs5j" Oct 29 22:56:12.378168 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:12.378074 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e92e0a25-60cc-4250-a75d-537ed970fd75-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-nqs5j\" (UID: \"e92e0a25-60cc-4250-a75d-537ed970fd75\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-nqs5j" Oct 29 22:56:12.378168 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:12.378100 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e92e0a25-60cc-4250-a75d-537ed970fd75-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-nqs5j\" (UID: \"e92e0a25-60cc-4250-a75d-537ed970fd75\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-nqs5j" Oct 29 22:56:12.378269 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:12.378223 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e92e0a25-60cc-4250-a75d-537ed970fd75-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-nqs5j\" (UID: \"e92e0a25-60cc-4250-a75d-537ed970fd75\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-nqs5j" Oct 29 22:56:12.378325 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:12.378280 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e92e0a25-60cc-4250-a75d-537ed970fd75-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-nqs5j\" (UID: \"e92e0a25-60cc-4250-a75d-537ed970fd75\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-nqs5j" Oct 29 22:56:12.378325 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:12.378308 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e92e0a25-60cc-4250-a75d-537ed970fd75-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-nqs5j\" (UID: \"e92e0a25-60cc-4250-a75d-537ed970fd75\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-nqs5j" Oct 29 22:56:12.480768 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:12.480149 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8bck\" (UniqueName: \"kubernetes.io/projected/e92e0a25-60cc-4250-a75d-537ed970fd75-kube-api-access-x8bck\") pod \"maas-default-gateway-openshift-default-6db586dd6c-nqs5j\" (UID: \"e92e0a25-60cc-4250-a75d-537ed970fd75\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-nqs5j" Oct 29 22:56:12.480768 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:12.480237 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e92e0a25-60cc-4250-a75d-537ed970fd75-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-nqs5j\" (UID: \"e92e0a25-60cc-4250-a75d-537ed970fd75\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-nqs5j" Oct 29 22:56:12.480768 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:12.480300 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e92e0a25-60cc-4250-a75d-537ed970fd75-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-nqs5j\" (UID: \"e92e0a25-60cc-4250-a75d-537ed970fd75\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-nqs5j" Oct 29 22:56:12.480768 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:12.480371 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e92e0a25-60cc-4250-a75d-537ed970fd75-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-nqs5j\" (UID: \"e92e0a25-60cc-4250-a75d-537ed970fd75\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-nqs5j" Oct 29 22:56:12.480768 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:12.480399 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e92e0a25-60cc-4250-a75d-537ed970fd75-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-nqs5j\" (UID: \"e92e0a25-60cc-4250-a75d-537ed970fd75\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-nqs5j" Oct 29 22:56:12.480768 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:12.480559 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e92e0a25-60cc-4250-a75d-537ed970fd75-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-nqs5j\" (UID: \"e92e0a25-60cc-4250-a75d-537ed970fd75\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-nqs5j" Oct 29 22:56:12.480768 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:12.480645 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e92e0a25-60cc-4250-a75d-537ed970fd75-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-nqs5j\" (UID: \"e92e0a25-60cc-4250-a75d-537ed970fd75\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-nqs5j" Oct 29 22:56:12.480768 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:12.480676 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e92e0a25-60cc-4250-a75d-537ed970fd75-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-nqs5j\" (UID: \"e92e0a25-60cc-4250-a75d-537ed970fd75\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-nqs5j" Oct 29 22:56:12.480768 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:12.480750 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e92e0a25-60cc-4250-a75d-537ed970fd75-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-nqs5j\" (UID: \"e92e0a25-60cc-4250-a75d-537ed970fd75\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-nqs5j" Oct 29 22:56:12.481433 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:12.480928 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e92e0a25-60cc-4250-a75d-537ed970fd75-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-nqs5j\" (UID: \"e92e0a25-60cc-4250-a75d-537ed970fd75\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-nqs5j" Oct 29 22:56:12.481433 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:12.481100 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e92e0a25-60cc-4250-a75d-537ed970fd75-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-nqs5j\" (UID: \"e92e0a25-60cc-4250-a75d-537ed970fd75\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-nqs5j" Oct 29 22:56:12.481433 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:12.481157 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e92e0a25-60cc-4250-a75d-537ed970fd75-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-nqs5j\" (UID: \"e92e0a25-60cc-4250-a75d-537ed970fd75\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-nqs5j" Oct 29 22:56:12.482007 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:12.481969 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e92e0a25-60cc-4250-a75d-537ed970fd75-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-nqs5j\" (UID: \"e92e0a25-60cc-4250-a75d-537ed970fd75\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-nqs5j" Oct 29 22:56:12.482323 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:12.482298 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e92e0a25-60cc-4250-a75d-537ed970fd75-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-nqs5j\" (UID: \"e92e0a25-60cc-4250-a75d-537ed970fd75\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-nqs5j" Oct 29 22:56:12.486371 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:12.485144 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e92e0a25-60cc-4250-a75d-537ed970fd75-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-nqs5j\" (UID: \"e92e0a25-60cc-4250-a75d-537ed970fd75\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-nqs5j" Oct 29 22:56:12.486371 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:12.485298 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e92e0a25-60cc-4250-a75d-537ed970fd75-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-nqs5j\" (UID: \"e92e0a25-60cc-4250-a75d-537ed970fd75\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-nqs5j" Oct 29 22:56:12.502671 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:12.502649 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e92e0a25-60cc-4250-a75d-537ed970fd75-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-nqs5j\" (UID: \"e92e0a25-60cc-4250-a75d-537ed970fd75\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-nqs5j" Oct 29 22:56:12.503563 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:12.503538 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8bck\" (UniqueName: \"kubernetes.io/projected/e92e0a25-60cc-4250-a75d-537ed970fd75-kube-api-access-x8bck\") pod \"maas-default-gateway-openshift-default-6db586dd6c-nqs5j\" (UID: \"e92e0a25-60cc-4250-a75d-537ed970fd75\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-nqs5j" Oct 29 22:56:12.546538 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:12.546513 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-nqs5j" Oct 29 22:56:12.879856 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:56:12.879812 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode92e0a25_60cc_4250_a75d_537ed970fd75.slice/crio-d17328e6ecca338321bfd68f579d120e0cff4db018383ad0b27eff84f26f5e76 WatchSource:0}: Error finding container d17328e6ecca338321bfd68f579d120e0cff4db018383ad0b27eff84f26f5e76: Status 404 returned error can't find the container with id d17328e6ecca338321bfd68f579d120e0cff4db018383ad0b27eff84f26f5e76 Oct 29 22:56:12.880274 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:12.879896 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-nqs5j"] Oct 29 22:56:12.882910 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:12.882871 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:56:12.883019 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:12.882957 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:56:12.883019 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:12.883000 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:56:12.928252 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:12.928213 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:56:13.741785 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:13.741743 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-nqs5j" event={"ID":"e92e0a25-60cc-4250-a75d-537ed970fd75","Type":"ContainerStarted","Data":"544c3043d8e8310fc575a6b477dae151e759b093cb37b25fe5e6890439cad09b"} Oct 29 22:56:13.741999 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:13.741814 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-nqs5j" event={"ID":"e92e0a25-60cc-4250-a75d-537ed970fd75","Type":"ContainerStarted","Data":"d17328e6ecca338321bfd68f579d120e0cff4db018383ad0b27eff84f26f5e76"} Oct 29 22:56:13.820362 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:13.809506 5057 patch_prober.go:29] interesting pod/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-4mwxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 22:56:13.820362 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:13.809626 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-4mwxl" podUID="bd32fc20-abda-413c-88c1-6b3b6156e777" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:56:13.820362 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:13.818876 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-nqs5j" podStartSLOduration=1.818852486 podStartE2EDuration="1.818852486s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:56:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:56:13.793815061 +0000 UTC m=+1354.577028489" watchObservedRunningTime="2026-10-29 22:56:13.818852486 +0000 UTC m=+1354.602065915" Oct 29 22:56:14.254534 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:14.254494 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-4mwxl" Oct 29 22:56:14.255884 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:14.255859 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-4mwxl" Oct 29 22:56:14.524559 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:14.524476 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:56:14.547364 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:14.547325 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-nqs5j" Oct 29 22:56:14.549260 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:14.549237 5057 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-nqs5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Oct 29 22:56:14.549394 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:14.549282 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-nqs5j" podUID="e92e0a25-60cc-4250-a75d-537ed970fd75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Oct 29 22:56:15.546862 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:15.546827 5057 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-nqs5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Oct 29 22:56:15.547369 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:15.546883 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-nqs5j" podUID="e92e0a25-60cc-4250-a75d-537ed970fd75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Oct 29 22:56:16.008811 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:16.008759 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:56:16.547337 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:16.547298 5057 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-nqs5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Oct 29 22:56:16.547728 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:16.547363 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-nqs5j" podUID="e92e0a25-60cc-4250-a75d-537ed970fd75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Oct 29 22:56:17.547189 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:17.547141 5057 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-nqs5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Oct 29 22:56:17.547381 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:17.547274 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-nqs5j" podUID="e92e0a25-60cc-4250-a75d-537ed970fd75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Oct 29 22:56:17.575131 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:17.575067 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:56:17.740356 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:17.739891 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f3092a4d-c79c57cf-wzjtm" Oct 29 22:56:18.321688 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:18.321649 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:56:18.547809 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:18.547777 5057 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-nqs5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Oct 29 22:56:18.548292 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:18.548254 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-nqs5j" podUID="e92e0a25-60cc-4250-a75d-537ed970fd75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Oct 29 22:56:18.811535 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:18.811506 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:56:18.968358 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:18.965227 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:56:19.041523 ip-10-0-140-74 kubenswrapper[5057]: E1029 22:56:19.041473 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d2dnn" containerName="extractor" Oct 29 22:56:19.547704 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:19.547666 5057 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-nqs5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Oct 29 22:56:19.547878 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:19.547742 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-nqs5j" podUID="e92e0a25-60cc-4250-a75d-537ed970fd75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Oct 29 22:56:19.550198 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:19.550170 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:56:20.547306 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:20.547273 5057 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-nqs5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Oct 29 22:56:20.547513 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:20.547381 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-nqs5j" podUID="e92e0a25-60cc-4250-a75d-537ed970fd75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Oct 29 22:56:21.111598 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:21.111566 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-e4a049-66d6fd5c46-c5sgf"] Oct 29 22:56:21.117782 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:21.117760 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e4a049-66d6fd5c46-c5sgf" Oct 29 22:56:21.144241 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:21.144211 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-e4a049-serving-cert\"" Oct 29 22:56:21.155889 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:21.155770 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-e4a049-66d6fd5c46-c5sgf"] Oct 29 22:56:21.252998 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:21.252953 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/50643c46-b0b4-4a44-9c51-539844e01e44-maas-api-tls\") pod \"maas-api-e2e-api-a-e4a049-66d6fd5c46-c5sgf\" (UID: \"50643c46-b0b4-4a44-9c51-539844e01e44\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e4a049-66d6fd5c46-c5sgf" Oct 29 22:56:21.253202 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:21.253175 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/50643c46-b0b4-4a44-9c51-539844e01e44-openshift-service-ca\") pod \"maas-api-e2e-api-a-e4a049-66d6fd5c46-c5sgf\" (UID: \"50643c46-b0b4-4a44-9c51-539844e01e44\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e4a049-66d6fd5c46-c5sgf" Oct 29 22:56:21.253296 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:21.253274 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50643c46-b0b4-4a44-9c51-539844e01e44-metrics-certs\") pod \"maas-api-e2e-api-a-e4a049-66d6fd5c46-c5sgf\" (UID: \"50643c46-b0b4-4a44-9c51-539844e01e44\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e4a049-66d6fd5c46-c5sgf" Oct 29 22:56:21.253484 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:21.253407 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-msjv9\" (UniqueName: \"kubernetes.io/projected/50643c46-b0b4-4a44-9c51-539844e01e44-kube-api-access-msjv9\") pod \"maas-api-e2e-api-a-e4a049-66d6fd5c46-c5sgf\" (UID: \"50643c46-b0b4-4a44-9c51-539844e01e44\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e4a049-66d6fd5c46-c5sgf" Oct 29 22:56:21.301185 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:21.301131 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:56:21.360656 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:21.360620 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/50643c46-b0b4-4a44-9c51-539844e01e44-openshift-service-ca\") pod \"maas-api-e2e-api-a-e4a049-66d6fd5c46-c5sgf\" (UID: \"50643c46-b0b4-4a44-9c51-539844e01e44\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e4a049-66d6fd5c46-c5sgf" Oct 29 22:56:21.360856 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:21.360758 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50643c46-b0b4-4a44-9c51-539844e01e44-metrics-certs\") pod \"maas-api-e2e-api-a-e4a049-66d6fd5c46-c5sgf\" (UID: \"50643c46-b0b4-4a44-9c51-539844e01e44\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e4a049-66d6fd5c46-c5sgf" Oct 29 22:56:21.360856 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:21.360833 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-msjv9\" (UniqueName: \"kubernetes.io/projected/50643c46-b0b4-4a44-9c51-539844e01e44-kube-api-access-msjv9\") pod \"maas-api-e2e-api-a-e4a049-66d6fd5c46-c5sgf\" (UID: \"50643c46-b0b4-4a44-9c51-539844e01e44\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e4a049-66d6fd5c46-c5sgf" Oct 29 22:56:21.361370 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:21.361219 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/50643c46-b0b4-4a44-9c51-539844e01e44-maas-api-tls\") pod \"maas-api-e2e-api-a-e4a049-66d6fd5c46-c5sgf\" (UID: \"50643c46-b0b4-4a44-9c51-539844e01e44\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e4a049-66d6fd5c46-c5sgf" Oct 29 22:56:21.364540 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:21.364479 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/50643c46-b0b4-4a44-9c51-539844e01e44-openshift-service-ca\") pod \"maas-api-e2e-api-a-e4a049-66d6fd5c46-c5sgf\" (UID: \"50643c46-b0b4-4a44-9c51-539844e01e44\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e4a049-66d6fd5c46-c5sgf" Oct 29 22:56:21.367510 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:21.367465 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/50643c46-b0b4-4a44-9c51-539844e01e44-maas-api-tls\") pod \"maas-api-e2e-api-a-e4a049-66d6fd5c46-c5sgf\" (UID: \"50643c46-b0b4-4a44-9c51-539844e01e44\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e4a049-66d6fd5c46-c5sgf" Oct 29 22:56:21.481876 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:21.481788 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:56:21.487597 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:21.487536 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50643c46-b0b4-4a44-9c51-539844e01e44-metrics-certs\") pod \"maas-api-e2e-api-a-e4a049-66d6fd5c46-c5sgf\" (UID: \"50643c46-b0b4-4a44-9c51-539844e01e44\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e4a049-66d6fd5c46-c5sgf" Oct 29 22:56:21.491281 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:21.491247 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-msjv9\" (UniqueName: \"kubernetes.io/projected/50643c46-b0b4-4a44-9c51-539844e01e44-kube-api-access-msjv9\") pod \"maas-api-e2e-api-a-e4a049-66d6fd5c46-c5sgf\" (UID: \"50643c46-b0b4-4a44-9c51-539844e01e44\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e4a049-66d6fd5c46-c5sgf" Oct 29 22:56:21.765584 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:21.765544 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e4a049-66d6fd5c46-c5sgf" Oct 29 22:56:22.115278 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:22.115248 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-e4a049-66d6fd5c46-c5sgf"] Oct 29 22:56:22.115818 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:56:22.115786 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50643c46_b0b4_4a44_9c51_539844e01e44.slice/crio-0185e7855ae755f388c2025c79a3c0dfc1a10b537a8444593107308c0b7b0ed2 WatchSource:0}: Error finding container 0185e7855ae755f388c2025c79a3c0dfc1a10b537a8444593107308c0b7b0ed2: Status 404 returned error can't find the container with id 0185e7855ae755f388c2025c79a3c0dfc1a10b537a8444593107308c0b7b0ed2 Oct 29 22:56:22.181490 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:22.181461 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-nqs5j" Oct 29 22:56:22.182493 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:22.182365 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-nqs5j" Oct 29 22:56:22.183751 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:22.183698 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-nqs5j" Oct 29 22:56:22.873193 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:22.873107 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e4a049-66d6fd5c46-c5sgf" event={"ID":"50643c46-b0b4-4a44-9c51-539844e01e44","Type":"ContainerStarted","Data":"71b991db9e0eeb1a83f14f3976f2e3b04cfa20fba936562ff5684ae9c2c5af52"} Oct 29 22:56:22.873193 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:22.873172 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e4a049-66d6fd5c46-c5sgf" event={"ID":"50643c46-b0b4-4a44-9c51-539844e01e44","Type":"ContainerStarted","Data":"0185e7855ae755f388c2025c79a3c0dfc1a10b537a8444593107308c0b7b0ed2"} Oct 29 22:56:22.873431 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:22.873199 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e4a049-66d6fd5c46-c5sgf" Oct 29 22:56:22.918073 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:22.918007 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e4a049-66d6fd5c46-c5sgf" podStartSLOduration=1.578143749 podStartE2EDuration="1.917988066s" totalImagesPullingTime="339.844317ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:56:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:56:22.911139461 +0000 UTC m=+1363.694352887" watchObservedRunningTime="2026-10-29 22:56:22.917988066 +0000 UTC m=+1363.701201493" Oct 29 22:56:22.972922 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:22.972892 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:56:23.097376 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:23.097312 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:56:23.953503 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:23.953470 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:56:24.726389 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:24.726317 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:56:24.839608 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:24.839573 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:56:25.811240 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:25.811198 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:56:27.850159 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:27.850125 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:56:28.400125 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:28.400089 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:56:28.886086 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:28.886055 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e4a049-66d6fd5c46-c5sgf" Oct 29 22:56:29.832212 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:29.832176 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:56:31.428451 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:31.428418 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:56:31.611238 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:31.611200 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:56:32.914069 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:32.914026 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:56:33.218016 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:33.217972 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v2tfm"] Oct 29 22:56:34.522957 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:34.522923 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:56:36.197086 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:36.197042 5057 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-17394a/distinct-2-w3-17394a-kserve-6545c766c4-h4chf" podUID="f29e8b65-b7dd-45d4-86b0-f13a29ef8e66" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:56:36.304744 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:36.304706 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-qpb7q"] Oct 29 22:56:36.458826 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:36.453397 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-08e15d/distinct-w0-08e15d-kserve-85695c78bb-p67gk" podUID="e664f012-3b80-4a8a-ac19-d767d92c0a33" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:56:36.715653 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:36.715549 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:56:36.716730 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:36.716589 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-qpb7q"] Oct 29 22:56:36.718413 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:36.716964 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-qpb7q" Oct 29 22:56:36.839352 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:36.831456 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/708063fe-d579-4b1b-982d-63faeb84dac7-workload-socket\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-qpb7q\" (UID: \"708063fe-d579-4b1b-982d-63faeb84dac7\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-qpb7q" Oct 29 22:56:36.839352 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:36.831523 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/708063fe-d579-4b1b-982d-63faeb84dac7-istio-data\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-qpb7q\" (UID: \"708063fe-d579-4b1b-982d-63faeb84dac7\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-qpb7q" Oct 29 22:56:36.839352 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:36.831576 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/708063fe-d579-4b1b-982d-63faeb84dac7-workload-certs\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-qpb7q\" (UID: \"708063fe-d579-4b1b-982d-63faeb84dac7\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-qpb7q" Oct 29 22:56:36.839352 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:36.831700 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/708063fe-d579-4b1b-982d-63faeb84dac7-istio-podinfo\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-qpb7q\" (UID: \"708063fe-d579-4b1b-982d-63faeb84dac7\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-qpb7q" Oct 29 22:56:36.839352 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:36.831787 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/708063fe-d579-4b1b-982d-63faeb84dac7-istiod-ca-cert\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-qpb7q\" (UID: \"708063fe-d579-4b1b-982d-63faeb84dac7\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-qpb7q" Oct 29 22:56:36.839352 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:36.831856 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/708063fe-d579-4b1b-982d-63faeb84dac7-credential-socket\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-qpb7q\" (UID: \"708063fe-d579-4b1b-982d-63faeb84dac7\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-qpb7q" Oct 29 22:56:36.839352 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:36.831929 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/708063fe-d579-4b1b-982d-63faeb84dac7-istio-token\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-qpb7q\" (UID: \"708063fe-d579-4b1b-982d-63faeb84dac7\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-qpb7q" Oct 29 22:56:36.839352 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:36.831977 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4ps9\" (UniqueName: \"kubernetes.io/projected/708063fe-d579-4b1b-982d-63faeb84dac7-kube-api-access-w4ps9\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-qpb7q\" (UID: \"708063fe-d579-4b1b-982d-63faeb84dac7\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-qpb7q" Oct 29 22:56:36.839352 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:36.832017 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/708063fe-d579-4b1b-982d-63faeb84dac7-istio-envoy\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-qpb7q\" (UID: \"708063fe-d579-4b1b-982d-63faeb84dac7\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-qpb7q" Oct 29 22:56:36.950358 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:36.947477 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/708063fe-d579-4b1b-982d-63faeb84dac7-workload-socket\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-qpb7q\" (UID: \"708063fe-d579-4b1b-982d-63faeb84dac7\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-qpb7q" Oct 29 22:56:36.950358 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:36.947551 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/708063fe-d579-4b1b-982d-63faeb84dac7-istio-data\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-qpb7q\" (UID: \"708063fe-d579-4b1b-982d-63faeb84dac7\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-qpb7q" Oct 29 22:56:36.950358 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:36.947599 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/708063fe-d579-4b1b-982d-63faeb84dac7-workload-certs\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-qpb7q\" (UID: \"708063fe-d579-4b1b-982d-63faeb84dac7\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-qpb7q" Oct 29 22:56:36.950358 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:36.947708 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/708063fe-d579-4b1b-982d-63faeb84dac7-istio-podinfo\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-qpb7q\" (UID: \"708063fe-d579-4b1b-982d-63faeb84dac7\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-qpb7q" Oct 29 22:56:36.950358 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:36.947775 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/708063fe-d579-4b1b-982d-63faeb84dac7-istiod-ca-cert\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-qpb7q\" (UID: \"708063fe-d579-4b1b-982d-63faeb84dac7\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-qpb7q" Oct 29 22:56:36.950358 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:36.947831 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/708063fe-d579-4b1b-982d-63faeb84dac7-credential-socket\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-qpb7q\" (UID: \"708063fe-d579-4b1b-982d-63faeb84dac7\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-qpb7q" Oct 29 22:56:36.950358 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:36.947884 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/708063fe-d579-4b1b-982d-63faeb84dac7-istio-token\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-qpb7q\" (UID: \"708063fe-d579-4b1b-982d-63faeb84dac7\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-qpb7q" Oct 29 22:56:36.950358 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:36.947927 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4ps9\" (UniqueName: \"kubernetes.io/projected/708063fe-d579-4b1b-982d-63faeb84dac7-kube-api-access-w4ps9\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-qpb7q\" (UID: \"708063fe-d579-4b1b-982d-63faeb84dac7\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-qpb7q" Oct 29 22:56:36.950358 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:36.947964 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/708063fe-d579-4b1b-982d-63faeb84dac7-istio-envoy\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-qpb7q\" (UID: \"708063fe-d579-4b1b-982d-63faeb84dac7\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-qpb7q" Oct 29 22:56:36.994929 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:36.985086 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/708063fe-d579-4b1b-982d-63faeb84dac7-workload-certs\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-qpb7q\" (UID: \"708063fe-d579-4b1b-982d-63faeb84dac7\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-qpb7q" Oct 29 22:56:36.994929 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:36.986680 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/708063fe-d579-4b1b-982d-63faeb84dac7-workload-socket\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-qpb7q\" (UID: \"708063fe-d579-4b1b-982d-63faeb84dac7\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-qpb7q" Oct 29 22:56:36.994929 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:36.986969 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/708063fe-d579-4b1b-982d-63faeb84dac7-istio-data\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-qpb7q\" (UID: \"708063fe-d579-4b1b-982d-63faeb84dac7\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-qpb7q" Oct 29 22:56:36.994929 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:36.987205 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/708063fe-d579-4b1b-982d-63faeb84dac7-credential-socket\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-qpb7q\" (UID: \"708063fe-d579-4b1b-982d-63faeb84dac7\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-qpb7q" Oct 29 22:56:37.058377 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:37.026215 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wcqcf"] Oct 29 22:56:37.195391 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:37.027312 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/708063fe-d579-4b1b-982d-63faeb84dac7-istiod-ca-cert\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-qpb7q\" (UID: \"708063fe-d579-4b1b-982d-63faeb84dac7\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-qpb7q" Oct 29 22:56:37.195391 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:37.154919 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wcqcf" Oct 29 22:56:37.195391 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:37.172053 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/708063fe-d579-4b1b-982d-63faeb84dac7-istio-token\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-qpb7q\" (UID: \"708063fe-d579-4b1b-982d-63faeb84dac7\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-qpb7q" Oct 29 22:56:37.195391 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:37.173859 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Oct 29 22:56:37.195391 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:37.174072 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-dpk7p\"" Oct 29 22:56:37.195391 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:37.174119 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Oct 29 22:56:37.262563 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:37.256537 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/708063fe-d579-4b1b-982d-63faeb84dac7-istio-podinfo\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-qpb7q\" (UID: \"708063fe-d579-4b1b-982d-63faeb84dac7\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-qpb7q" Oct 29 22:56:37.340362 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:37.336805 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91fd5f87-ec89-46b8-8d57-e07cfec8a5b7-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wcqcf\" (UID: \"91fd5f87-ec89-46b8-8d57-e07cfec8a5b7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wcqcf" Oct 29 22:56:37.340362 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:37.336922 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/91fd5f87-ec89-46b8-8d57-e07cfec8a5b7-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wcqcf\" (UID: \"91fd5f87-ec89-46b8-8d57-e07cfec8a5b7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wcqcf" Oct 29 22:56:37.340362 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:37.337012 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/91fd5f87-ec89-46b8-8d57-e07cfec8a5b7-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wcqcf\" (UID: \"91fd5f87-ec89-46b8-8d57-e07cfec8a5b7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wcqcf" Oct 29 22:56:37.340362 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:37.337095 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/91fd5f87-ec89-46b8-8d57-e07cfec8a5b7-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wcqcf\" (UID: \"91fd5f87-ec89-46b8-8d57-e07cfec8a5b7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wcqcf" Oct 29 22:56:37.340362 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:37.337158 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zf5s\" (UniqueName: \"kubernetes.io/projected/91fd5f87-ec89-46b8-8d57-e07cfec8a5b7-kube-api-access-7zf5s\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wcqcf\" (UID: \"91fd5f87-ec89-46b8-8d57-e07cfec8a5b7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wcqcf" Oct 29 22:56:37.340362 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:37.337380 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/91fd5f87-ec89-46b8-8d57-e07cfec8a5b7-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wcqcf\" (UID: \"91fd5f87-ec89-46b8-8d57-e07cfec8a5b7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wcqcf" Oct 29 22:56:37.340362 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:37.337441 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/91fd5f87-ec89-46b8-8d57-e07cfec8a5b7-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wcqcf\" (UID: \"91fd5f87-ec89-46b8-8d57-e07cfec8a5b7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wcqcf" Oct 29 22:56:37.468746 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:37.466097 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Oct 29 22:56:37.468746 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:37.466588 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/708063fe-d579-4b1b-982d-63faeb84dac7-istio-envoy\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-qpb7q\" (UID: \"708063fe-d579-4b1b-982d-63faeb84dac7\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-qpb7q" Oct 29 22:56:37.527481 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:37.480234 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4ps9\" (UniqueName: \"kubernetes.io/projected/708063fe-d579-4b1b-982d-63faeb84dac7-kube-api-access-w4ps9\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-qpb7q\" (UID: \"708063fe-d579-4b1b-982d-63faeb84dac7\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-qpb7q" Oct 29 22:56:37.590364 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:37.532658 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91fd5f87-ec89-46b8-8d57-e07cfec8a5b7-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wcqcf\" (UID: \"91fd5f87-ec89-46b8-8d57-e07cfec8a5b7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wcqcf" Oct 29 22:56:37.594358 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:37.591816 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/91fd5f87-ec89-46b8-8d57-e07cfec8a5b7-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wcqcf\" (UID: \"91fd5f87-ec89-46b8-8d57-e07cfec8a5b7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wcqcf" Oct 29 22:56:37.594358 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:37.591923 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/91fd5f87-ec89-46b8-8d57-e07cfec8a5b7-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wcqcf\" (UID: \"91fd5f87-ec89-46b8-8d57-e07cfec8a5b7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wcqcf" Oct 29 22:56:37.594358 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:37.592009 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/91fd5f87-ec89-46b8-8d57-e07cfec8a5b7-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wcqcf\" (UID: \"91fd5f87-ec89-46b8-8d57-e07cfec8a5b7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wcqcf" Oct 29 22:56:37.602258 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:37.595814 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zf5s\" (UniqueName: \"kubernetes.io/projected/91fd5f87-ec89-46b8-8d57-e07cfec8a5b7-kube-api-access-7zf5s\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wcqcf\" (UID: \"91fd5f87-ec89-46b8-8d57-e07cfec8a5b7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wcqcf" Oct 29 22:56:37.602258 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:37.596040 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/91fd5f87-ec89-46b8-8d57-e07cfec8a5b7-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wcqcf\" (UID: \"91fd5f87-ec89-46b8-8d57-e07cfec8a5b7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wcqcf" Oct 29 22:56:37.602258 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:37.596102 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/91fd5f87-ec89-46b8-8d57-e07cfec8a5b7-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wcqcf\" (UID: \"91fd5f87-ec89-46b8-8d57-e07cfec8a5b7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wcqcf" Oct 29 22:56:37.653366 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:37.619696 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wcqcf"] Oct 29 22:56:37.654226 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:37.654193 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/91fd5f87-ec89-46b8-8d57-e07cfec8a5b7-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wcqcf\" (UID: \"91fd5f87-ec89-46b8-8d57-e07cfec8a5b7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wcqcf" Oct 29 22:56:37.731684 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:37.731636 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-qpb7q" Oct 29 22:56:37.885943 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:37.885902 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/91fd5f87-ec89-46b8-8d57-e07cfec8a5b7-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wcqcf\" (UID: \"91fd5f87-ec89-46b8-8d57-e07cfec8a5b7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wcqcf" Oct 29 22:56:38.004123 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:38.004051 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/91fd5f87-ec89-46b8-8d57-e07cfec8a5b7-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wcqcf\" (UID: \"91fd5f87-ec89-46b8-8d57-e07cfec8a5b7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wcqcf" Oct 29 22:56:38.061413 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:38.061366 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zf5s\" (UniqueName: \"kubernetes.io/projected/91fd5f87-ec89-46b8-8d57-e07cfec8a5b7-kube-api-access-7zf5s\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wcqcf\" (UID: \"91fd5f87-ec89-46b8-8d57-e07cfec8a5b7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wcqcf" Oct 29 22:56:38.062140 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:38.062085 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/91fd5f87-ec89-46b8-8d57-e07cfec8a5b7-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wcqcf\" (UID: \"91fd5f87-ec89-46b8-8d57-e07cfec8a5b7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wcqcf" Oct 29 22:56:38.062792 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:38.062752 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/91fd5f87-ec89-46b8-8d57-e07cfec8a5b7-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wcqcf\" (UID: \"91fd5f87-ec89-46b8-8d57-e07cfec8a5b7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wcqcf" Oct 29 22:56:38.109515 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:38.109407 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91fd5f87-ec89-46b8-8d57-e07cfec8a5b7-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wcqcf\" (UID: \"91fd5f87-ec89-46b8-8d57-e07cfec8a5b7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wcqcf" Oct 29 22:56:38.154357 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:38.135047 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-dmw4m"] Oct 29 22:56:38.374901 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:38.374812 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wcqcf" Oct 29 22:56:38.427365 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:38.405068 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-08e15d/distinct-w0-08e15d-kserve-85695c78bb-p67gk" podUID="e664f012-3b80-4a8a-ac19-d767d92c0a33" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:56:38.686677 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:38.686527 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/distinct-2-w2-575d59-kserve-d4565b76c-hdgbf" podUID="06bb975e-d825-426f-b030-17bd1f99bb65" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:56:38.735545 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:38.734462 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-dmw4m"] Oct 29 22:56:38.743437 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:38.735725 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-dmw4m" Oct 29 22:56:38.872404 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:38.872366 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e49752e2-a116-4dd9-b6c6-712153a196f9-istiod-ca-cert\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-dmw4m\" (UID: \"e49752e2-a116-4dd9-b6c6-712153a196f9\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-dmw4m" Oct 29 22:56:38.872740 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:38.872712 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e49752e2-a116-4dd9-b6c6-712153a196f9-credential-socket\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-dmw4m\" (UID: \"e49752e2-a116-4dd9-b6c6-712153a196f9\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-dmw4m" Oct 29 22:56:38.872950 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:38.872930 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e49752e2-a116-4dd9-b6c6-712153a196f9-workload-socket\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-dmw4m\" (UID: \"e49752e2-a116-4dd9-b6c6-712153a196f9\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-dmw4m" Oct 29 22:56:38.881359 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:38.873144 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e49752e2-a116-4dd9-b6c6-712153a196f9-istio-envoy\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-dmw4m\" (UID: \"e49752e2-a116-4dd9-b6c6-712153a196f9\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-dmw4m" Oct 29 22:56:38.995355 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:38.986563 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e49752e2-a116-4dd9-b6c6-712153a196f9-istio-podinfo\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-dmw4m\" (UID: \"e49752e2-a116-4dd9-b6c6-712153a196f9\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-dmw4m" Oct 29 22:56:38.995355 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:38.990531 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e49752e2-a116-4dd9-b6c6-712153a196f9-workload-certs\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-dmw4m\" (UID: \"e49752e2-a116-4dd9-b6c6-712153a196f9\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-dmw4m" Oct 29 22:56:39.019426 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:38.994543 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e49752e2-a116-4dd9-b6c6-712153a196f9-istio-data\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-dmw4m\" (UID: \"e49752e2-a116-4dd9-b6c6-712153a196f9\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-dmw4m" Oct 29 22:56:39.020302 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:39.019760 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e49752e2-a116-4dd9-b6c6-712153a196f9-istio-token\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-dmw4m\" (UID: \"e49752e2-a116-4dd9-b6c6-712153a196f9\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-dmw4m" Oct 29 22:56:39.020568 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:39.020546 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4kt26\" (UniqueName: \"kubernetes.io/projected/e49752e2-a116-4dd9-b6c6-712153a196f9-kube-api-access-4kt26\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-dmw4m\" (UID: \"e49752e2-a116-4dd9-b6c6-712153a196f9\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-dmw4m" Oct 29 22:56:39.138359 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:39.126417 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e49752e2-a116-4dd9-b6c6-712153a196f9-istiod-ca-cert\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-dmw4m\" (UID: \"e49752e2-a116-4dd9-b6c6-712153a196f9\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-dmw4m" Oct 29 22:56:39.142664 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:39.142635 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e49752e2-a116-4dd9-b6c6-712153a196f9-credential-socket\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-dmw4m\" (UID: \"e49752e2-a116-4dd9-b6c6-712153a196f9\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-dmw4m" Oct 29 22:56:39.157318 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:39.146462 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e49752e2-a116-4dd9-b6c6-712153a196f9-workload-socket\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-dmw4m\" (UID: \"e49752e2-a116-4dd9-b6c6-712153a196f9\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-dmw4m" Oct 29 22:56:39.157318 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:39.146567 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e49752e2-a116-4dd9-b6c6-712153a196f9-istio-envoy\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-dmw4m\" (UID: \"e49752e2-a116-4dd9-b6c6-712153a196f9\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-dmw4m" Oct 29 22:56:39.157318 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:39.146656 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e49752e2-a116-4dd9-b6c6-712153a196f9-istio-podinfo\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-dmw4m\" (UID: \"e49752e2-a116-4dd9-b6c6-712153a196f9\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-dmw4m" Oct 29 22:56:39.157318 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:39.146705 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e49752e2-a116-4dd9-b6c6-712153a196f9-workload-certs\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-dmw4m\" (UID: \"e49752e2-a116-4dd9-b6c6-712153a196f9\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-dmw4m" Oct 29 22:56:39.157318 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:39.146783 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e49752e2-a116-4dd9-b6c6-712153a196f9-istio-data\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-dmw4m\" (UID: \"e49752e2-a116-4dd9-b6c6-712153a196f9\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-dmw4m" Oct 29 22:56:39.157318 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:39.146836 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e49752e2-a116-4dd9-b6c6-712153a196f9-istio-token\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-dmw4m\" (UID: \"e49752e2-a116-4dd9-b6c6-712153a196f9\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-dmw4m" Oct 29 22:56:39.157318 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:39.146882 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4kt26\" (UniqueName: \"kubernetes.io/projected/e49752e2-a116-4dd9-b6c6-712153a196f9-kube-api-access-4kt26\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-dmw4m\" (UID: \"e49752e2-a116-4dd9-b6c6-712153a196f9\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-dmw4m" Oct 29 22:56:39.189145 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:39.137658 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e49752e2-a116-4dd9-b6c6-712153a196f9-istiod-ca-cert\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-dmw4m\" (UID: \"e49752e2-a116-4dd9-b6c6-712153a196f9\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-dmw4m" Oct 29 22:56:39.189145 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:39.182114 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e49752e2-a116-4dd9-b6c6-712153a196f9-credential-socket\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-dmw4m\" (UID: \"e49752e2-a116-4dd9-b6c6-712153a196f9\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-dmw4m" Oct 29 22:56:39.189145 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:39.137707 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-31226c/distinct-w6-31226c-kserve-858d95fd76-bkf7c" podUID="f3fc11c6-03c9-4314-99c3-f857b149415c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded" Oct 29 22:56:39.189145 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:39.182740 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e49752e2-a116-4dd9-b6c6-712153a196f9-workload-socket\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-dmw4m\" (UID: \"e49752e2-a116-4dd9-b6c6-712153a196f9\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-dmw4m" Oct 29 22:56:39.189145 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:39.137750 5057 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-31226c/distinct-w6-31226c-kserve-858d95fd76-bkf7c" podUID="f3fc11c6-03c9-4314-99c3-f857b149415c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/health\": context deadline exceeded" Oct 29 22:56:39.238901 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:39.238867 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e49752e2-a116-4dd9-b6c6-712153a196f9-workload-certs\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-dmw4m\" (UID: \"e49752e2-a116-4dd9-b6c6-712153a196f9\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-dmw4m" Oct 29 22:56:39.239381 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:39.239358 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e49752e2-a116-4dd9-b6c6-712153a196f9-istio-data\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-dmw4m\" (UID: \"e49752e2-a116-4dd9-b6c6-712153a196f9\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-dmw4m" Oct 29 22:56:39.765567 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:39.748193 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e49752e2-a116-4dd9-b6c6-712153a196f9-istio-token\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-dmw4m\" (UID: \"e49752e2-a116-4dd9-b6c6-712153a196f9\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-dmw4m" Oct 29 22:56:41.348356 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:39.834807 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/distinct-2-w2-575d59-kserve-d4565b76c-hdgbf" podUID="06bb975e-d825-426f-b030-17bd1f99bb65" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": context deadline exceeded" Oct 29 22:56:41.460899 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:40.014397 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-08e15d/distinct-w0-08e15d-kserve-85695c78bb-p67gk" podUID="e664f012-3b80-4a8a-ac19-d767d92c0a33" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": context deadline exceeded" Oct 29 22:56:41.466580 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:40.314190 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-31226c/distinct-w6-31226c-kserve-858d95fd76-bkf7c" podUID="f3fc11c6-03c9-4314-99c3-f857b149415c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded" Oct 29 22:56:41.471441 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:40.344699 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17394a/distinct-2-w3-17394a-kserve-6545c766c4-h4chf" podUID="f29e8b65-b7dd-45d4-86b0-f13a29ef8e66" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:56:41.475422 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:40.912951 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e49752e2-a116-4dd9-b6c6-712153a196f9-istio-envoy\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-dmw4m\" (UID: \"e49752e2-a116-4dd9-b6c6-712153a196f9\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-dmw4m" Oct 29 22:56:41.475618 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:40.913785 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e49752e2-a116-4dd9-b6c6-712153a196f9-istio-podinfo\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-dmw4m\" (UID: \"e49752e2-a116-4dd9-b6c6-712153a196f9\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-dmw4m" Oct 29 22:56:41.475766 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:41.152113 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4kt26\" (UniqueName: \"kubernetes.io/projected/e49752e2-a116-4dd9-b6c6-712153a196f9-kube-api-access-4kt26\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-dmw4m\" (UID: \"e49752e2-a116-4dd9-b6c6-712153a196f9\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-dmw4m" Oct 29 22:56:41.624855 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:41.398741 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-dbdjq" podUID="3ee5747c-e4e9-48f6-ad11-ab3673116724" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded" Oct 29 22:56:41.625347 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:41.398780 5057 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-vp6zb" podUID="3254fb15-5180-4c0e-afdb-69062103f3e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/health\": context deadline exceeded" Oct 29 22:56:41.677658 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:41.677617 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w6-31226c-84fb67d984-mpgbh" podUID="3481d9c3-f557-4b9d-b0f2-b0c0bdb81bd9" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.68:50052\" within 1s: context deadline exceeded" Oct 29 22:56:41.747118 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:41.747082 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-dmw4m" Oct 29 22:56:41.758995 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:41.752905 5057 patch_prober.go:29] interesting pod/monitoring-plugin-77f4f76594-mjnxx container/monitoring-plugin namespace/openshift-monitoring: Readiness probe status=failure output="Get \"https://10.132.0.38:9443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 22:56:41.762488 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:41.762454 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-monitoring/monitoring-plugin-77f4f76594-mjnxx" podUID="957c3493-0edc-4c9a-a85b-9ed7b23cb7ba" containerName="monitoring-plugin" probeResult="failure" output="Get \"https://10.132.0.38:9443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:56:42.157356 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:42.131514 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-575d59/distinct-2-w2-575d59-kserve-d4565b76c-hdgbf" Oct 29 22:56:42.201106 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:56:42.185537 5057 logging.go:55] [core] [Channel #770 SubChannel #774]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.68:50052", ServerName: "10.132.0.68:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: failed to write client preface: write tcp 10.132.0.2:43858->10.132.0.68:50052: use of closed network connection" Oct 29 22:56:42.265092 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:56:42.265058 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91fd5f87_ec89_46b8_8d57_e07cfec8a5b7.slice/crio-ef6891e6d359d56e36b70a629a9ac243ca3835693505e8a2bf08bf62615e89fb WatchSource:0}: Error finding container ef6891e6d359d56e36b70a629a9ac243ca3835693505e8a2bf08bf62615e89fb: Status 404 returned error can't find the container with id ef6891e6d359d56e36b70a629a9ac243ca3835693505e8a2bf08bf62615e89fb Oct 29 22:56:42.496396 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:42.470616 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-31226c/distinct-w6-31226c-kserve-858d95fd76-bkf7c" podUID="f3fc11c6-03c9-4314-99c3-f857b149415c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:56:42.496396 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:42.472320 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17394a/distinct-2-w3-17394a-kserve-6545c766c4-h4chf" podUID="f29e8b65-b7dd-45d4-86b0-f13a29ef8e66" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:56:42.533457 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:42.533417 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-08e15d/distinct-w0-08e15d-kserve-85695c78bb-p67gk" podUID="e664f012-3b80-4a8a-ac19-d767d92c0a33" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": context deadline exceeded" Oct 29 22:56:42.548041 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:42.536035 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:56:42.664635 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:42.545243 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/distinct-2-w2-575d59-kserve-d4565b76c-hdgbf" podUID="06bb975e-d825-426f-b030-17bd1f99bb65" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:56:42.666042 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:42.664688 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-qpb7q"] Oct 29 22:56:42.666421 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:42.666393 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-08e15d/distinct-w0-08e15d-kserve-85695c78bb-p67gk" Oct 29 22:56:42.666627 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:42.666605 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-31226c/distinct-w6-31226c-kserve-858d95fd76-bkf7c" Oct 29 22:56:42.666777 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:42.666765 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-17394a/distinct-2-w3-17394a-kserve-6545c766c4-h4chf" Oct 29 22:56:42.666905 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:42.666891 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wcqcf"] Oct 29 22:56:42.667087 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:42.667075 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-08e15d/distinct-w0-08e15d-kserve-85695c78bb-p67gk" Oct 29 22:56:42.668113 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:42.665108 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:56:42.668355 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:42.668312 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:56:42.750391 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:42.747141 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-17394a/distinct-2-w3-17394a-kserve-6545c766c4-h4chf" Oct 29 22:56:42.846869 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:42.791484 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-31226c/distinct-w6-31226c-kserve-858d95fd76-bkf7c" Oct 29 22:56:42.961838 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:42.961530 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-575d59/distinct-2-w2-575d59-kserve-d4565b76c-hdgbf" Oct 29 22:56:43.144924 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:43.144882 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wcqcf" event={"ID":"91fd5f87-ec89-46b8-8d57-e07cfec8a5b7","Type":"ContainerStarted","Data":"ef6891e6d359d56e36b70a629a9ac243ca3835693505e8a2bf08bf62615e89fb"} Oct 29 22:56:43.153422 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:43.153236 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-qpb7q" event={"ID":"708063fe-d579-4b1b-982d-63faeb84dac7","Type":"ContainerStarted","Data":"e2011d07395a149958f8faab372c0e8f52f2b933c37191c1824191e413b48bc8"} Oct 29 22:56:43.354565 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:43.354519 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:56:43.354664 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:43.354613 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:56:43.354703 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:43.354678 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:56:43.355855 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:43.355817 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-dmw4m"] Oct 29 22:56:44.177387 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:44.166497 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-qpb7q" event={"ID":"708063fe-d579-4b1b-982d-63faeb84dac7","Type":"ContainerStarted","Data":"439ca738d78e30ca61c85ad476b4595de3d3bb044711cc2b9f212752831d4580"} Oct 29 22:56:44.177387 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:44.174750 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-dmw4m" event={"ID":"e49752e2-a116-4dd9-b6c6-712153a196f9","Type":"ContainerStarted","Data":"64e48b5dcfb6775775fef1fe42be0e24654acb0887fa1144196a619c0ff1c256"} Oct 29 22:56:44.177387 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:44.174830 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-dmw4m" event={"ID":"e49752e2-a116-4dd9-b6c6-712153a196f9","Type":"ContainerStarted","Data":"4cbbdbf442485c76719fc3470187146a5203c9b9c3825384ba05b344f9d595ab"} Oct 29 22:56:44.247987 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:44.244804 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-qpb7q" podStartSLOduration=8.244784512 podStartE2EDuration="8.244784512s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:56:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:56:44.236158085 +0000 UTC m=+1385.019371537" watchObservedRunningTime="2026-10-29 22:56:44.244784512 +0000 UTC m=+1385.027997938" Oct 29 22:56:44.360455 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:44.341825 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-dmw4m" podStartSLOduration=6.341784938 podStartE2EDuration="6.341784938s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:56:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:56:44.28386736 +0000 UTC m=+1385.067080786" watchObservedRunningTime="2026-10-29 22:56:44.341784938 +0000 UTC m=+1385.124998370" Oct 29 22:56:44.746352 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:44.745985 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-qpb7q" Oct 29 22:56:44.746352 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:44.746097 5057 patch_prober.go:29] interesting pod/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-qpb7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body= Oct 29 22:56:44.746352 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:44.746140 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-qpb7q" podUID="708063fe-d579-4b1b-982d-63faeb84dac7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" Oct 29 22:56:44.753396 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:44.750654 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-dmw4m" Oct 29 22:56:44.753552 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:44.753444 5057 patch_prober.go:29] interesting pod/e2e-worker-w0-08e15d-openshift-default-d579fdc75-dmw4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body= Oct 29 22:56:44.753552 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:44.753500 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-dmw4m" podUID="e49752e2-a116-4dd9-b6c6-712153a196f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" Oct 29 22:56:45.600781 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:45.600711 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:56:45.746165 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:45.745696 5057 patch_prober.go:29] interesting pod/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-qpb7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body= Oct 29 22:56:45.746165 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:45.745771 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-qpb7q" podUID="708063fe-d579-4b1b-982d-63faeb84dac7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" Oct 29 22:56:45.760380 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:45.758405 5057 patch_prober.go:29] interesting pod/e2e-worker-w0-08e15d-openshift-default-d579fdc75-dmw4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body= Oct 29 22:56:45.760380 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:45.758479 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-dmw4m" podUID="e49752e2-a116-4dd9-b6c6-712153a196f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" Oct 29 22:56:46.746354 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:46.743926 5057 patch_prober.go:29] interesting pod/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-qpb7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body= Oct 29 22:56:46.746354 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:46.743996 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-qpb7q" podUID="708063fe-d579-4b1b-982d-63faeb84dac7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" Oct 29 22:56:46.749957 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:46.749931 5057 patch_prober.go:29] interesting pod/e2e-worker-w0-08e15d-openshift-default-d579fdc75-dmw4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body= Oct 29 22:56:46.750042 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:46.749986 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-dmw4m" podUID="e49752e2-a116-4dd9-b6c6-712153a196f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" Oct 29 22:56:47.754181 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:47.754132 5057 patch_prober.go:29] interesting pod/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-qpb7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body= Oct 29 22:56:47.754705 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:47.754200 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-qpb7q" podUID="708063fe-d579-4b1b-982d-63faeb84dac7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" Oct 29 22:56:47.754941 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:47.754906 5057 patch_prober.go:29] interesting pod/e2e-worker-w0-08e15d-openshift-default-d579fdc75-dmw4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body= Oct 29 22:56:47.755055 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:47.754955 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-dmw4m" podUID="e49752e2-a116-4dd9-b6c6-712153a196f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" Oct 29 22:56:48.530355 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:48.529748 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:56:48.756948 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:48.756907 5057 patch_prober.go:29] interesting pod/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-qpb7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body= Oct 29 22:56:48.757443 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:48.756982 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-qpb7q" podUID="708063fe-d579-4b1b-982d-63faeb84dac7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" Oct 29 22:56:48.757443 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:48.757050 5057 patch_prober.go:29] interesting pod/e2e-worker-w0-08e15d-openshift-default-d579fdc75-dmw4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body= Oct 29 22:56:48.757443 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:48.757106 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-dmw4m" podUID="e49752e2-a116-4dd9-b6c6-712153a196f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" Oct 29 22:56:49.145066 ip-10-0-140-74 kubenswrapper[5057]: E1029 22:56:49.145023 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d2dnn" containerName="extractor" Oct 29 22:56:49.744651 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:49.744620 5057 patch_prober.go:29] interesting pod/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-qpb7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body= Oct 29 22:56:49.744980 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:49.744951 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-qpb7q" podUID="708063fe-d579-4b1b-982d-63faeb84dac7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" Oct 29 22:56:49.749777 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:49.749757 5057 patch_prober.go:29] interesting pod/e2e-worker-w0-08e15d-openshift-default-d579fdc75-dmw4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body= Oct 29 22:56:49.749977 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:49.749956 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-dmw4m" podUID="e49752e2-a116-4dd9-b6c6-712153a196f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" Oct 29 22:56:50.205266 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:50.205055 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:56:50.205266 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:50.205142 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:56:50.746890 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:50.746853 5057 patch_prober.go:29] interesting pod/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-qpb7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body= Oct 29 22:56:50.749508 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:50.749473 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-qpb7q" podUID="708063fe-d579-4b1b-982d-63faeb84dac7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" Oct 29 22:56:50.750428 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:50.750404 5057 patch_prober.go:29] interesting pod/e2e-worker-w0-08e15d-openshift-default-d579fdc75-dmw4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body= Oct 29 22:56:50.750594 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:50.750572 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-dmw4m" podUID="e49752e2-a116-4dd9-b6c6-712153a196f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" Oct 29 22:56:50.899169 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:50.895323 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:56:51.463907 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:51.463860 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wcqcf" event={"ID":"91fd5f87-ec89-46b8-8d57-e07cfec8a5b7","Type":"ContainerStarted","Data":"88623bf82f730409d092c1c267bcccb1362d460ad50721336562d0e9d6152b8a"} Oct 29 22:56:51.464496 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:51.464470 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wcqcf" Oct 29 22:56:51.506439 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:51.506407 5057 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-wcqcf container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 22:56:51.506679 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:51.506651 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wcqcf" podUID="91fd5f87-ec89-46b8-8d57-e07cfec8a5b7" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 22:56:51.562379 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:51.562263 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wcqcf" podStartSLOduration=7.893580283 podStartE2EDuration="15.562243536s" totalImagesPullingTime="7.668663253s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:56:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:56:51.534020756 +0000 UTC m=+1392.317234211" watchObservedRunningTime="2026-10-29 22:56:51.562243536 +0000 UTC m=+1392.345456962" Oct 29 22:56:51.749288 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:51.749214 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-qpb7q" Oct 29 22:56:51.750915 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:51.750893 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-qpb7q" Oct 29 22:56:51.751806 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:51.751779 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-qpb7q" Oct 29 22:56:51.757047 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:51.757023 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-dmw4m" Oct 29 22:56:51.757505 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:51.757481 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-dmw4m" Oct 29 22:56:51.759252 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:51.759235 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-dmw4m" Oct 29 22:56:52.455599 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:52.455566 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wcqcf" Oct 29 22:56:53.773418 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:53.773377 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:56:53.941127 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:53.941090 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:56:54.115586 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:54.115500 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:56:54.830419 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:54.830382 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:56:55.351959 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:55.351925 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:56:55.574570 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:55.574539 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:56:56.884920 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:56.884883 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-h7w59"] Oct 29 22:56:56.891593 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:56.891569 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-h7w59" Oct 29 22:56:56.925153 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:56.925081 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-h7w59"] Oct 29 22:56:56.986576 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:56.986541 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-e4a049-openshift-default-dockercfg-msz4p\"" Oct 29 22:56:57.018122 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:57.018090 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c92c699-743e-4118-9887-d6ecc1568b88-istio-podinfo\") pod \"e2e-api-a-e4a049-openshift-default-55bd8fc7f9-h7w59\" (UID: \"4c92c699-743e-4118-9887-d6ecc1568b88\") " pod="openshift-ingress/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-h7w59" Oct 29 22:56:57.018591 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:57.018572 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c92c699-743e-4118-9887-d6ecc1568b88-workload-socket\") pod \"e2e-api-a-e4a049-openshift-default-55bd8fc7f9-h7w59\" (UID: \"4c92c699-743e-4118-9887-d6ecc1568b88\") " pod="openshift-ingress/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-h7w59" Oct 29 22:56:57.018641 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:57.018602 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c92c699-743e-4118-9887-d6ecc1568b88-istio-envoy\") pod \"e2e-api-a-e4a049-openshift-default-55bd8fc7f9-h7w59\" (UID: \"4c92c699-743e-4118-9887-d6ecc1568b88\") " pod="openshift-ingress/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-h7w59" Oct 29 22:56:57.018929 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:57.018907 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c92c699-743e-4118-9887-d6ecc1568b88-workload-certs\") pod \"e2e-api-a-e4a049-openshift-default-55bd8fc7f9-h7w59\" (UID: \"4c92c699-743e-4118-9887-d6ecc1568b88\") " pod="openshift-ingress/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-h7w59" Oct 29 22:56:57.018966 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:57.018940 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c92c699-743e-4118-9887-d6ecc1568b88-istiod-ca-cert\") pod \"e2e-api-a-e4a049-openshift-default-55bd8fc7f9-h7w59\" (UID: \"4c92c699-743e-4118-9887-d6ecc1568b88\") " pod="openshift-ingress/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-h7w59" Oct 29 22:56:57.019077 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:57.019064 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wxtn\" (UniqueName: \"kubernetes.io/projected/4c92c699-743e-4118-9887-d6ecc1568b88-kube-api-access-4wxtn\") pod \"e2e-api-a-e4a049-openshift-default-55bd8fc7f9-h7w59\" (UID: \"4c92c699-743e-4118-9887-d6ecc1568b88\") " pod="openshift-ingress/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-h7w59" Oct 29 22:56:57.019138 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:57.019126 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c92c699-743e-4118-9887-d6ecc1568b88-istio-token\") pod \"e2e-api-a-e4a049-openshift-default-55bd8fc7f9-h7w59\" (UID: \"4c92c699-743e-4118-9887-d6ecc1568b88\") " pod="openshift-ingress/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-h7w59" Oct 29 22:56:57.019174 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:57.019151 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c92c699-743e-4118-9887-d6ecc1568b88-istio-data\") pod \"e2e-api-a-e4a049-openshift-default-55bd8fc7f9-h7w59\" (UID: \"4c92c699-743e-4118-9887-d6ecc1568b88\") " pod="openshift-ingress/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-h7w59" Oct 29 22:56:57.019174 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:57.019170 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c92c699-743e-4118-9887-d6ecc1568b88-credential-socket\") pod \"e2e-api-a-e4a049-openshift-default-55bd8fc7f9-h7w59\" (UID: \"4c92c699-743e-4118-9887-d6ecc1568b88\") " pod="openshift-ingress/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-h7w59" Oct 29 22:56:57.120968 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:57.120933 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c92c699-743e-4118-9887-d6ecc1568b88-istio-token\") pod \"e2e-api-a-e4a049-openshift-default-55bd8fc7f9-h7w59\" (UID: \"4c92c699-743e-4118-9887-d6ecc1568b88\") " pod="openshift-ingress/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-h7w59" Oct 29 22:56:57.121151 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:57.120981 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c92c699-743e-4118-9887-d6ecc1568b88-istio-data\") pod \"e2e-api-a-e4a049-openshift-default-55bd8fc7f9-h7w59\" (UID: \"4c92c699-743e-4118-9887-d6ecc1568b88\") " pod="openshift-ingress/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-h7w59" Oct 29 22:56:57.121151 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:57.121015 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c92c699-743e-4118-9887-d6ecc1568b88-credential-socket\") pod \"e2e-api-a-e4a049-openshift-default-55bd8fc7f9-h7w59\" (UID: \"4c92c699-743e-4118-9887-d6ecc1568b88\") " pod="openshift-ingress/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-h7w59" Oct 29 22:56:57.121151 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:57.121076 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c92c699-743e-4118-9887-d6ecc1568b88-istio-podinfo\") pod \"e2e-api-a-e4a049-openshift-default-55bd8fc7f9-h7w59\" (UID: \"4c92c699-743e-4118-9887-d6ecc1568b88\") " pod="openshift-ingress/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-h7w59" Oct 29 22:56:57.121151 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:57.121147 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c92c699-743e-4118-9887-d6ecc1568b88-workload-socket\") pod \"e2e-api-a-e4a049-openshift-default-55bd8fc7f9-h7w59\" (UID: \"4c92c699-743e-4118-9887-d6ecc1568b88\") " pod="openshift-ingress/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-h7w59" Oct 29 22:56:57.121376 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:57.121174 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c92c699-743e-4118-9887-d6ecc1568b88-istio-envoy\") pod \"e2e-api-a-e4a049-openshift-default-55bd8fc7f9-h7w59\" (UID: \"4c92c699-743e-4118-9887-d6ecc1568b88\") " pod="openshift-ingress/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-h7w59" Oct 29 22:56:57.121376 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:57.121310 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c92c699-743e-4118-9887-d6ecc1568b88-workload-certs\") pod \"e2e-api-a-e4a049-openshift-default-55bd8fc7f9-h7w59\" (UID: \"4c92c699-743e-4118-9887-d6ecc1568b88\") " pod="openshift-ingress/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-h7w59" Oct 29 22:56:57.121376 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:57.121357 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c92c699-743e-4118-9887-d6ecc1568b88-istiod-ca-cert\") pod \"e2e-api-a-e4a049-openshift-default-55bd8fc7f9-h7w59\" (UID: \"4c92c699-743e-4118-9887-d6ecc1568b88\") " pod="openshift-ingress/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-h7w59" Oct 29 22:56:57.121561 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:57.121506 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wxtn\" (UniqueName: \"kubernetes.io/projected/4c92c699-743e-4118-9887-d6ecc1568b88-kube-api-access-4wxtn\") pod \"e2e-api-a-e4a049-openshift-default-55bd8fc7f9-h7w59\" (UID: \"4c92c699-743e-4118-9887-d6ecc1568b88\") " pod="openshift-ingress/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-h7w59" Oct 29 22:56:57.121561 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:57.121548 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c92c699-743e-4118-9887-d6ecc1568b88-workload-socket\") pod \"e2e-api-a-e4a049-openshift-default-55bd8fc7f9-h7w59\" (UID: \"4c92c699-743e-4118-9887-d6ecc1568b88\") " pod="openshift-ingress/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-h7w59" Oct 29 22:56:57.121706 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:57.121548 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c92c699-743e-4118-9887-d6ecc1568b88-credential-socket\") pod \"e2e-api-a-e4a049-openshift-default-55bd8fc7f9-h7w59\" (UID: \"4c92c699-743e-4118-9887-d6ecc1568b88\") " pod="openshift-ingress/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-h7w59" Oct 29 22:56:57.121706 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:57.121658 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c92c699-743e-4118-9887-d6ecc1568b88-istio-data\") pod \"e2e-api-a-e4a049-openshift-default-55bd8fc7f9-h7w59\" (UID: \"4c92c699-743e-4118-9887-d6ecc1568b88\") " pod="openshift-ingress/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-h7w59" Oct 29 22:56:57.121785 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:57.121743 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c92c699-743e-4118-9887-d6ecc1568b88-workload-certs\") pod \"e2e-api-a-e4a049-openshift-default-55bd8fc7f9-h7w59\" (UID: \"4c92c699-743e-4118-9887-d6ecc1568b88\") " pod="openshift-ingress/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-h7w59" Oct 29 22:56:57.122177 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:57.122160 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c92c699-743e-4118-9887-d6ecc1568b88-istiod-ca-cert\") pod \"e2e-api-a-e4a049-openshift-default-55bd8fc7f9-h7w59\" (UID: \"4c92c699-743e-4118-9887-d6ecc1568b88\") " pod="openshift-ingress/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-h7w59" Oct 29 22:56:57.123558 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:57.123534 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c92c699-743e-4118-9887-d6ecc1568b88-istio-envoy\") pod \"e2e-api-a-e4a049-openshift-default-55bd8fc7f9-h7w59\" (UID: \"4c92c699-743e-4118-9887-d6ecc1568b88\") " pod="openshift-ingress/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-h7w59" Oct 29 22:56:57.123732 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:57.123714 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c92c699-743e-4118-9887-d6ecc1568b88-istio-podinfo\") pod \"e2e-api-a-e4a049-openshift-default-55bd8fc7f9-h7w59\" (UID: \"4c92c699-743e-4118-9887-d6ecc1568b88\") " pod="openshift-ingress/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-h7w59" Oct 29 22:56:57.131974 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:57.131949 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:56:57.142809 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:57.142755 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c92c699-743e-4118-9887-d6ecc1568b88-istio-token\") pod \"e2e-api-a-e4a049-openshift-default-55bd8fc7f9-h7w59\" (UID: \"4c92c699-743e-4118-9887-d6ecc1568b88\") " pod="openshift-ingress/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-h7w59" Oct 29 22:56:57.142902 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:57.142823 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wxtn\" (UniqueName: \"kubernetes.io/projected/4c92c699-743e-4118-9887-d6ecc1568b88-kube-api-access-4wxtn\") pod \"e2e-api-a-e4a049-openshift-default-55bd8fc7f9-h7w59\" (UID: \"4c92c699-743e-4118-9887-d6ecc1568b88\") " pod="openshift-ingress/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-h7w59" Oct 29 22:56:57.214929 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:57.214898 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-h7w59" Oct 29 22:56:57.336928 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:57.336889 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:56:57.553102 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:57.553066 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-h7w59"] Oct 29 22:56:57.556134 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:57.556080 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:56:57.556425 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:57.556319 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:56:57.556578 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:57.556551 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:56:58.123231 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:58.123183 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:56:58.258449 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:58.258237 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:56:58.543108 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:58.543072 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-h7w59" event={"ID":"4c92c699-743e-4118-9887-d6ecc1568b88","Type":"ContainerStarted","Data":"aab1b00be86f3010ff586c6b1140eecbc0b4cf03f3ef596c066765576440edda"} Oct 29 22:56:58.543348 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:58.543124 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-h7w59" event={"ID":"4c92c699-743e-4118-9887-d6ecc1568b88","Type":"ContainerStarted","Data":"3881ad7b4d1045e525acf78ed8ce1712bac6a967496ff96a9ba43a25446bf954"} Oct 29 22:56:58.614890 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:58.614820 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-h7w59" podStartSLOduration=2.614797125 podStartE2EDuration="2.614797125s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:56:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:56:58.598858817 +0000 UTC m=+1399.382072245" watchObservedRunningTime="2026-10-29 22:56:58.614797125 +0000 UTC m=+1399.398010556" Oct 29 22:56:59.216403 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:59.216365 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-h7w59" Oct 29 22:56:59.217584 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:59.217552 5057 patch_prober.go:29] interesting pod/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-h7w59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body= Oct 29 22:56:59.217721 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:59.217651 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-h7w59" podUID="4c92c699-743e-4118-9887-d6ecc1568b88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" Oct 29 22:56:59.504623 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:56:59.502952 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:57:00.215725 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:57:00.215677 5057 patch_prober.go:29] interesting pod/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-h7w59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body= Oct 29 22:57:00.215942 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:57:00.215750 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-h7w59" podUID="4c92c699-743e-4118-9887-d6ecc1568b88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" Oct 29 22:57:01.215284 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:57:01.215253 5057 patch_prober.go:29] interesting pod/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-h7w59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body= Oct 29 22:57:01.215707 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:57:01.215304 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-h7w59" podUID="4c92c699-743e-4118-9887-d6ecc1568b88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" Oct 29 22:57:01.350271 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:57:01.350213 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:57:02.215422 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:57:02.215381 5057 patch_prober.go:29] interesting pod/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-h7w59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body= Oct 29 22:57:02.215954 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:57:02.215473 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-h7w59" podUID="4c92c699-743e-4118-9887-d6ecc1568b88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" Oct 29 22:57:03.215496 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:57:03.215460 5057 patch_prober.go:29] interesting pod/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-h7w59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body= Oct 29 22:57:03.215979 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:57:03.215946 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-h7w59" podUID="4c92c699-743e-4118-9887-d6ecc1568b88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" Oct 29 22:57:03.310096 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:57:03.310037 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:57:04.221001 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:57:04.220967 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-h7w59" Oct 29 22:57:04.221476 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:57:04.221442 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-h7w59" Oct 29 22:57:04.222123 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:57:04.222100 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-h7w59" Oct 29 22:57:04.630355 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:57:04.630259 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:57:04.754695 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:57:04.754658 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:57:04.826522 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:57:04.826487 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-g26dp"] Oct 29 22:57:04.833659 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:57:04.833633 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-g26dp" Oct 29 22:57:04.854104 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:57:04.854069 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-g26dp"] Oct 29 22:57:04.956218 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:57:04.956189 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5a6ed02-7acb-46de-9107-81a50942fec6-istio-envoy\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-g26dp\" (UID: \"c5a6ed02-7acb-46de-9107-81a50942fec6\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-g26dp" Oct 29 22:57:04.956396 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:57:04.956241 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5a6ed02-7acb-46de-9107-81a50942fec6-istio-podinfo\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-g26dp\" (UID: \"c5a6ed02-7acb-46de-9107-81a50942fec6\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-g26dp" Oct 29 22:57:04.956396 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:57:04.956365 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5a6ed02-7acb-46de-9107-81a50942fec6-workload-socket\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-g26dp\" (UID: \"c5a6ed02-7acb-46de-9107-81a50942fec6\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-g26dp" Oct 29 22:57:04.956496 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:57:04.956416 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5a6ed02-7acb-46de-9107-81a50942fec6-workload-certs\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-g26dp\" (UID: \"c5a6ed02-7acb-46de-9107-81a50942fec6\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-g26dp" Oct 29 22:57:04.956531 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:57:04.956498 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5a6ed02-7acb-46de-9107-81a50942fec6-istio-data\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-g26dp\" (UID: \"c5a6ed02-7acb-46de-9107-81a50942fec6\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-g26dp" Oct 29 22:57:04.956571 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:57:04.956528 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gvr6\" (UniqueName: \"kubernetes.io/projected/c5a6ed02-7acb-46de-9107-81a50942fec6-kube-api-access-6gvr6\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-g26dp\" (UID: \"c5a6ed02-7acb-46de-9107-81a50942fec6\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-g26dp" Oct 29 22:57:04.956678 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:57:04.956660 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5a6ed02-7acb-46de-9107-81a50942fec6-credential-socket\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-g26dp\" (UID: \"c5a6ed02-7acb-46de-9107-81a50942fec6\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-g26dp" Oct 29 22:57:04.956822 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:57:04.956807 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5a6ed02-7acb-46de-9107-81a50942fec6-istio-token\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-g26dp\" (UID: \"c5a6ed02-7acb-46de-9107-81a50942fec6\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-g26dp" Oct 29 22:57:04.956854 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:57:04.956834 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5a6ed02-7acb-46de-9107-81a50942fec6-istiod-ca-cert\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-g26dp\" (UID: \"c5a6ed02-7acb-46de-9107-81a50942fec6\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-g26dp" Oct 29 22:57:05.058714 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:57:05.058680 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5a6ed02-7acb-46de-9107-81a50942fec6-istio-token\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-g26dp\" (UID: \"c5a6ed02-7acb-46de-9107-81a50942fec6\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-g26dp" Oct 29 22:57:05.058714 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:57:05.058715 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5a6ed02-7acb-46de-9107-81a50942fec6-istiod-ca-cert\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-g26dp\" (UID: \"c5a6ed02-7acb-46de-9107-81a50942fec6\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-g26dp" Oct 29 22:57:05.058897 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:57:05.058793 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5a6ed02-7acb-46de-9107-81a50942fec6-istio-envoy\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-g26dp\" (UID: \"c5a6ed02-7acb-46de-9107-81a50942fec6\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-g26dp" Oct 29 22:57:05.058897 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:57:05.058819 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5a6ed02-7acb-46de-9107-81a50942fec6-istio-podinfo\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-g26dp\" (UID: \"c5a6ed02-7acb-46de-9107-81a50942fec6\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-g26dp" Oct 29 22:57:05.058897 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:57:05.058873 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5a6ed02-7acb-46de-9107-81a50942fec6-workload-socket\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-g26dp\" (UID: \"c5a6ed02-7acb-46de-9107-81a50942fec6\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-g26dp" Oct 29 22:57:05.059041 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:57:05.058896 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5a6ed02-7acb-46de-9107-81a50942fec6-workload-certs\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-g26dp\" (UID: \"c5a6ed02-7acb-46de-9107-81a50942fec6\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-g26dp" Oct 29 22:57:05.059041 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:57:05.058961 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5a6ed02-7acb-46de-9107-81a50942fec6-istio-data\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-g26dp\" (UID: \"c5a6ed02-7acb-46de-9107-81a50942fec6\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-g26dp" Oct 29 22:57:05.059041 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:57:05.058996 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gvr6\" (UniqueName: \"kubernetes.io/projected/c5a6ed02-7acb-46de-9107-81a50942fec6-kube-api-access-6gvr6\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-g26dp\" (UID: \"c5a6ed02-7acb-46de-9107-81a50942fec6\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-g26dp" Oct 29 22:57:05.059187 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:57:05.059102 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5a6ed02-7acb-46de-9107-81a50942fec6-credential-socket\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-g26dp\" (UID: \"c5a6ed02-7acb-46de-9107-81a50942fec6\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-g26dp" Oct 29 22:57:05.059355 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:57:05.059316 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5a6ed02-7acb-46de-9107-81a50942fec6-workload-socket\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-g26dp\" (UID: \"c5a6ed02-7acb-46de-9107-81a50942fec6\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-g26dp" Oct 29 22:57:05.059631 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:57:05.059600 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5a6ed02-7acb-46de-9107-81a50942fec6-credential-socket\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-g26dp\" (UID: \"c5a6ed02-7acb-46de-9107-81a50942fec6\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-g26dp" Oct 29 22:57:05.059743 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:57:05.059641 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5a6ed02-7acb-46de-9107-81a50942fec6-istio-data\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-g26dp\" (UID: \"c5a6ed02-7acb-46de-9107-81a50942fec6\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-g26dp" Oct 29 22:57:05.059743 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:57:05.059687 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5a6ed02-7acb-46de-9107-81a50942fec6-istiod-ca-cert\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-g26dp\" (UID: \"c5a6ed02-7acb-46de-9107-81a50942fec6\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-g26dp" Oct 29 22:57:05.059743 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:57:05.059733 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5a6ed02-7acb-46de-9107-81a50942fec6-workload-certs\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-g26dp\" (UID: \"c5a6ed02-7acb-46de-9107-81a50942fec6\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-g26dp" Oct 29 22:57:05.061272 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:57:05.061249 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5a6ed02-7acb-46de-9107-81a50942fec6-istio-envoy\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-g26dp\" (UID: \"c5a6ed02-7acb-46de-9107-81a50942fec6\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-g26dp" Oct 29 22:57:05.061415 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:57:05.061396 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5a6ed02-7acb-46de-9107-81a50942fec6-istio-podinfo\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-g26dp\" (UID: \"c5a6ed02-7acb-46de-9107-81a50942fec6\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-g26dp" Oct 29 22:57:05.080539 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:57:05.080511 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5a6ed02-7acb-46de-9107-81a50942fec6-istio-token\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-g26dp\" (UID: \"c5a6ed02-7acb-46de-9107-81a50942fec6\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-g26dp" Oct 29 22:57:05.081003 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:57:05.080982 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gvr6\" (UniqueName: \"kubernetes.io/projected/c5a6ed02-7acb-46de-9107-81a50942fec6-kube-api-access-6gvr6\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-g26dp\" (UID: \"c5a6ed02-7acb-46de-9107-81a50942fec6\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-g26dp" Oct 29 22:57:05.154578 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:57:05.154548 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-g26dp" Oct 29 22:57:05.430818 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:57:05.430772 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:57:05.490640 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:57:05.490608 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5a6ed02_7acb_46de_9107_81a50942fec6.slice/crio-862ba5faa0a2630a3d653ca0286736e7fa8c9fb0eb4ca445fffdc47ee218f259 WatchSource:0}: Error finding container 862ba5faa0a2630a3d653ca0286736e7fa8c9fb0eb4ca445fffdc47ee218f259: Status 404 returned error can't find the container with id 862ba5faa0a2630a3d653ca0286736e7fa8c9fb0eb4ca445fffdc47ee218f259 Oct 29 22:57:05.491343 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:57:05.491307 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-g26dp"] Oct 29 22:57:05.494284 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:57:05.494245 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:57:05.494409 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:57:05.494322 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:57:05.494409 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:57:05.494397 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:57:05.655352 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:57:05.655296 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-g26dp" event={"ID":"c5a6ed02-7acb-46de-9107-81a50942fec6","Type":"ContainerStarted","Data":"a49f83c0b74f10a2028641bd8083619178b8b43f6810524e831c6bfc19036507"} Oct 29 22:57:05.655518 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:57:05.655470 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-g26dp" event={"ID":"c5a6ed02-7acb-46de-9107-81a50942fec6","Type":"ContainerStarted","Data":"862ba5faa0a2630a3d653ca0286736e7fa8c9fb0eb4ca445fffdc47ee218f259"} Oct 29 22:57:05.705664 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:57:05.705598 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-g26dp" podStartSLOduration=1.7055778240000001 podStartE2EDuration="1.705577824s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:57:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:57:05.699067999 +0000 UTC m=+1406.482281425" watchObservedRunningTime="2026-10-29 22:57:05.705577824 +0000 UTC m=+1406.488791255" Oct 29 22:57:06.154872 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:57:06.154841 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-g26dp" Oct 29 22:57:06.156180 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:57:06.156159 5057 patch_prober.go:29] interesting pod/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-g26dp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body= Oct 29 22:57:06.156254 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:57:06.156202 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-g26dp" podUID="c5a6ed02-7acb-46de-9107-81a50942fec6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" Oct 29 22:57:06.239952 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:57:06.239885 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:57:06.357552 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:57:06.357515 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:57:06.475558 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:57:06.475514 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:57:06.594055 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:57:06.594012 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:57:07.155302 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:57:07.155267 5057 patch_prober.go:29] interesting pod/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-g26dp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body= Oct 29 22:57:07.155488 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:57:07.155321 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-g26dp" podUID="c5a6ed02-7acb-46de-9107-81a50942fec6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" Oct 29 22:57:08.001574 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:57:08.001537 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:57:08.040554 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:57:08.040506 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-6bggl"] Oct 29 22:57:08.048120 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:57:08.048087 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-6bggl" Oct 29 22:57:08.068827 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:57:08.068794 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-6bggl"] Oct 29 22:57:08.114355 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:57:08.108432 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb648622-516b-4da4-90c9-8716c09bf068-istio-envoy\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-6bggl\" (UID: \"bb648622-516b-4da4-90c9-8716c09bf068\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-6bggl" Oct 29 22:57:08.114355 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:57:08.108508 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb648622-516b-4da4-90c9-8716c09bf068-credential-socket\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-6bggl\" (UID: \"bb648622-516b-4da4-90c9-8716c09bf068\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-6bggl" Oct 29 22:57:08.114355 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:57:08.108556 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb648622-516b-4da4-90c9-8716c09bf068-istio-token\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-6bggl\" (UID: \"bb648622-516b-4da4-90c9-8716c09bf068\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-6bggl" Oct 29 22:57:08.114355 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:57:08.108589 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb648622-516b-4da4-90c9-8716c09bf068-workload-certs\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-6bggl\" (UID: \"bb648622-516b-4da4-90c9-8716c09bf068\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-6bggl" Oct 29 22:57:08.114355 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:57:08.108635 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb648622-516b-4da4-90c9-8716c09bf068-istiod-ca-cert\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-6bggl\" (UID: \"bb648622-516b-4da4-90c9-8716c09bf068\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-6bggl" Oct 29 22:57:08.114355 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:57:08.108719 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb648622-516b-4da4-90c9-8716c09bf068-workload-socket\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-6bggl\" (UID: \"bb648622-516b-4da4-90c9-8716c09bf068\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-6bggl" Oct 29 22:57:08.114355 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:57:08.108744 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb648622-516b-4da4-90c9-8716c09bf068-istio-data\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-6bggl\" (UID: \"bb648622-516b-4da4-90c9-8716c09bf068\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-6bggl" Oct 29 22:57:08.114355 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:57:08.108910 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m75v6\" (UniqueName: \"kubernetes.io/projected/bb648622-516b-4da4-90c9-8716c09bf068-kube-api-access-m75v6\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-6bggl\" (UID: \"bb648622-516b-4da4-90c9-8716c09bf068\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-6bggl" Oct 29 22:57:08.114355 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:57:08.108968 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb648622-516b-4da4-90c9-8716c09bf068-istio-podinfo\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-6bggl\" (UID: \"bb648622-516b-4da4-90c9-8716c09bf068\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-6bggl" Oct 29 22:57:08.154874 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:57:08.154839 5057 patch_prober.go:29] interesting pod/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-g26dp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body= Oct 29 22:57:08.155134 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:57:08.155107 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-g26dp" podUID="c5a6ed02-7acb-46de-9107-81a50942fec6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" Oct 29 22:57:08.225390 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:57:08.211622 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb648622-516b-4da4-90c9-8716c09bf068-istio-envoy\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-6bggl\" (UID: \"bb648622-516b-4da4-90c9-8716c09bf068\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-6bggl" Oct 29 22:57:08.225390 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:57:08.211698 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb648622-516b-4da4-90c9-8716c09bf068-credential-socket\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-6bggl\" (UID: \"bb648622-516b-4da4-90c9-8716c09bf068\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-6bggl" Oct 29 22:57:08.225390 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:57:08.211743 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb648622-516b-4da4-90c9-8716c09bf068-istio-token\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-6bggl\" (UID: \"bb648622-516b-4da4-90c9-8716c09bf068\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-6bggl" Oct 29 22:57:08.225390 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:57:08.211780 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb648622-516b-4da4-90c9-8716c09bf068-workload-certs\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-6bggl\" (UID: \"bb648622-516b-4da4-90c9-8716c09bf068\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-6bggl" Oct 29 22:57:08.225390 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:57:08.211825 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb648622-516b-4da4-90c9-8716c09bf068-istiod-ca-cert\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-6bggl\" (UID: \"bb648622-516b-4da4-90c9-8716c09bf068\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-6bggl" Oct 29 22:57:08.225390 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:57:08.211913 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb648622-516b-4da4-90c9-8716c09bf068-workload-socket\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-6bggl\" (UID: \"bb648622-516b-4da4-90c9-8716c09bf068\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-6bggl" Oct 29 22:57:08.225390 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:57:08.211938 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb648622-516b-4da4-90c9-8716c09bf068-istio-data\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-6bggl\" (UID: \"bb648622-516b-4da4-90c9-8716c09bf068\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-6bggl" Oct 29 22:57:08.225390 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:57:08.212060 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m75v6\" (UniqueName: \"kubernetes.io/projected/bb648622-516b-4da4-90c9-8716c09bf068-kube-api-access-m75v6\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-6bggl\" (UID: \"bb648622-516b-4da4-90c9-8716c09bf068\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-6bggl" Oct 29 22:57:08.225390 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:57:08.212110 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb648622-516b-4da4-90c9-8716c09bf068-istio-podinfo\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-6bggl\" (UID: \"bb648622-516b-4da4-90c9-8716c09bf068\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-6bggl" Oct 29 22:57:08.225390 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:57:08.225201 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb648622-516b-4da4-90c9-8716c09bf068-workload-certs\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-6bggl\" (UID: \"bb648622-516b-4da4-90c9-8716c09bf068\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-6bggl" Oct 29 22:57:08.226798 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:57:08.226473 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb648622-516b-4da4-90c9-8716c09bf068-istio-data\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-6bggl\" (UID: \"bb648622-516b-4da4-90c9-8716c09bf068\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-6bggl" Oct 29 22:57:08.226798 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:57:08.226730 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb648622-516b-4da4-90c9-8716c09bf068-workload-socket\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-6bggl\" (UID: \"bb648622-516b-4da4-90c9-8716c09bf068\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-6bggl" Oct 29 22:57:08.226798 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:57:08.226734 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb648622-516b-4da4-90c9-8716c09bf068-istiod-ca-cert\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-6bggl\" (UID: \"bb648622-516b-4da4-90c9-8716c09bf068\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-6bggl" Oct 29 22:57:08.229111 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:57:08.229090 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb648622-516b-4da4-90c9-8716c09bf068-credential-socket\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-6bggl\" (UID: \"bb648622-516b-4da4-90c9-8716c09bf068\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-6bggl" Oct 29 22:57:08.229427 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:57:08.229384 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb648622-516b-4da4-90c9-8716c09bf068-istio-envoy\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-6bggl\" (UID: \"bb648622-516b-4da4-90c9-8716c09bf068\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-6bggl" Oct 29 22:57:08.233983 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:57:08.233068 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb648622-516b-4da4-90c9-8716c09bf068-istio-podinfo\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-6bggl\" (UID: \"bb648622-516b-4da4-90c9-8716c09bf068\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-6bggl" Oct 29 22:57:08.259881 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:57:08.259798 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb648622-516b-4da4-90c9-8716c09bf068-istio-token\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-6bggl\" (UID: \"bb648622-516b-4da4-90c9-8716c09bf068\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-6bggl" Oct 29 22:57:08.261305 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:57:08.261277 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m75v6\" (UniqueName: \"kubernetes.io/projected/bb648622-516b-4da4-90c9-8716c09bf068-kube-api-access-m75v6\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-6bggl\" (UID: \"bb648622-516b-4da4-90c9-8716c09bf068\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-6bggl" Oct 29 22:57:08.396368 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:57:08.380604 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-6bggl" Oct 29 22:57:08.814591 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:57:08.814500 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-6bggl"] Oct 29 22:57:08.824617 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:57:08.824579 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:57:08.824733 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:57:08.824653 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:57:08.824733 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:57:08.824689 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:57:09.155182 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:57:09.155097 5057 patch_prober.go:29] interesting pod/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-g26dp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body= Oct 29 22:57:09.155182 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:57:09.155155 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-g26dp" podUID="c5a6ed02-7acb-46de-9107-81a50942fec6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" Oct 29 22:57:09.519020 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:57:09.518985 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:57:09.638638 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:57:09.638159 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:57:09.752466 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:57:09.750572 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-6bggl" event={"ID":"bb648622-516b-4da4-90c9-8716c09bf068","Type":"ContainerStarted","Data":"528ce9db4e0b868b1b70bae2ab0f7de031eb494ae772b6dfdf93d60db8ac29b8"} Oct 29 22:57:09.752466 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:57:09.750641 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-6bggl" event={"ID":"bb648622-516b-4da4-90c9-8716c09bf068","Type":"ContainerStarted","Data":"4e9169f2e12ab4050ddcd6cc73f9be384cb5b84ee773ede792de2900275b7c0a"} Oct 29 22:57:09.914407 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:57:09.914174 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-6bggl" podStartSLOduration=1.914153142 podStartE2EDuration="1.914153142s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:57:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:57:09.79570551 +0000 UTC m=+1410.578919390" watchObservedRunningTime="2026-10-29 22:57:09.914153142 +0000 UTC m=+1410.697366571" Oct 29 22:57:10.162028 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:57:10.156640 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:57:10.162534 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:57:10.162437 5057 patch_prober.go:29] interesting pod/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-g26dp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body= Oct 29 22:57:10.162534 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:57:10.162492 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-g26dp" podUID="c5a6ed02-7acb-46de-9107-81a50942fec6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" Oct 29 22:57:10.432635 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:57:10.413799 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-6bggl" Oct 29 22:57:10.432635 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:57:10.413911 5057 patch_prober.go:29] interesting pod/e2e-worker-w5-b2921b-openshift-default-5547c88fff-6bggl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body= Oct 29 22:57:10.432635 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:57:10.413962 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-6bggl" podUID="bb648622-516b-4da4-90c9-8716c09bf068" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" Oct 29 22:57:11.155795 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:57:11.155753 5057 patch_prober.go:29] interesting pod/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-g26dp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body= Oct 29 22:57:11.155968 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:57:11.155813 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-g26dp" podUID="c5a6ed02-7acb-46de-9107-81a50942fec6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" Oct 29 22:57:11.402356 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:57:11.401871 5057 patch_prober.go:29] interesting pod/e2e-worker-w5-b2921b-openshift-default-5547c88fff-6bggl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body= Oct 29 22:57:11.402356 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:57:11.401927 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-6bggl" podUID="bb648622-516b-4da4-90c9-8716c09bf068" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" Oct 29 22:57:11.928169 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:57:11.927976 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:57:12.162361 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:57:12.160895 5057 patch_prober.go:29] interesting pod/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-g26dp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body= Oct 29 22:57:12.162361 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:57:12.160970 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-g26dp" podUID="c5a6ed02-7acb-46de-9107-81a50942fec6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" Oct 29 22:57:12.399358 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:57:12.397552 5057 patch_prober.go:29] interesting pod/e2e-worker-w5-b2921b-openshift-default-5547c88fff-6bggl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body= Oct 29 22:57:12.399358 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:57:12.397612 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-6bggl" podUID="bb648622-516b-4da4-90c9-8716c09bf068" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" Oct 29 22:57:13.155059 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:57:13.155025 5057 patch_prober.go:29] interesting pod/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-g26dp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body= Oct 29 22:57:13.155468 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:57:13.155083 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-g26dp" podUID="c5a6ed02-7acb-46de-9107-81a50942fec6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" Oct 29 22:57:13.401496 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:57:13.401460 5057 patch_prober.go:29] interesting pod/e2e-worker-w5-b2921b-openshift-default-5547c88fff-6bggl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body= Oct 29 22:57:13.401772 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:57:13.401740 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-6bggl" podUID="bb648622-516b-4da4-90c9-8716c09bf068" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" Oct 29 22:57:14.004354 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:57:13.990142 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:57:14.162882 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:57:14.162853 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-g26dp" Oct 29 22:57:14.163324 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:57:14.163303 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-g26dp" Oct 29 22:57:14.164183 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:57:14.164153 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-g26dp" Oct 29 22:57:14.422480 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:57:14.422386 5057 patch_prober.go:29] interesting pod/e2e-worker-w5-b2921b-openshift-default-5547c88fff-6bggl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body= Oct 29 22:57:14.422480 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:57:14.422453 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-6bggl" podUID="bb648622-516b-4da4-90c9-8716c09bf068" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" Oct 29 22:57:15.397549 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:57:15.397512 5057 patch_prober.go:29] interesting pod/e2e-worker-w5-b2921b-openshift-default-5547c88fff-6bggl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body= Oct 29 22:57:15.397939 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:57:15.397580 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-6bggl" podUID="bb648622-516b-4da4-90c9-8716c09bf068" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" Oct 29 22:57:15.981495 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:57:15.981456 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:57:16.697821 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:57:16.697791 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-6bggl" Oct 29 22:57:16.698531 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:57:16.698502 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-6bggl" Oct 29 22:57:16.700061 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:57:16.700036 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-6bggl" Oct 29 22:57:20.454721 ip-10-0-140-74 kubenswrapper[5057]: E1029 22:57:20.454683 5057 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.107s" Oct 29 22:57:20.589377 ip-10-0-140-74 kubenswrapper[5057]: E1029 22:57:20.561476 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d2dnn" containerName="extractor" Oct 29 22:57:20.673800 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:57:20.673751 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/distinct-2-w2-575d59-kserve-d4565b76c-hdgbf" podUID="06bb975e-d825-426f-b030-17bd1f99bb65" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:20.794742 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:57:20.788115 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17394a/distinct-2-w3-17394a-kserve-6545c766c4-h4chf" podUID="f29e8b65-b7dd-45d4-86b0-f13a29ef8e66" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": context deadline exceeded" Oct 29 22:57:20.980841 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:57:20.980812 5057 trace.go:236] Trace[1693937348]: "Calculate volume metrics of manifests for pod opendatahub-operator-system/opendatahub-operator-controller-manager-855c4cdcb6-s4nbq" (29-Oct-2026 22:57:19.171) (total time: 1809ms): Oct 29 22:57:20.980841 ip-10-0-140-74 kubenswrapper[5057]: Trace[1693937348]: [1.809219036s] [1.809219036s] END Oct 29 22:57:21.151160 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:57:21.151054 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-31226c/distinct-w6-31226c-kserve-858d95fd76-bkf7c" podUID="f3fc11c6-03c9-4314-99c3-f857b149415c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:21.829361 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:57:21.829298 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17394a/distinct-2-w3-17394a-kserve-6545c766c4-h4chf" podUID="f29e8b65-b7dd-45d4-86b0-f13a29ef8e66" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": context deadline exceeded" Oct 29 22:57:21.866156 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:57:21.849796 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-17394a/distinct-2-w3-17394a-kserve-6545c766c4-h4chf" Oct 29 22:57:21.874932 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:57:21.874904 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-17394a/distinct-2-w3-17394a-kserve-6545c766c4-h4chf" Oct 29 22:57:22.076660 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:57:22.076619 5057 patch_prober.go:29] interesting pod/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-k8ws5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 22:57:22.080131 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:57:22.080042 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-k8ws5" podUID="f53c1835-d1af-41c6-b8b9-e749ed7ef210" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:22.396916 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:57:22.396756 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:57:26.264888 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:57:26.264843 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:57:27.990942 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:57:27.990898 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:57:29.293440 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:57:29.293393 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:57:29.611912 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:57:29.611824 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:57:30.009748 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:57:30.009693 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:57:30.501170 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:57:30.501131 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:57:30.612392 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:57:30.610294 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:57:31.455420 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:57:31.455383 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:57:31.597657 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:57:31.591265 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:57:31.763387 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:57:31.763277 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:57:31.881753 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:57:31.881720 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:57:33.421378 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:57:33.421311 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:57:33.645470 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:57:33.645432 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:57:33.770992 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:57:33.770958 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:57:35.143822 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:57:35.143791 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:57:35.253919 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:57:35.253883 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:57:35.360279 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:57:35.360247 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:57:36.201289 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:57:36.201249 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:57:36.322669 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:57:36.322637 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:57:37.457080 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:57:37.457042 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:57:39.222134 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:57:39.222095 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:57:39.867546 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:57:39.867512 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:57:40.828574 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:57:40.828538 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:57:42.392977 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:57:42.392943 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:57:43.980863 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:57:43.980819 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:57:44.806241 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:57:44.806206 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:57:45.301507 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:57:45.301469 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:57:45.422141 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:57:45.422111 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:57:46.438106 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:57:46.438063 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:57:47.198934 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:57:47.198898 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:57:47.333136 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:57:47.333100 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:57:47.823306 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:57:47.823215 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:57:47.962423 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:57:47.960014 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:57:49.033507 ip-10-0-140-74 kubenswrapper[5057]: E1029 22:57:49.033471 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d2dnn" containerName="extractor" Oct 29 22:57:49.358560 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:57:49.358465 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:57:49.470731 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:57:49.470702 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:57:49.587979 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:57:49.587915 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:57:51.109587 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:57:51.109541 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:57:51.235893 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:57:51.235855 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:57:51.550489 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:57:51.550276 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:57:52.462891 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:57:52.462854 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:57:52.572279 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:57:52.572237 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:57:53.541279 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:57:53.541242 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:57:54.186471 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:57:54.186436 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:57:54.305260 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:57:54.305219 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:57:54.366923 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:57:54.366890 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a1d1da-7fd76d4dcf-bnwgx"] Oct 29 22:57:54.373633 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:57:54.373610 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a1d1da-7fd76d4dcf-bnwgx" Oct 29 22:57:54.395734 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:57:54.395520 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-a1d1da\"" Oct 29 22:57:54.395734 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:57:54.395576 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-a1d1da-dockercfg-4nt6d\"" Oct 29 22:57:54.399171 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:57:54.399143 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a1d1da-7fd76d4dcf-bnwgx"] Oct 29 22:57:54.499350 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:57:54.496600 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd8950b3-7cd5-4759-866e-1352ef928f64-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-a1d1da-7fd76d4dcf-bnwgx\" (UID: \"dd8950b3-7cd5-4759-866e-1352ef928f64\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a1d1da-7fd76d4dcf-bnwgx" Oct 29 22:57:54.511158 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:57:54.499818 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6lqh\" (UniqueName: \"kubernetes.io/projected/dd8950b3-7cd5-4759-866e-1352ef928f64-kube-api-access-n6lqh\") pod \"payload-pre-processing-e2e-ipp-a-a1d1da-7fd76d4dcf-bnwgx\" (UID: \"dd8950b3-7cd5-4759-866e-1352ef928f64\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a1d1da-7fd76d4dcf-bnwgx" Oct 29 22:57:54.614356 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:57:54.613594 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd8950b3-7cd5-4759-866e-1352ef928f64-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-a1d1da-7fd76d4dcf-bnwgx\" (UID: \"dd8950b3-7cd5-4759-866e-1352ef928f64\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a1d1da-7fd76d4dcf-bnwgx" Oct 29 22:57:54.614356 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:57:54.613743 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6lqh\" (UniqueName: \"kubernetes.io/projected/dd8950b3-7cd5-4759-866e-1352ef928f64-kube-api-access-n6lqh\") pod \"payload-pre-processing-e2e-ipp-a-a1d1da-7fd76d4dcf-bnwgx\" (UID: \"dd8950b3-7cd5-4759-866e-1352ef928f64\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a1d1da-7fd76d4dcf-bnwgx" Oct 29 22:57:54.615199 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:57:54.615092 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd8950b3-7cd5-4759-866e-1352ef928f64-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-a1d1da-7fd76d4dcf-bnwgx\" (UID: \"dd8950b3-7cd5-4759-866e-1352ef928f64\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a1d1da-7fd76d4dcf-bnwgx" Oct 29 22:57:54.638355 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:57:54.637744 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6lqh\" (UniqueName: \"kubernetes.io/projected/dd8950b3-7cd5-4759-866e-1352ef928f64-kube-api-access-n6lqh\") pod \"payload-pre-processing-e2e-ipp-a-a1d1da-7fd76d4dcf-bnwgx\" (UID: \"dd8950b3-7cd5-4759-866e-1352ef928f64\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a1d1da-7fd76d4dcf-bnwgx" Oct 29 22:57:54.695383 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:57:54.695320 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a1d1da-7fd76d4dcf-bnwgx" Oct 29 22:57:55.034837 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:57:55.034803 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a1d1da-7fd76d4dcf-bnwgx"] Oct 29 22:57:55.158964 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:57:55.158925 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a1d1da-7fd76d4dcf-bnwgx" event={"ID":"dd8950b3-7cd5-4759-866e-1352ef928f64","Type":"ContainerStarted","Data":"2e0224f4770b4db0f1572e69f086e2b0b4b0297850fabf753a710f036849c376"} Oct 29 22:57:55.159213 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:57:55.158993 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a1d1da-7fd76d4dcf-bnwgx" event={"ID":"dd8950b3-7cd5-4759-866e-1352ef928f64","Type":"ContainerStarted","Data":"32d7033642bee43a4b0c7bd147738fd17959563524de7815b0c432e5a6d00232"} Oct 29 22:57:55.160438 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:57:55.160410 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a1d1da-7fd76d4dcf-bnwgx" Oct 29 22:57:55.269389 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:57:55.254385 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v2tfm"] Oct 29 22:57:55.269389 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:57:55.255402 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a1d1da-7fd76d4dcf-bnwgx" podStartSLOduration=1.255384268 podStartE2EDuration="1.255384268s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:57:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:57:55.229768094 +0000 UTC m=+1456.012981521" watchObservedRunningTime="2026-10-29 22:57:55.255384268 +0000 UTC m=+1456.038597694" Oct 29 22:57:55.380406 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:57:55.371343 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:57:58.469468 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:57:58.469427 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-08e15d/distinct-w0-08e15d-kserve-85695c78bb-p67gk" podUID="e664f012-3b80-4a8a-ac19-d767d92c0a33" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": context deadline exceeded" Oct 29 22:57:58.675063 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:57:58.675011 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/distinct-2-w2-575d59-kserve-d4565b76c-hdgbf" podUID="06bb975e-d825-426f-b030-17bd1f99bb65" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:58.760543 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:57:58.760447 5057 patch_prober.go:29] interesting pod/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-nz2fk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 22:57:58.762353 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:57:58.762299 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-nz2fk" podUID="03a26d04-61f6-4e70-b52d-f6263220a95d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:58.887470 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:57:58.887432 5057 patch_prober.go:29] interesting pod/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-pk92s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 22:57:58.887751 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:57:58.887719 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-pk92s" podUID="9033e6ee-4fd2-41a2-8a7d-132b6ca20128" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:59.157442 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:57:59.144882 5057 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-31226c/distinct-w6-31226c-kserve-858d95fd76-bkf7c" podUID="f3fc11c6-03c9-4314-99c3-f857b149415c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:59.568354 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:57:59.535557 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-08e15d/distinct-w0-08e15d-kserve-85695c78bb-p67gk" podUID="e664f012-3b80-4a8a-ac19-d767d92c0a33" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": context deadline exceeded" Oct 29 22:57:59.718352 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:57:59.701625 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/distinct-2-w2-575d59-kserve-d4565b76c-hdgbf" podUID="06bb975e-d825-426f-b030-17bd1f99bb65" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:59.839408 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:57:59.829446 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-31226c/distinct-w6-31226c-kserve-858d95fd76-bkf7c" podUID="f3fc11c6-03c9-4314-99c3-f857b149415c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:59.901355 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:57:59.898595 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17394a/distinct-2-w3-17394a-kserve-6545c766c4-h4chf" podUID="f29e8b65-b7dd-45d4-86b0-f13a29ef8e66" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": context deadline exceeded" Oct 29 22:58:00.042357 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:00.021582 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-08e15d/distinct-w0-08e15d-kserve-85695c78bb-p67gk" Oct 29 22:58:00.077362 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:00.044650 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-a0dbf5-84c95d7876-qghzq"] Oct 29 22:58:00.190450 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:00.030778 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:58:00.370358 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:00.344818 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-575d59/distinct-2-w2-575d59-kserve-d4565b76c-hdgbf" Oct 29 22:58:00.370631 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:00.370607 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-a0dbf5-84c95d7876-qghzq"] Oct 29 22:58:00.370783 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:00.351097 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a0dbf5-84c95d7876-qghzq" Oct 29 22:58:00.397515 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:00.349941 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:58:00.456442 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:00.428581 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/70be02c2-cd2f-43dc-b689-3e088d9ebe10-maas-api-tls\") pod \"maas-api-e2e-api-b-a0dbf5-84c95d7876-qghzq\" (UID: \"70be02c2-cd2f-43dc-b689-3e088d9ebe10\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a0dbf5-84c95d7876-qghzq" Oct 29 22:58:00.660359 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:00.659893 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70be02c2-cd2f-43dc-b689-3e088d9ebe10-metrics-certs\") pod \"maas-api-e2e-api-b-a0dbf5-84c95d7876-qghzq\" (UID: \"70be02c2-cd2f-43dc-b689-3e088d9ebe10\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a0dbf5-84c95d7876-qghzq" Oct 29 22:58:00.660359 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:00.660072 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hs9r2\" (UniqueName: \"kubernetes.io/projected/70be02c2-cd2f-43dc-b689-3e088d9ebe10-kube-api-access-hs9r2\") pod \"maas-api-e2e-api-b-a0dbf5-84c95d7876-qghzq\" (UID: \"70be02c2-cd2f-43dc-b689-3e088d9ebe10\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a0dbf5-84c95d7876-qghzq" Oct 29 22:58:00.672231 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:00.501112 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-a0dbf5-serving-cert\"" Oct 29 22:58:00.696434 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:00.564154 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-08e15d/distinct-w0-08e15d-kserve-85695c78bb-p67gk" Oct 29 22:58:00.696685 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:00.684660 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/70be02c2-cd2f-43dc-b689-3e088d9ebe10-openshift-service-ca\") pod \"maas-api-e2e-api-b-a0dbf5-84c95d7876-qghzq\" (UID: \"70be02c2-cd2f-43dc-b689-3e088d9ebe10\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a0dbf5-84c95d7876-qghzq" Oct 29 22:58:00.737325 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:00.737184 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:58:00.843377 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:00.814530 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hs9r2\" (UniqueName: \"kubernetes.io/projected/70be02c2-cd2f-43dc-b689-3e088d9ebe10-kube-api-access-hs9r2\") pod \"maas-api-e2e-api-b-a0dbf5-84c95d7876-qghzq\" (UID: \"70be02c2-cd2f-43dc-b689-3e088d9ebe10\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a0dbf5-84c95d7876-qghzq" Oct 29 22:58:00.843377 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:00.814621 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/70be02c2-cd2f-43dc-b689-3e088d9ebe10-openshift-service-ca\") pod \"maas-api-e2e-api-b-a0dbf5-84c95d7876-qghzq\" (UID: \"70be02c2-cd2f-43dc-b689-3e088d9ebe10\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a0dbf5-84c95d7876-qghzq" Oct 29 22:58:00.843377 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:00.814964 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/70be02c2-cd2f-43dc-b689-3e088d9ebe10-maas-api-tls\") pod \"maas-api-e2e-api-b-a0dbf5-84c95d7876-qghzq\" (UID: \"70be02c2-cd2f-43dc-b689-3e088d9ebe10\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a0dbf5-84c95d7876-qghzq" Oct 29 22:58:00.843377 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:00.815000 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70be02c2-cd2f-43dc-b689-3e088d9ebe10-metrics-certs\") pod \"maas-api-e2e-api-b-a0dbf5-84c95d7876-qghzq\" (UID: \"70be02c2-cd2f-43dc-b689-3e088d9ebe10\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a0dbf5-84c95d7876-qghzq" Oct 29 22:58:00.843377 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:00.816170 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/70be02c2-cd2f-43dc-b689-3e088d9ebe10-openshift-service-ca\") pod \"maas-api-e2e-api-b-a0dbf5-84c95d7876-qghzq\" (UID: \"70be02c2-cd2f-43dc-b689-3e088d9ebe10\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a0dbf5-84c95d7876-qghzq" Oct 29 22:58:01.277531 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:01.218915 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70be02c2-cd2f-43dc-b689-3e088d9ebe10-metrics-certs\") pod \"maas-api-e2e-api-b-a0dbf5-84c95d7876-qghzq\" (UID: \"70be02c2-cd2f-43dc-b689-3e088d9ebe10\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a0dbf5-84c95d7876-qghzq" Oct 29 22:58:01.277531 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:01.219483 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/70be02c2-cd2f-43dc-b689-3e088d9ebe10-maas-api-tls\") pod \"maas-api-e2e-api-b-a0dbf5-84c95d7876-qghzq\" (UID: \"70be02c2-cd2f-43dc-b689-3e088d9ebe10\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a0dbf5-84c95d7876-qghzq" Oct 29 22:58:01.278716 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:01.277847 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17394a/distinct-2-w3-17394a-kserve-6545c766c4-h4chf" podUID="f29e8b65-b7dd-45d4-86b0-f13a29ef8e66" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:58:01.299354 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:01.278902 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-31226c/distinct-w6-31226c-kserve-858d95fd76-bkf7c" podUID="f3fc11c6-03c9-4314-99c3-f857b149415c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded" Oct 29 22:58:01.299354 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:01.282026 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hs9r2\" (UniqueName: \"kubernetes.io/projected/70be02c2-cd2f-43dc-b689-3e088d9ebe10-kube-api-access-hs9r2\") pod \"maas-api-e2e-api-b-a0dbf5-84c95d7876-qghzq\" (UID: \"70be02c2-cd2f-43dc-b689-3e088d9ebe10\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a0dbf5-84c95d7876-qghzq" Oct 29 22:58:01.299354 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:01.292167 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-17394a/distinct-2-w3-17394a-kserve-6545c766c4-h4chf" Oct 29 22:58:01.299354 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:01.292248 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-31226c/distinct-w6-31226c-kserve-858d95fd76-bkf7c" Oct 29 22:58:01.301508 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:01.301476 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:58:01.385611 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:01.385567 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/distinct-2-w2-575d59-kserve-d4565b76c-hdgbf" podUID="06bb975e-d825-426f-b030-17bd1f99bb65" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:58:01.460359 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:01.440811 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a0dbf5-84c95d7876-qghzq" Oct 29 22:58:02.053404 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:01.993200 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-575d59/distinct-2-w2-575d59-kserve-d4565b76c-hdgbf" Oct 29 22:58:02.113497 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:02.113452 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-31226c/distinct-w6-31226c-kserve-858d95fd76-bkf7c" Oct 29 22:58:02.113825 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:02.113804 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-17394a/distinct-2-w3-17394a-kserve-6545c766c4-h4chf" Oct 29 22:58:03.060079 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:58:03.053460 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70be02c2_cd2f_43dc_b689_3e088d9ebe10.slice/crio-f5f36edca6af26be61bdbc7c4a4cd16b341ab195a2c69eecde23f08b2b680c7f WatchSource:0}: Error finding container f5f36edca6af26be61bdbc7c4a4cd16b341ab195a2c69eecde23f08b2b680c7f: Status 404 returned error can't find the container with id f5f36edca6af26be61bdbc7c4a4cd16b341ab195a2c69eecde23f08b2b680c7f Oct 29 22:58:03.060079 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:03.054201 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-a0dbf5-84c95d7876-qghzq"] Oct 29 22:58:03.543159 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:03.543128 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a0dbf5-84c95d7876-qghzq" event={"ID":"70be02c2-cd2f-43dc-b689-3e088d9ebe10","Type":"ContainerStarted","Data":"e7fd367a9f94477df8aa8f79f20493c3fcff59d46e9ae3a5ac0addfa6425e131"} Oct 29 22:58:03.543392 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:03.543183 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a0dbf5-84c95d7876-qghzq" event={"ID":"70be02c2-cd2f-43dc-b689-3e088d9ebe10","Type":"ContainerStarted","Data":"f5f36edca6af26be61bdbc7c4a4cd16b341ab195a2c69eecde23f08b2b680c7f"} Oct 29 22:58:03.543392 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:03.543252 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a0dbf5-84c95d7876-qghzq" Oct 29 22:58:03.589522 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:03.589453 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a0dbf5-84c95d7876-qghzq" podStartSLOduration=4.216462861 podStartE2EDuration="4.589432099s" totalImagesPullingTime="372.969238ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:57:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:58:03.582755047 +0000 UTC m=+1464.365968476" watchObservedRunningTime="2026-10-29 22:58:03.589432099 +0000 UTC m=+1464.372645525" Oct 29 22:58:07.068212 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:07.068171 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:58:07.237983 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:07.237904 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a1d1da-7fd76d4dcf-bnwgx" Oct 29 22:58:09.500177 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:09.500139 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:58:09.573607 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:09.573573 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a0dbf5-84c95d7876-qghzq" Oct 29 22:58:10.558261 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:10.558230 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-27d16a-openshift-default-869755f997-ws2s6"] Oct 29 22:58:10.563372 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:10.563322 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-27d16a-openshift-default-869755f997-ws2s6" Oct 29 22:58:10.595014 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:10.594966 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-27d16a-openshift-default-869755f997-ws2s6"] Oct 29 22:58:10.624608 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:10.624565 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6af2d78-ba9f-4f94-a985-47985aa30dd3-istio-envoy\") pod \"e2e-api-b-27d16a-openshift-default-869755f997-ws2s6\" (UID: \"e6af2d78-ba9f-4f94-a985-47985aa30dd3\") " pod="openshift-ingress/e2e-api-b-27d16a-openshift-default-869755f997-ws2s6" Oct 29 22:58:10.624783 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:10.624617 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6af2d78-ba9f-4f94-a985-47985aa30dd3-istiod-ca-cert\") pod \"e2e-api-b-27d16a-openshift-default-869755f997-ws2s6\" (UID: \"e6af2d78-ba9f-4f94-a985-47985aa30dd3\") " pod="openshift-ingress/e2e-api-b-27d16a-openshift-default-869755f997-ws2s6" Oct 29 22:58:10.624783 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:10.624697 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9dkpt\" (UniqueName: \"kubernetes.io/projected/e6af2d78-ba9f-4f94-a985-47985aa30dd3-kube-api-access-9dkpt\") pod \"e2e-api-b-27d16a-openshift-default-869755f997-ws2s6\" (UID: \"e6af2d78-ba9f-4f94-a985-47985aa30dd3\") " pod="openshift-ingress/e2e-api-b-27d16a-openshift-default-869755f997-ws2s6" Oct 29 22:58:10.624783 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:10.624757 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6af2d78-ba9f-4f94-a985-47985aa30dd3-workload-certs\") pod \"e2e-api-b-27d16a-openshift-default-869755f997-ws2s6\" (UID: \"e6af2d78-ba9f-4f94-a985-47985aa30dd3\") " pod="openshift-ingress/e2e-api-b-27d16a-openshift-default-869755f997-ws2s6" Oct 29 22:58:10.624967 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:10.624791 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6af2d78-ba9f-4f94-a985-47985aa30dd3-istio-podinfo\") pod \"e2e-api-b-27d16a-openshift-default-869755f997-ws2s6\" (UID: \"e6af2d78-ba9f-4f94-a985-47985aa30dd3\") " pod="openshift-ingress/e2e-api-b-27d16a-openshift-default-869755f997-ws2s6" Oct 29 22:58:10.624967 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:10.624914 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6af2d78-ba9f-4f94-a985-47985aa30dd3-istio-data\") pod \"e2e-api-b-27d16a-openshift-default-869755f997-ws2s6\" (UID: \"e6af2d78-ba9f-4f94-a985-47985aa30dd3\") " pod="openshift-ingress/e2e-api-b-27d16a-openshift-default-869755f997-ws2s6" Oct 29 22:58:10.625076 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:10.624977 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6af2d78-ba9f-4f94-a985-47985aa30dd3-workload-socket\") pod \"e2e-api-b-27d16a-openshift-default-869755f997-ws2s6\" (UID: \"e6af2d78-ba9f-4f94-a985-47985aa30dd3\") " pod="openshift-ingress/e2e-api-b-27d16a-openshift-default-869755f997-ws2s6" Oct 29 22:58:10.625076 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:10.625015 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6af2d78-ba9f-4f94-a985-47985aa30dd3-istio-token\") pod \"e2e-api-b-27d16a-openshift-default-869755f997-ws2s6\" (UID: \"e6af2d78-ba9f-4f94-a985-47985aa30dd3\") " pod="openshift-ingress/e2e-api-b-27d16a-openshift-default-869755f997-ws2s6" Oct 29 22:58:10.625076 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:10.625031 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6af2d78-ba9f-4f94-a985-47985aa30dd3-credential-socket\") pod \"e2e-api-b-27d16a-openshift-default-869755f997-ws2s6\" (UID: \"e6af2d78-ba9f-4f94-a985-47985aa30dd3\") " pod="openshift-ingress/e2e-api-b-27d16a-openshift-default-869755f997-ws2s6" Oct 29 22:58:10.667668 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:10.667638 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-27d16a-openshift-default-dockercfg-dq7sj\"" Oct 29 22:58:10.726782 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:10.726748 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6af2d78-ba9f-4f94-a985-47985aa30dd3-istio-data\") pod \"e2e-api-b-27d16a-openshift-default-869755f997-ws2s6\" (UID: \"e6af2d78-ba9f-4f94-a985-47985aa30dd3\") " pod="openshift-ingress/e2e-api-b-27d16a-openshift-default-869755f997-ws2s6" Oct 29 22:58:10.726928 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:10.726820 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6af2d78-ba9f-4f94-a985-47985aa30dd3-workload-socket\") pod \"e2e-api-b-27d16a-openshift-default-869755f997-ws2s6\" (UID: \"e6af2d78-ba9f-4f94-a985-47985aa30dd3\") " pod="openshift-ingress/e2e-api-b-27d16a-openshift-default-869755f997-ws2s6" Oct 29 22:58:10.726928 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:10.726855 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6af2d78-ba9f-4f94-a985-47985aa30dd3-istio-token\") pod \"e2e-api-b-27d16a-openshift-default-869755f997-ws2s6\" (UID: \"e6af2d78-ba9f-4f94-a985-47985aa30dd3\") " pod="openshift-ingress/e2e-api-b-27d16a-openshift-default-869755f997-ws2s6" Oct 29 22:58:10.726928 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:10.726870 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6af2d78-ba9f-4f94-a985-47985aa30dd3-credential-socket\") pod \"e2e-api-b-27d16a-openshift-default-869755f997-ws2s6\" (UID: \"e6af2d78-ba9f-4f94-a985-47985aa30dd3\") " pod="openshift-ingress/e2e-api-b-27d16a-openshift-default-869755f997-ws2s6" Oct 29 22:58:10.727099 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:10.726934 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6af2d78-ba9f-4f94-a985-47985aa30dd3-istio-envoy\") pod \"e2e-api-b-27d16a-openshift-default-869755f997-ws2s6\" (UID: \"e6af2d78-ba9f-4f94-a985-47985aa30dd3\") " pod="openshift-ingress/e2e-api-b-27d16a-openshift-default-869755f997-ws2s6" Oct 29 22:58:10.727099 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:10.726959 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6af2d78-ba9f-4f94-a985-47985aa30dd3-istiod-ca-cert\") pod \"e2e-api-b-27d16a-openshift-default-869755f997-ws2s6\" (UID: \"e6af2d78-ba9f-4f94-a985-47985aa30dd3\") " pod="openshift-ingress/e2e-api-b-27d16a-openshift-default-869755f997-ws2s6" Oct 29 22:58:10.727099 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:10.726981 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dkpt\" (UniqueName: \"kubernetes.io/projected/e6af2d78-ba9f-4f94-a985-47985aa30dd3-kube-api-access-9dkpt\") pod \"e2e-api-b-27d16a-openshift-default-869755f997-ws2s6\" (UID: \"e6af2d78-ba9f-4f94-a985-47985aa30dd3\") " pod="openshift-ingress/e2e-api-b-27d16a-openshift-default-869755f997-ws2s6" Oct 29 22:58:10.727099 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:10.727044 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6af2d78-ba9f-4f94-a985-47985aa30dd3-workload-certs\") pod \"e2e-api-b-27d16a-openshift-default-869755f997-ws2s6\" (UID: \"e6af2d78-ba9f-4f94-a985-47985aa30dd3\") " pod="openshift-ingress/e2e-api-b-27d16a-openshift-default-869755f997-ws2s6" Oct 29 22:58:10.727099 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:10.727083 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6af2d78-ba9f-4f94-a985-47985aa30dd3-istio-podinfo\") pod \"e2e-api-b-27d16a-openshift-default-869755f997-ws2s6\" (UID: \"e6af2d78-ba9f-4f94-a985-47985aa30dd3\") " pod="openshift-ingress/e2e-api-b-27d16a-openshift-default-869755f997-ws2s6" Oct 29 22:58:10.727390 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:10.727245 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6af2d78-ba9f-4f94-a985-47985aa30dd3-istio-data\") pod \"e2e-api-b-27d16a-openshift-default-869755f997-ws2s6\" (UID: \"e6af2d78-ba9f-4f94-a985-47985aa30dd3\") " pod="openshift-ingress/e2e-api-b-27d16a-openshift-default-869755f997-ws2s6" Oct 29 22:58:10.727390 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:10.727285 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6af2d78-ba9f-4f94-a985-47985aa30dd3-workload-socket\") pod \"e2e-api-b-27d16a-openshift-default-869755f997-ws2s6\" (UID: \"e6af2d78-ba9f-4f94-a985-47985aa30dd3\") " pod="openshift-ingress/e2e-api-b-27d16a-openshift-default-869755f997-ws2s6" Oct 29 22:58:10.727510 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:10.727496 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6af2d78-ba9f-4f94-a985-47985aa30dd3-workload-certs\") pod \"e2e-api-b-27d16a-openshift-default-869755f997-ws2s6\" (UID: \"e6af2d78-ba9f-4f94-a985-47985aa30dd3\") " pod="openshift-ingress/e2e-api-b-27d16a-openshift-default-869755f997-ws2s6" Oct 29 22:58:10.727785 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:10.727761 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6af2d78-ba9f-4f94-a985-47985aa30dd3-istiod-ca-cert\") pod \"e2e-api-b-27d16a-openshift-default-869755f997-ws2s6\" (UID: \"e6af2d78-ba9f-4f94-a985-47985aa30dd3\") " pod="openshift-ingress/e2e-api-b-27d16a-openshift-default-869755f997-ws2s6" Oct 29 22:58:10.727881 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:10.727806 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6af2d78-ba9f-4f94-a985-47985aa30dd3-credential-socket\") pod \"e2e-api-b-27d16a-openshift-default-869755f997-ws2s6\" (UID: \"e6af2d78-ba9f-4f94-a985-47985aa30dd3\") " pod="openshift-ingress/e2e-api-b-27d16a-openshift-default-869755f997-ws2s6" Oct 29 22:58:10.729948 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:10.729923 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6af2d78-ba9f-4f94-a985-47985aa30dd3-istio-envoy\") pod \"e2e-api-b-27d16a-openshift-default-869755f997-ws2s6\" (UID: \"e6af2d78-ba9f-4f94-a985-47985aa30dd3\") " pod="openshift-ingress/e2e-api-b-27d16a-openshift-default-869755f997-ws2s6" Oct 29 22:58:10.730062 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:10.730045 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6af2d78-ba9f-4f94-a985-47985aa30dd3-istio-podinfo\") pod \"e2e-api-b-27d16a-openshift-default-869755f997-ws2s6\" (UID: \"e6af2d78-ba9f-4f94-a985-47985aa30dd3\") " pod="openshift-ingress/e2e-api-b-27d16a-openshift-default-869755f997-ws2s6" Oct 29 22:58:10.748721 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:10.748703 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6af2d78-ba9f-4f94-a985-47985aa30dd3-istio-token\") pod \"e2e-api-b-27d16a-openshift-default-869755f997-ws2s6\" (UID: \"e6af2d78-ba9f-4f94-a985-47985aa30dd3\") " pod="openshift-ingress/e2e-api-b-27d16a-openshift-default-869755f997-ws2s6" Oct 29 22:58:10.748802 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:10.748760 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9dkpt\" (UniqueName: \"kubernetes.io/projected/e6af2d78-ba9f-4f94-a985-47985aa30dd3-kube-api-access-9dkpt\") pod \"e2e-api-b-27d16a-openshift-default-869755f997-ws2s6\" (UID: \"e6af2d78-ba9f-4f94-a985-47985aa30dd3\") " pod="openshift-ingress/e2e-api-b-27d16a-openshift-default-869755f997-ws2s6" Oct 29 22:58:10.886223 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:10.886147 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-27d16a-openshift-default-869755f997-ws2s6" Oct 29 22:58:11.321360 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:11.321239 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:58:11.324724 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:11.321576 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:58:11.324724 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:11.321653 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:58:11.354669 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:11.354637 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-27d16a-openshift-default-869755f997-ws2s6"] Oct 29 22:58:11.572969 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:11.572890 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:58:11.619432 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:11.619391 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-27d16a-openshift-default-869755f997-ws2s6" event={"ID":"e6af2d78-ba9f-4f94-a985-47985aa30dd3","Type":"ContainerStarted","Data":"0148cf646e079db62e52d31a8ddc05db1e5e580fbd3a9b7063aeaf4ecec39bb2"} Oct 29 22:58:11.619699 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:11.619681 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-27d16a-openshift-default-869755f997-ws2s6" event={"ID":"e6af2d78-ba9f-4f94-a985-47985aa30dd3","Type":"ContainerStarted","Data":"ccbd0be5546ace2f2768c2f9581ce0e4b895121b24c2998b7215bcf0b1d0f834"} Oct 29 22:58:11.667562 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:11.667505 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-27d16a-openshift-default-869755f997-ws2s6" podStartSLOduration=1.667488224 podStartE2EDuration="1.667488224s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:58:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:58:11.662148875 +0000 UTC m=+1472.445362302" watchObservedRunningTime="2026-10-29 22:58:11.667488224 +0000 UTC m=+1472.450701650" Oct 29 22:58:11.889129 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:11.889037 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-27d16a-openshift-default-869755f997-ws2s6" Oct 29 22:58:12.204407 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:12.195277 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:58:12.872576 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:12.872541 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:58:12.895356 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:12.892555 5057 patch_prober.go:29] interesting pod/e2e-api-b-27d16a-openshift-default-869755f997-ws2s6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Oct 29 22:58:12.895356 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:12.892611 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-27d16a-openshift-default-869755f997-ws2s6" podUID="e6af2d78-ba9f-4f94-a985-47985aa30dd3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Oct 29 22:58:13.374675 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:13.373658 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-cd525f-openshift-default-5567b55479-lhtg7"] Oct 29 22:58:13.415358 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:13.407730 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-cd525f-openshift-default-5567b55479-lhtg7" Oct 29 22:58:13.564362 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:13.547203 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-cd525f-openshift-default-dockercfg-rkhnr\"" Oct 29 22:58:13.564928 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:13.564891 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3616f622-b298-42ce-bf47-80e53935b0f0-workload-certs\") pod \"e2e-api-a-cd525f-openshift-default-5567b55479-lhtg7\" (UID: \"3616f622-b298-42ce-bf47-80e53935b0f0\") " pod="openshift-ingress/e2e-api-a-cd525f-openshift-default-5567b55479-lhtg7" Oct 29 22:58:13.578354 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:13.565090 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3616f622-b298-42ce-bf47-80e53935b0f0-istiod-ca-cert\") pod \"e2e-api-a-cd525f-openshift-default-5567b55479-lhtg7\" (UID: \"3616f622-b298-42ce-bf47-80e53935b0f0\") " pod="openshift-ingress/e2e-api-a-cd525f-openshift-default-5567b55479-lhtg7" Oct 29 22:58:13.578354 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:13.565163 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3616f622-b298-42ce-bf47-80e53935b0f0-workload-socket\") pod \"e2e-api-a-cd525f-openshift-default-5567b55479-lhtg7\" (UID: \"3616f622-b298-42ce-bf47-80e53935b0f0\") " pod="openshift-ingress/e2e-api-a-cd525f-openshift-default-5567b55479-lhtg7" Oct 29 22:58:13.578354 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:13.565231 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3616f622-b298-42ce-bf47-80e53935b0f0-istio-envoy\") pod \"e2e-api-a-cd525f-openshift-default-5567b55479-lhtg7\" (UID: \"3616f622-b298-42ce-bf47-80e53935b0f0\") " pod="openshift-ingress/e2e-api-a-cd525f-openshift-default-5567b55479-lhtg7" Oct 29 22:58:13.578354 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:13.565270 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3616f622-b298-42ce-bf47-80e53935b0f0-istio-podinfo\") pod \"e2e-api-a-cd525f-openshift-default-5567b55479-lhtg7\" (UID: \"3616f622-b298-42ce-bf47-80e53935b0f0\") " pod="openshift-ingress/e2e-api-a-cd525f-openshift-default-5567b55479-lhtg7" Oct 29 22:58:13.579233 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:13.565324 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzfb9\" (UniqueName: \"kubernetes.io/projected/3616f622-b298-42ce-bf47-80e53935b0f0-kube-api-access-rzfb9\") pod \"e2e-api-a-cd525f-openshift-default-5567b55479-lhtg7\" (UID: \"3616f622-b298-42ce-bf47-80e53935b0f0\") " pod="openshift-ingress/e2e-api-a-cd525f-openshift-default-5567b55479-lhtg7" Oct 29 22:58:13.579233 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:13.578801 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3616f622-b298-42ce-bf47-80e53935b0f0-credential-socket\") pod \"e2e-api-a-cd525f-openshift-default-5567b55479-lhtg7\" (UID: \"3616f622-b298-42ce-bf47-80e53935b0f0\") " pod="openshift-ingress/e2e-api-a-cd525f-openshift-default-5567b55479-lhtg7" Oct 29 22:58:13.579233 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:13.578839 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3616f622-b298-42ce-bf47-80e53935b0f0-istio-token\") pod \"e2e-api-a-cd525f-openshift-default-5567b55479-lhtg7\" (UID: \"3616f622-b298-42ce-bf47-80e53935b0f0\") " pod="openshift-ingress/e2e-api-a-cd525f-openshift-default-5567b55479-lhtg7" Oct 29 22:58:13.579233 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:13.578892 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3616f622-b298-42ce-bf47-80e53935b0f0-istio-data\") pod \"e2e-api-a-cd525f-openshift-default-5567b55479-lhtg7\" (UID: \"3616f622-b298-42ce-bf47-80e53935b0f0\") " pod="openshift-ingress/e2e-api-a-cd525f-openshift-default-5567b55479-lhtg7" Oct 29 22:58:13.637471 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:13.636577 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:58:13.799601 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:13.797197 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:58:13.801839 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:13.801794 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3616f622-b298-42ce-bf47-80e53935b0f0-workload-certs\") pod \"e2e-api-a-cd525f-openshift-default-5567b55479-lhtg7\" (UID: \"3616f622-b298-42ce-bf47-80e53935b0f0\") " pod="openshift-ingress/e2e-api-a-cd525f-openshift-default-5567b55479-lhtg7" Oct 29 22:58:13.802136 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:13.802022 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3616f622-b298-42ce-bf47-80e53935b0f0-istiod-ca-cert\") pod \"e2e-api-a-cd525f-openshift-default-5567b55479-lhtg7\" (UID: \"3616f622-b298-42ce-bf47-80e53935b0f0\") " pod="openshift-ingress/e2e-api-a-cd525f-openshift-default-5567b55479-lhtg7" Oct 29 22:58:13.802403 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:13.802251 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3616f622-b298-42ce-bf47-80e53935b0f0-workload-socket\") pod \"e2e-api-a-cd525f-openshift-default-5567b55479-lhtg7\" (UID: \"3616f622-b298-42ce-bf47-80e53935b0f0\") " pod="openshift-ingress/e2e-api-a-cd525f-openshift-default-5567b55479-lhtg7" Oct 29 22:58:13.802740 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:13.802685 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3616f622-b298-42ce-bf47-80e53935b0f0-istio-envoy\") pod \"e2e-api-a-cd525f-openshift-default-5567b55479-lhtg7\" (UID: \"3616f622-b298-42ce-bf47-80e53935b0f0\") " pod="openshift-ingress/e2e-api-a-cd525f-openshift-default-5567b55479-lhtg7" Oct 29 22:58:13.802990 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:13.802966 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3616f622-b298-42ce-bf47-80e53935b0f0-istio-podinfo\") pod \"e2e-api-a-cd525f-openshift-default-5567b55479-lhtg7\" (UID: \"3616f622-b298-42ce-bf47-80e53935b0f0\") " pod="openshift-ingress/e2e-api-a-cd525f-openshift-default-5567b55479-lhtg7" Oct 29 22:58:13.803263 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:13.803230 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzfb9\" (UniqueName: \"kubernetes.io/projected/3616f622-b298-42ce-bf47-80e53935b0f0-kube-api-access-rzfb9\") pod \"e2e-api-a-cd525f-openshift-default-5567b55479-lhtg7\" (UID: \"3616f622-b298-42ce-bf47-80e53935b0f0\") " pod="openshift-ingress/e2e-api-a-cd525f-openshift-default-5567b55479-lhtg7" Oct 29 22:58:13.803494 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:13.803408 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3616f622-b298-42ce-bf47-80e53935b0f0-credential-socket\") pod \"e2e-api-a-cd525f-openshift-default-5567b55479-lhtg7\" (UID: \"3616f622-b298-42ce-bf47-80e53935b0f0\") " pod="openshift-ingress/e2e-api-a-cd525f-openshift-default-5567b55479-lhtg7" Oct 29 22:58:13.803873 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:13.803851 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3616f622-b298-42ce-bf47-80e53935b0f0-istio-token\") pod \"e2e-api-a-cd525f-openshift-default-5567b55479-lhtg7\" (UID: \"3616f622-b298-42ce-bf47-80e53935b0f0\") " pod="openshift-ingress/e2e-api-a-cd525f-openshift-default-5567b55479-lhtg7" Oct 29 22:58:13.804113 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:13.804077 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3616f622-b298-42ce-bf47-80e53935b0f0-istio-data\") pod \"e2e-api-a-cd525f-openshift-default-5567b55479-lhtg7\" (UID: \"3616f622-b298-42ce-bf47-80e53935b0f0\") " pod="openshift-ingress/e2e-api-a-cd525f-openshift-default-5567b55479-lhtg7" Oct 29 22:58:13.809402 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:13.809365 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3616f622-b298-42ce-bf47-80e53935b0f0-workload-certs\") pod \"e2e-api-a-cd525f-openshift-default-5567b55479-lhtg7\" (UID: \"3616f622-b298-42ce-bf47-80e53935b0f0\") " pod="openshift-ingress/e2e-api-a-cd525f-openshift-default-5567b55479-lhtg7" Oct 29 22:58:13.811024 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:13.811000 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3616f622-b298-42ce-bf47-80e53935b0f0-istio-data\") pod \"e2e-api-a-cd525f-openshift-default-5567b55479-lhtg7\" (UID: \"3616f622-b298-42ce-bf47-80e53935b0f0\") " pod="openshift-ingress/e2e-api-a-cd525f-openshift-default-5567b55479-lhtg7" Oct 29 22:58:13.823539 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:13.822468 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3616f622-b298-42ce-bf47-80e53935b0f0-istiod-ca-cert\") pod \"e2e-api-a-cd525f-openshift-default-5567b55479-lhtg7\" (UID: \"3616f622-b298-42ce-bf47-80e53935b0f0\") " pod="openshift-ingress/e2e-api-a-cd525f-openshift-default-5567b55479-lhtg7" Oct 29 22:58:13.823539 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:13.822909 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3616f622-b298-42ce-bf47-80e53935b0f0-workload-socket\") pod \"e2e-api-a-cd525f-openshift-default-5567b55479-lhtg7\" (UID: \"3616f622-b298-42ce-bf47-80e53935b0f0\") " pod="openshift-ingress/e2e-api-a-cd525f-openshift-default-5567b55479-lhtg7" Oct 29 22:58:13.846348 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:13.838689 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3616f622-b298-42ce-bf47-80e53935b0f0-credential-socket\") pod \"e2e-api-a-cd525f-openshift-default-5567b55479-lhtg7\" (UID: \"3616f622-b298-42ce-bf47-80e53935b0f0\") " pod="openshift-ingress/e2e-api-a-cd525f-openshift-default-5567b55479-lhtg7" Oct 29 22:58:13.848834 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:13.848561 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3616f622-b298-42ce-bf47-80e53935b0f0-istio-envoy\") pod \"e2e-api-a-cd525f-openshift-default-5567b55479-lhtg7\" (UID: \"3616f622-b298-42ce-bf47-80e53935b0f0\") " pod="openshift-ingress/e2e-api-a-cd525f-openshift-default-5567b55479-lhtg7" Oct 29 22:58:13.849489 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:13.849458 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3616f622-b298-42ce-bf47-80e53935b0f0-istio-podinfo\") pod \"e2e-api-a-cd525f-openshift-default-5567b55479-lhtg7\" (UID: \"3616f622-b298-42ce-bf47-80e53935b0f0\") " pod="openshift-ingress/e2e-api-a-cd525f-openshift-default-5567b55479-lhtg7" Oct 29 22:58:13.863308 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:13.863275 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzfb9\" (UniqueName: \"kubernetes.io/projected/3616f622-b298-42ce-bf47-80e53935b0f0-kube-api-access-rzfb9\") pod \"e2e-api-a-cd525f-openshift-default-5567b55479-lhtg7\" (UID: \"3616f622-b298-42ce-bf47-80e53935b0f0\") " pod="openshift-ingress/e2e-api-a-cd525f-openshift-default-5567b55479-lhtg7" Oct 29 22:58:13.866645 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:13.866616 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3616f622-b298-42ce-bf47-80e53935b0f0-istio-token\") pod \"e2e-api-a-cd525f-openshift-default-5567b55479-lhtg7\" (UID: \"3616f622-b298-42ce-bf47-80e53935b0f0\") " pod="openshift-ingress/e2e-api-a-cd525f-openshift-default-5567b55479-lhtg7" Oct 29 22:58:13.918410 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:13.915028 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-cd525f-openshift-default-5567b55479-lhtg7"] Oct 29 22:58:13.918410 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:13.915250 5057 patch_prober.go:29] interesting pod/e2e-api-b-27d16a-openshift-default-869755f997-ws2s6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Oct 29 22:58:13.918410 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:13.915320 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-27d16a-openshift-default-869755f997-ws2s6" podUID="e6af2d78-ba9f-4f94-a985-47985aa30dd3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Oct 29 22:58:14.162112 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:14.162079 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-cd525f-openshift-default-5567b55479-lhtg7" Oct 29 22:58:14.573472 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:14.573443 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-cd525f-openshift-default-5567b55479-lhtg7"] Oct 29 22:58:14.574948 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:58:14.574923 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3616f622_b298_42ce_bf47_80e53935b0f0.slice/crio-5dbcedba39e1debea8d2c66aa06574f5448167c71b5bebca5cf42ecd6e743ff2 WatchSource:0}: Error finding container 5dbcedba39e1debea8d2c66aa06574f5448167c71b5bebca5cf42ecd6e743ff2: Status 404 returned error can't find the container with id 5dbcedba39e1debea8d2c66aa06574f5448167c71b5bebca5cf42ecd6e743ff2 Oct 29 22:58:14.608100 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:14.608059 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:58:14.608253 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:14.608134 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:58:14.608253 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:14.608165 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 22:58:14.749845 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:14.749814 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:58:14.888411 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:14.887742 5057 patch_prober.go:29] interesting pod/e2e-api-b-27d16a-openshift-default-869755f997-ws2s6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Oct 29 22:58:14.888411 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:14.887804 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-27d16a-openshift-default-869755f997-ws2s6" podUID="e6af2d78-ba9f-4f94-a985-47985aa30dd3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Oct 29 22:58:14.931355 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:14.924009 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:58:14.938684 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:14.938648 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-cd525f-openshift-default-5567b55479-lhtg7" event={"ID":"3616f622-b298-42ce-bf47-80e53935b0f0","Type":"ContainerStarted","Data":"3dfbe1f9f3c510824304b205b4edc2da21024f49a1107319eb6078094b500f34"} Oct 29 22:58:14.938880 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:14.938859 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-cd525f-openshift-default-5567b55479-lhtg7" event={"ID":"3616f622-b298-42ce-bf47-80e53935b0f0","Type":"ContainerStarted","Data":"5dbcedba39e1debea8d2c66aa06574f5448167c71b5bebca5cf42ecd6e743ff2"} Oct 29 22:58:15.034780 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:15.034727 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-cd525f-openshift-default-5567b55479-lhtg7" podStartSLOduration=2.034711949 podStartE2EDuration="2.034711949s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:58:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:58:15.030877464 +0000 UTC m=+1475.814090908" watchObservedRunningTime="2026-10-29 22:58:15.034711949 +0000 UTC m=+1475.817925375" Oct 29 22:58:15.163279 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:15.163186 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-cd525f-openshift-default-5567b55479-lhtg7" Oct 29 22:58:15.164641 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:15.164403 5057 patch_prober.go:29] interesting pod/e2e-api-a-cd525f-openshift-default-5567b55479-lhtg7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Oct 29 22:58:15.164641 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:15.164447 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-cd525f-openshift-default-5567b55479-lhtg7" podUID="3616f622-b298-42ce-bf47-80e53935b0f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Oct 29 22:58:15.559842 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:15.559703 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:58:15.896411 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:15.894764 5057 patch_prober.go:29] interesting pod/e2e-api-b-27d16a-openshift-default-869755f997-ws2s6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Oct 29 22:58:15.896411 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:15.894827 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-27d16a-openshift-default-869755f997-ws2s6" podUID="e6af2d78-ba9f-4f94-a985-47985aa30dd3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Oct 29 22:58:16.162934 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:16.162843 5057 patch_prober.go:29] interesting pod/e2e-api-a-cd525f-openshift-default-5567b55479-lhtg7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Oct 29 22:58:16.163544 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:16.163508 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-cd525f-openshift-default-5567b55479-lhtg7" podUID="3616f622-b298-42ce-bf47-80e53935b0f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Oct 29 22:58:16.916395 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:16.900946 5057 patch_prober.go:29] interesting pod/e2e-api-b-27d16a-openshift-default-869755f997-ws2s6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Oct 29 22:58:16.916395 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:16.901024 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-27d16a-openshift-default-869755f997-ws2s6" podUID="e6af2d78-ba9f-4f94-a985-47985aa30dd3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Oct 29 22:58:17.162712 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:17.162676 5057 patch_prober.go:29] interesting pod/e2e-api-a-cd525f-openshift-default-5567b55479-lhtg7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Oct 29 22:58:17.162889 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:17.162723 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-cd525f-openshift-default-5567b55479-lhtg7" podUID="3616f622-b298-42ce-bf47-80e53935b0f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Oct 29 22:58:17.356218 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:17.356031 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:58:17.745766 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:17.745732 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:58:17.865945 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:17.865866 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:58:17.893413 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:17.892551 5057 patch_prober.go:29] interesting pod/e2e-api-b-27d16a-openshift-default-869755f997-ws2s6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Oct 29 22:58:17.893413 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:17.892642 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-27d16a-openshift-default-869755f997-ws2s6" podUID="e6af2d78-ba9f-4f94-a985-47985aa30dd3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Oct 29 22:58:18.163589 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:18.163499 5057 patch_prober.go:29] interesting pod/e2e-api-a-cd525f-openshift-default-5567b55479-lhtg7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Oct 29 22:58:18.163589 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:18.163565 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-cd525f-openshift-default-5567b55479-lhtg7" podUID="3616f622-b298-42ce-bf47-80e53935b0f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Oct 29 22:58:18.901658 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:18.901622 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-27d16a-openshift-default-869755f997-ws2s6" Oct 29 22:58:18.904553 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:18.904530 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-27d16a-openshift-default-869755f997-ws2s6" Oct 29 22:58:18.910610 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:18.910589 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-27d16a-openshift-default-869755f997-ws2s6" Oct 29 22:58:19.107957 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:19.107918 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:58:19.112525 ip-10-0-140-74 kubenswrapper[5057]: E1029 22:58:19.112482 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d2dnn" containerName="extractor" Oct 29 22:58:19.167805 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:19.167142 5057 patch_prober.go:29] interesting pod/e2e-api-a-cd525f-openshift-default-5567b55479-lhtg7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Oct 29 22:58:19.167805 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:19.167202 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-cd525f-openshift-default-5567b55479-lhtg7" podUID="3616f622-b298-42ce-bf47-80e53935b0f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Oct 29 22:58:19.469028 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:19.468988 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:58:19.599188 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:19.599151 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:58:20.163658 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:20.163620 5057 patch_prober.go:29] interesting pod/e2e-api-a-cd525f-openshift-default-5567b55479-lhtg7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Oct 29 22:58:20.164046 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:20.163676 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-cd525f-openshift-default-5567b55479-lhtg7" podUID="3616f622-b298-42ce-bf47-80e53935b0f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Oct 29 22:58:21.163274 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:21.163200 5057 patch_prober.go:29] interesting pod/e2e-api-a-cd525f-openshift-default-5567b55479-lhtg7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Oct 29 22:58:21.163274 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:21.163254 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-cd525f-openshift-default-5567b55479-lhtg7" podUID="3616f622-b298-42ce-bf47-80e53935b0f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Oct 29 22:58:21.365427 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:21.365386 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:58:22.167822 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:22.167787 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-cd525f-openshift-default-5567b55479-lhtg7" Oct 29 22:58:22.168312 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:22.168288 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-cd525f-openshift-default-5567b55479-lhtg7" Oct 29 22:58:22.169227 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:22.169203 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-cd525f-openshift-default-5567b55479-lhtg7" Oct 29 22:58:22.431471 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:22.431296 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f3092a4d-7f9bb6f4fd-2kck8"] Oct 29 22:58:22.474313 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:22.474278 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-f3092a4d-7745bbd6ff-vsstw"] Oct 29 22:58:22.591743 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:22.591706 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:58:22.715878 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:22.715842 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:58:23.429988 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:23.429929 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-36befd-8479fb9fcd-vkvl2"] Oct 29 22:58:23.437779 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:23.437751 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-36befd-8479fb9fcd-vkvl2" Oct 29 22:58:23.456673 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:23.455657 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-36befd-8479fb9fcd-vkvl2"] Oct 29 22:58:23.489447 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:23.488460 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-36befd-serving-cert\"" Oct 29 22:58:23.493262 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:23.493225 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a921041d-2670-4930-b375-b8bbdef6d573-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-36befd-8479fb9fcd-vkvl2\" (UID: \"a921041d-2670-4930-b375-b8bbdef6d573\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-36befd-8479fb9fcd-vkvl2" Oct 29 22:58:23.493444 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:23.493290 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-glbrg\" (UniqueName: \"kubernetes.io/projected/a921041d-2670-4930-b375-b8bbdef6d573-kube-api-access-glbrg\") pod \"maas-api-e2e-ipp-a-36befd-8479fb9fcd-vkvl2\" (UID: \"a921041d-2670-4930-b375-b8bbdef6d573\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-36befd-8479fb9fcd-vkvl2" Oct 29 22:58:23.493444 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:23.493381 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a921041d-2670-4930-b375-b8bbdef6d573-metrics-certs\") pod \"maas-api-e2e-ipp-a-36befd-8479fb9fcd-vkvl2\" (UID: \"a921041d-2670-4930-b375-b8bbdef6d573\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-36befd-8479fb9fcd-vkvl2" Oct 29 22:58:23.493580 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:23.493483 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a921041d-2670-4930-b375-b8bbdef6d573-maas-api-tls\") pod \"maas-api-e2e-ipp-a-36befd-8479fb9fcd-vkvl2\" (UID: \"a921041d-2670-4930-b375-b8bbdef6d573\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-36befd-8479fb9fcd-vkvl2" Oct 29 22:58:23.515892 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:23.515846 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:58:23.595980 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:23.595938 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a921041d-2670-4930-b375-b8bbdef6d573-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-36befd-8479fb9fcd-vkvl2\" (UID: \"a921041d-2670-4930-b375-b8bbdef6d573\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-36befd-8479fb9fcd-vkvl2" Oct 29 22:58:23.596202 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:23.596018 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-glbrg\" (UniqueName: \"kubernetes.io/projected/a921041d-2670-4930-b375-b8bbdef6d573-kube-api-access-glbrg\") pod \"maas-api-e2e-ipp-a-36befd-8479fb9fcd-vkvl2\" (UID: \"a921041d-2670-4930-b375-b8bbdef6d573\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-36befd-8479fb9fcd-vkvl2" Oct 29 22:58:23.596202 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:23.596107 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a921041d-2670-4930-b375-b8bbdef6d573-metrics-certs\") pod \"maas-api-e2e-ipp-a-36befd-8479fb9fcd-vkvl2\" (UID: \"a921041d-2670-4930-b375-b8bbdef6d573\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-36befd-8479fb9fcd-vkvl2" Oct 29 22:58:23.596320 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:23.596231 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a921041d-2670-4930-b375-b8bbdef6d573-maas-api-tls\") pod \"maas-api-e2e-ipp-a-36befd-8479fb9fcd-vkvl2\" (UID: \"a921041d-2670-4930-b375-b8bbdef6d573\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-36befd-8479fb9fcd-vkvl2" Oct 29 22:58:23.597367 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:23.597288 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a921041d-2670-4930-b375-b8bbdef6d573-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-36befd-8479fb9fcd-vkvl2\" (UID: \"a921041d-2670-4930-b375-b8bbdef6d573\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-36befd-8479fb9fcd-vkvl2" Oct 29 22:58:23.601922 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:23.601897 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a921041d-2670-4930-b375-b8bbdef6d573-metrics-certs\") pod \"maas-api-e2e-ipp-a-36befd-8479fb9fcd-vkvl2\" (UID: \"a921041d-2670-4930-b375-b8bbdef6d573\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-36befd-8479fb9fcd-vkvl2" Oct 29 22:58:23.602683 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:23.602657 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a921041d-2670-4930-b375-b8bbdef6d573-maas-api-tls\") pod \"maas-api-e2e-ipp-a-36befd-8479fb9fcd-vkvl2\" (UID: \"a921041d-2670-4930-b375-b8bbdef6d573\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-36befd-8479fb9fcd-vkvl2" Oct 29 22:58:23.624506 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:23.620030 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-glbrg\" (UniqueName: \"kubernetes.io/projected/a921041d-2670-4930-b375-b8bbdef6d573-kube-api-access-glbrg\") pod \"maas-api-e2e-ipp-a-36befd-8479fb9fcd-vkvl2\" (UID: \"a921041d-2670-4930-b375-b8bbdef6d573\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-36befd-8479fb9fcd-vkvl2" Oct 29 22:58:23.629160 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:23.629128 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:58:23.741054 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:23.741017 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:58:23.771689 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:23.771652 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-36befd-8479fb9fcd-vkvl2" Oct 29 22:58:24.099149 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:58:24.099106 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda921041d_2670_4930_b375_b8bbdef6d573.slice/crio-cffa743933dec6847a375cf36c502f4ff1b61382aa3fb66edf211cd8a7bde820 WatchSource:0}: Error finding container cffa743933dec6847a375cf36c502f4ff1b61382aa3fb66edf211cd8a7bde820: Status 404 returned error can't find the container with id cffa743933dec6847a375cf36c502f4ff1b61382aa3fb66edf211cd8a7bde820 Oct 29 22:58:24.099492 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:24.099466 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-36befd-8479fb9fcd-vkvl2"] Oct 29 22:58:25.134428 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:25.134379 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-36befd-8479fb9fcd-vkvl2" event={"ID":"a921041d-2670-4930-b375-b8bbdef6d573","Type":"ContainerStarted","Data":"d873dea07ad065016612b8fec58aa99bc90bd99c6e3276ccae7240f3143a4cdf"} Oct 29 22:58:25.134854 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:25.134594 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-36befd-8479fb9fcd-vkvl2" Oct 29 22:58:25.134854 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:25.134649 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-36befd-8479fb9fcd-vkvl2" event={"ID":"a921041d-2670-4930-b375-b8bbdef6d573","Type":"ContainerStarted","Data":"cffa743933dec6847a375cf36c502f4ff1b61382aa3fb66edf211cd8a7bde820"} Oct 29 22:58:25.179431 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:25.179372 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-36befd-8479fb9fcd-vkvl2" podStartSLOduration=1.884527309 podStartE2EDuration="2.179355568s" totalImagesPullingTime="294.828259ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:58:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:58:25.17326464 +0000 UTC m=+1485.956478068" watchObservedRunningTime="2026-10-29 22:58:25.179355568 +0000 UTC m=+1485.962569021" Oct 29 22:58:25.786829 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:25.786789 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:58:27.267601 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:27.267563 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:58:27.386427 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:27.381422 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:58:27.433354 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:27.432866 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-f3092a4d-7f9bb6f4fd-2kck8" podUID="e4b2f080-3a05-48e3-b667-ba1e4f93c5d5" containerName="payload-processing" containerID="cri-o://e5078bda077aeb4401c31ac98653e12ce108bfd69d78ca65bb93b3f8e1f0dd7c" gracePeriod=25 Oct 29 22:58:27.474661 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:27.474567 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-f3092a4d-7745bbd6ff-vsstw" podUID="ccfbe3ea-306d-4c9f-be5c-19317c09847d" containerName="payload-pre-processing" containerID="cri-o://72f34310c3ae1e29e98daeaef74a23e95182776d384b62684be8959a6e57e98f" gracePeriod=25 Oct 29 22:58:29.138082 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:29.138045 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:58:30.704623 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:30.704587 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:58:31.149780 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:31.149650 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-36befd-8479fb9fcd-vkvl2" Oct 29 22:58:31.902097 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:31.902060 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:58:31.927077 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:31.927035 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-f3092a4d-7f9bb6f4fd-2kck8" podUID="e4b2f080-3a05-48e3-b667-ba1e4f93c5d5" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 22:58:31.927818 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:31.927781 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-f3092a4d-7745bbd6ff-vsstw" podUID="ccfbe3ea-306d-4c9f-be5c-19317c09847d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 22:58:32.141067 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:32.141037 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:58:33.782820 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:33.782783 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:58:35.006830 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:35.006775 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:58:35.535872 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:35.535834 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:58:35.559588 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:35.559561 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v2tfm"] Oct 29 22:58:37.615967 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:37.615934 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:58:39.191493 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:39.191458 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:58:39.545983 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:39.545952 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-vj6md_aa6a190e-3283-458a-9dfd-2de7cb193fc5/console-operator/2.log" Oct 29 22:58:39.556759 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:39.556737 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-vj6md_aa6a190e-3283-458a-9dfd-2de7cb193fc5/console-operator/2.log" Oct 29 22:58:40.719125 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:40.719083 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:58:41.756920 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:41.756877 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:58:41.870573 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:41.870540 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:58:41.927377 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:41.927306 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-f3092a4d-7f9bb6f4fd-2kck8" podUID="e4b2f080-3a05-48e3-b667-ba1e4f93c5d5" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 22:58:41.927854 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:41.927826 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-f3092a4d-7745bbd6ff-vsstw" podUID="ccfbe3ea-306d-4c9f-be5c-19317c09847d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 22:58:42.264380 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:42.264326 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:58:42.413163 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:42.413060 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:58:43.991588 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:43.991556 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:58:45.489970 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:45.489939 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:58:46.972455 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:46.972405 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:58:47.104074 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:47.104030 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:58:47.994925 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:47.994530 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-f3092a4d-7f9bb6f4fd-2kck8" Oct 29 22:58:48.074493 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:48.074460 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-f3092a4d-7745bbd6ff-vsstw" Oct 29 22:58:48.152923 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:48.152888 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4b2f080-3a05-48e3-b667-ba1e4f93c5d5-kube-api-access-t5klq\" (UniqueName: \"kubernetes.io/projected/e4b2f080-3a05-48e3-b667-ba1e4f93c5d5-kube-api-access-t5klq\") pod \"e4b2f080-3a05-48e3-b667-ba1e4f93c5d5\" (UID: \"e4b2f080-3a05-48e3-b667-ba1e4f93c5d5\") " Oct 29 22:58:48.153176 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:48.153159 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4b2f080-3a05-48e3-b667-ba1e4f93c5d5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4b2f080-3a05-48e3-b667-ba1e4f93c5d5-plugins-config-volume\") pod \"e4b2f080-3a05-48e3-b667-ba1e4f93c5d5\" (UID: \"e4b2f080-3a05-48e3-b667-ba1e4f93c5d5\") " Oct 29 22:58:48.153604 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:48.153576 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4b2f080-3a05-48e3-b667-ba1e4f93c5d5-plugins-config-volume" pod "e4b2f080-3a05-48e3-b667-ba1e4f93c5d5" (UID: "e4b2f080-3a05-48e3-b667-ba1e4f93c5d5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:58:48.154010 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:48.153991 5057 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4b2f080-3a05-48e3-b667-ba1e4f93c5d5-plugins-config-volume\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 22:58:48.155580 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:48.155559 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4b2f080-3a05-48e3-b667-ba1e4f93c5d5-kube-api-access-t5klq" pod "e4b2f080-3a05-48e3-b667-ba1e4f93c5d5" (UID: "e4b2f080-3a05-48e3-b667-ba1e4f93c5d5"). InnerVolumeSpecName "kube-api-access-t5klq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:58:48.254652 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:48.254623 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ccfbe3ea-306d-4c9f-be5c-19317c09847d-kube-api-access-pv76g\" (UniqueName: \"kubernetes.io/projected/ccfbe3ea-306d-4c9f-be5c-19317c09847d-kube-api-access-pv76g\") pod \"ccfbe3ea-306d-4c9f-be5c-19317c09847d\" (UID: \"ccfbe3ea-306d-4c9f-be5c-19317c09847d\") " Oct 29 22:58:48.254817 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:48.254799 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ccfbe3ea-306d-4c9f-be5c-19317c09847d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ccfbe3ea-306d-4c9f-be5c-19317c09847d-plugins-config-volume\") pod \"ccfbe3ea-306d-4c9f-be5c-19317c09847d\" (UID: \"ccfbe3ea-306d-4c9f-be5c-19317c09847d\") " Oct 29 22:58:48.255161 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:48.255133 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ccfbe3ea-306d-4c9f-be5c-19317c09847d-plugins-config-volume" pod "ccfbe3ea-306d-4c9f-be5c-19317c09847d" (UID: "ccfbe3ea-306d-4c9f-be5c-19317c09847d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:58:48.255634 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:48.255615 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t5klq\" (UniqueName: \"kubernetes.io/projected/e4b2f080-3a05-48e3-b667-ba1e4f93c5d5-kube-api-access-t5klq\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 22:58:48.255634 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:48.255636 5057 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ccfbe3ea-306d-4c9f-be5c-19317c09847d-plugins-config-volume\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 22:58:48.256722 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:48.256704 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccfbe3ea-306d-4c9f-be5c-19317c09847d-kube-api-access-pv76g" pod "ccfbe3ea-306d-4c9f-be5c-19317c09847d" (UID: "ccfbe3ea-306d-4c9f-be5c-19317c09847d"). InnerVolumeSpecName "kube-api-access-pv76g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:58:48.357012 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:48.356929 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pv76g\" (UniqueName: \"kubernetes.io/projected/ccfbe3ea-306d-4c9f-be5c-19317c09847d-kube-api-access-pv76g\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 22:58:48.397367 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:48.397297 5057 generic.go:356] "Generic (PLEG): container finished" podID="ccfbe3ea-306d-4c9f-be5c-19317c09847d" containerID="72f34310c3ae1e29e98daeaef74a23e95182776d384b62684be8959a6e57e98f" exitCode=0 Oct 29 22:58:48.397566 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:48.397393 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-f3092a4d-7745bbd6ff-vsstw" Oct 29 22:58:48.397566 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:48.397407 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-f3092a4d-7745bbd6ff-vsstw" event={"ID":"ccfbe3ea-306d-4c9f-be5c-19317c09847d","Type":"ContainerDied","Data":"72f34310c3ae1e29e98daeaef74a23e95182776d384b62684be8959a6e57e98f"} Oct 29 22:58:48.397566 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:48.397470 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-f3092a4d-7745bbd6ff-vsstw" event={"ID":"ccfbe3ea-306d-4c9f-be5c-19317c09847d","Type":"ContainerDied","Data":"724c9a2b57615acabfc5a3c76958418f0e898b48a2e17caa5d87b8da958d22cd"} Oct 29 22:58:48.397566 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:48.397499 5057 scope.go:122] "RemoveContainer" containerID="72f34310c3ae1e29e98daeaef74a23e95182776d384b62684be8959a6e57e98f" Oct 29 22:58:48.399695 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:48.399666 5057 generic.go:356] "Generic (PLEG): container finished" podID="e4b2f080-3a05-48e3-b667-ba1e4f93c5d5" containerID="e5078bda077aeb4401c31ac98653e12ce108bfd69d78ca65bb93b3f8e1f0dd7c" exitCode=0 Oct 29 22:58:48.400777 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:48.400745 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-f3092a4d-7f9bb6f4fd-2kck8" event={"ID":"e4b2f080-3a05-48e3-b667-ba1e4f93c5d5","Type":"ContainerDied","Data":"e5078bda077aeb4401c31ac98653e12ce108bfd69d78ca65bb93b3f8e1f0dd7c"} Oct 29 22:58:48.400893 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:48.400805 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-f3092a4d-7f9bb6f4fd-2kck8" event={"ID":"e4b2f080-3a05-48e3-b667-ba1e4f93c5d5","Type":"ContainerDied","Data":"8bf71b7181d8beb5c24d50342a1d54f33397c351ec77f86cb8c22a9f65f15ffe"} Oct 29 22:58:48.400893 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:48.400883 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-f3092a4d-7f9bb6f4fd-2kck8" Oct 29 22:58:48.412621 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:48.412603 5057 scope.go:122] "RemoveContainer" containerID="72f34310c3ae1e29e98daeaef74a23e95182776d384b62684be8959a6e57e98f" Oct 29 22:58:48.412976 ip-10-0-140-74 kubenswrapper[5057]: E1029 22:58:48.412951 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72f34310c3ae1e29e98daeaef74a23e95182776d384b62684be8959a6e57e98f\": container with ID starting with 72f34310c3ae1e29e98daeaef74a23e95182776d384b62684be8959a6e57e98f not found: ID does not exist" containerID="72f34310c3ae1e29e98daeaef74a23e95182776d384b62684be8959a6e57e98f" Oct 29 22:58:48.413086 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:48.412987 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72f34310c3ae1e29e98daeaef74a23e95182776d384b62684be8959a6e57e98f"} err="failed to get container status \"72f34310c3ae1e29e98daeaef74a23e95182776d384b62684be8959a6e57e98f\": rpc error: code = NotFound desc = could not find container \"72f34310c3ae1e29e98daeaef74a23e95182776d384b62684be8959a6e57e98f\": container with ID starting with 72f34310c3ae1e29e98daeaef74a23e95182776d384b62684be8959a6e57e98f not found: ID does not exist" Oct 29 22:58:48.413086 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:48.413012 5057 scope.go:122] "RemoveContainer" containerID="e5078bda077aeb4401c31ac98653e12ce108bfd69d78ca65bb93b3f8e1f0dd7c" Oct 29 22:58:48.425877 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:48.425854 5057 scope.go:122] "RemoveContainer" containerID="e5078bda077aeb4401c31ac98653e12ce108bfd69d78ca65bb93b3f8e1f0dd7c" Oct 29 22:58:48.426173 ip-10-0-140-74 kubenswrapper[5057]: E1029 22:58:48.426151 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5078bda077aeb4401c31ac98653e12ce108bfd69d78ca65bb93b3f8e1f0dd7c\": container with ID starting with e5078bda077aeb4401c31ac98653e12ce108bfd69d78ca65bb93b3f8e1f0dd7c not found: ID does not exist" containerID="e5078bda077aeb4401c31ac98653e12ce108bfd69d78ca65bb93b3f8e1f0dd7c" Oct 29 22:58:48.426239 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:48.426179 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5078bda077aeb4401c31ac98653e12ce108bfd69d78ca65bb93b3f8e1f0dd7c"} err="failed to get container status \"e5078bda077aeb4401c31ac98653e12ce108bfd69d78ca65bb93b3f8e1f0dd7c\": rpc error: code = NotFound desc = could not find container \"e5078bda077aeb4401c31ac98653e12ce108bfd69d78ca65bb93b3f8e1f0dd7c\": container with ID starting with e5078bda077aeb4401c31ac98653e12ce108bfd69d78ca65bb93b3f8e1f0dd7c not found: ID does not exist" Oct 29 22:58:48.462474 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:48.462449 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f3092a4d-7f9bb6f4fd-2kck8"] Oct 29 22:58:48.494355 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:48.476840 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f3092a4d-7f9bb6f4fd-2kck8"] Oct 29 22:58:48.551057 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:48.551015 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-f3092a4d-7745bbd6ff-vsstw"] Oct 29 22:58:48.558753 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:48.558725 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-f3092a4d-7745bbd6ff-vsstw"] Oct 29 22:58:48.860499 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:48.860466 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:58:49.043803 ip-10-0-140-74 kubenswrapper[5057]: E1029 22:58:49.043761 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d2dnn" containerName="extractor" Oct 29 22:58:49.331525 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:49.331492 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ccfbe3ea-306d-4c9f-be5c-19317c09847d" path="/var/lib/kubelet/pods/ccfbe3ea-306d-4c9f-be5c-19317c09847d/volumes" Oct 29 22:58:49.332111 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:49.332087 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4b2f080-3a05-48e3-b667-ba1e4f93c5d5" path="/var/lib/kubelet/pods/e4b2f080-3a05-48e3-b667-ba1e4f93c5d5/volumes" Oct 29 22:58:50.159165 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:50.159121 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-e4a049-66d6fd5c46-c5sgf"] Oct 29 22:58:50.159733 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:50.159325 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e4a049-66d6fd5c46-c5sgf" podUID="50643c46-b0b4-4a44-9c51-539844e01e44" containerName="maas-api" containerID="cri-o://71b991db9e0eeb1a83f14f3976f2e3b04cfa20fba936562ff5684ae9c2c5af52" gracePeriod=30 Oct 29 22:58:50.219050 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:50.218885 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:58:50.597051 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:50.597007 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e4a049-66d6fd5c46-c5sgf" Oct 29 22:58:50.689399 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:50.688099 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:58:50.722488 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:50.721842 5057 generic.go:356] "Generic (PLEG): container finished" podID="50643c46-b0b4-4a44-9c51-539844e01e44" containerID="71b991db9e0eeb1a83f14f3976f2e3b04cfa20fba936562ff5684ae9c2c5af52" exitCode=0 Oct 29 22:58:50.722488 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:50.722182 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e4a049-66d6fd5c46-c5sgf" event={"ID":"50643c46-b0b4-4a44-9c51-539844e01e44","Type":"ContainerDied","Data":"71b991db9e0eeb1a83f14f3976f2e3b04cfa20fba936562ff5684ae9c2c5af52"} Oct 29 22:58:50.722488 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:50.722288 5057 scope.go:122] "RemoveContainer" containerID="71b991db9e0eeb1a83f14f3976f2e3b04cfa20fba936562ff5684ae9c2c5af52" Oct 29 22:58:50.728837 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:50.728065 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e4a049-66d6fd5c46-c5sgf" Oct 29 22:58:50.800440 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:50.800301 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:58:50.832949 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:50.832918 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50643c46-b0b4-4a44-9c51-539844e01e44-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/50643c46-b0b4-4a44-9c51-539844e01e44-openshift-service-ca\") pod \"50643c46-b0b4-4a44-9c51-539844e01e44\" (UID: \"50643c46-b0b4-4a44-9c51-539844e01e44\") " Oct 29 22:58:50.833116 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:50.833015 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50643c46-b0b4-4a44-9c51-539844e01e44-kube-api-access-msjv9\" (UniqueName: \"kubernetes.io/projected/50643c46-b0b4-4a44-9c51-539844e01e44-kube-api-access-msjv9\") pod \"50643c46-b0b4-4a44-9c51-539844e01e44\" (UID: \"50643c46-b0b4-4a44-9c51-539844e01e44\") " Oct 29 22:58:50.833116 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:50.833059 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/50643c46-b0b4-4a44-9c51-539844e01e44-metrics-certs\" (UniqueName: \"kubernetes.io/secret/50643c46-b0b4-4a44-9c51-539844e01e44-metrics-certs\") pod \"50643c46-b0b4-4a44-9c51-539844e01e44\" (UID: \"50643c46-b0b4-4a44-9c51-539844e01e44\") " Oct 29 22:58:50.833193 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:50.833144 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/50643c46-b0b4-4a44-9c51-539844e01e44-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/50643c46-b0b4-4a44-9c51-539844e01e44-maas-api-tls\") pod \"50643c46-b0b4-4a44-9c51-539844e01e44\" (UID: \"50643c46-b0b4-4a44-9c51-539844e01e44\") " Oct 29 22:58:50.833366 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:50.833318 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50643c46-b0b4-4a44-9c51-539844e01e44-openshift-service-ca" pod "50643c46-b0b4-4a44-9c51-539844e01e44" (UID: "50643c46-b0b4-4a44-9c51-539844e01e44"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:58:50.833919 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:50.833895 5057 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/50643c46-b0b4-4a44-9c51-539844e01e44-openshift-service-ca\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 22:58:50.835366 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:50.835260 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50643c46-b0b4-4a44-9c51-539844e01e44-maas-api-tls" pod "50643c46-b0b4-4a44-9c51-539844e01e44" (UID: "50643c46-b0b4-4a44-9c51-539844e01e44"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:58:50.835366 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:50.835312 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50643c46-b0b4-4a44-9c51-539844e01e44-metrics-certs" pod "50643c46-b0b4-4a44-9c51-539844e01e44" (UID: "50643c46-b0b4-4a44-9c51-539844e01e44"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:58:50.835492 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:50.835380 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50643c46-b0b4-4a44-9c51-539844e01e44-kube-api-access-msjv9" pod "50643c46-b0b4-4a44-9c51-539844e01e44" (UID: "50643c46-b0b4-4a44-9c51-539844e01e44"). InnerVolumeSpecName "kube-api-access-msjv9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:58:50.939021 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:50.938974 5057 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/50643c46-b0b4-4a44-9c51-539844e01e44-maas-api-tls\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 22:58:50.939248 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:50.939229 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-msjv9\" (UniqueName: \"kubernetes.io/projected/50643c46-b0b4-4a44-9c51-539844e01e44-kube-api-access-msjv9\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 22:58:50.939405 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:50.939389 5057 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50643c46-b0b4-4a44-9c51-539844e01e44-metrics-certs\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 22:58:51.107679 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:51.100477 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:58:51.174225 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:51.174188 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-e4a049-66d6fd5c46-c5sgf"] Oct 29 22:58:51.190495 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:51.190188 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-e4a049-66d6fd5c46-c5sgf"] Oct 29 22:58:51.217697 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:51.217654 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:58:51.351215 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:51.351177 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:58:51.387401 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:51.367958 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50643c46-b0b4-4a44-9c51-539844e01e44" path="/var/lib/kubelet/pods/50643c46-b0b4-4a44-9c51-539844e01e44/volumes" Oct 29 22:58:51.910281 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:51.910245 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:58:53.626090 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:53.626057 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:58:55.324004 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:55.323969 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:58:56.962729 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:56.962676 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:58:58.053033 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:58.052984 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:58:58.177741 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:58.177705 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:58:58.191730 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:58.191698 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f3092a4d-c79c57cf-wzjtm"] Oct 29 22:58:58.193056 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:58.193015 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f3092a4d-c79c57cf-wzjtm" podUID="979cc0cb-b14a-4c0a-84b1-d4fe1cad4279" containerName="maas-api" containerID="cri-o://590d522f203a0a01b1257aea23379cde3af0fea2bf8446b01135fb731a460798" gracePeriod=30 Oct 29 22:58:58.290974 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:58.290944 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:58:58.362766 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:58.362413 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-h7w59"] Oct 29 22:58:58.362766 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:58.362682 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-h7w59" podUID="4c92c699-743e-4118-9887-d6ecc1568b88" containerName="istio-proxy" containerID="cri-o://aab1b00be86f3010ff586c6b1140eecbc0b4cf03f3ef596c066765576440edda" gracePeriod=30 Oct 29 22:58:58.419379 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:58.414955 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:58:58.678354 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:58.678301 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f3092a4d-c79c57cf-wzjtm" Oct 29 22:58:58.765116 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:58.763047 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/979cc0cb-b14a-4c0a-84b1-d4fe1cad4279-kube-api-access-fbhkb\" (UniqueName: \"kubernetes.io/projected/979cc0cb-b14a-4c0a-84b1-d4fe1cad4279-kube-api-access-fbhkb\") pod \"979cc0cb-b14a-4c0a-84b1-d4fe1cad4279\" (UID: \"979cc0cb-b14a-4c0a-84b1-d4fe1cad4279\") " Oct 29 22:58:58.765116 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:58.763140 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/979cc0cb-b14a-4c0a-84b1-d4fe1cad4279-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/979cc0cb-b14a-4c0a-84b1-d4fe1cad4279-maas-api-tls\") pod \"979cc0cb-b14a-4c0a-84b1-d4fe1cad4279\" (UID: \"979cc0cb-b14a-4c0a-84b1-d4fe1cad4279\") " Oct 29 22:58:58.765116 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:58.763188 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/979cc0cb-b14a-4c0a-84b1-d4fe1cad4279-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/979cc0cb-b14a-4c0a-84b1-d4fe1cad4279-openshift-service-ca\") pod \"979cc0cb-b14a-4c0a-84b1-d4fe1cad4279\" (UID: \"979cc0cb-b14a-4c0a-84b1-d4fe1cad4279\") " Oct 29 22:58:58.765116 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:58.763211 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/979cc0cb-b14a-4c0a-84b1-d4fe1cad4279-metrics-certs\" (UniqueName: \"kubernetes.io/secret/979cc0cb-b14a-4c0a-84b1-d4fe1cad4279-metrics-certs\") pod \"979cc0cb-b14a-4c0a-84b1-d4fe1cad4279\" (UID: \"979cc0cb-b14a-4c0a-84b1-d4fe1cad4279\") " Oct 29 22:58:58.765116 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:58.764225 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/979cc0cb-b14a-4c0a-84b1-d4fe1cad4279-openshift-service-ca" pod "979cc0cb-b14a-4c0a-84b1-d4fe1cad4279" (UID: "979cc0cb-b14a-4c0a-84b1-d4fe1cad4279"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:58:58.765116 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:58.764700 5057 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/979cc0cb-b14a-4c0a-84b1-d4fe1cad4279-openshift-service-ca\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 22:58:58.775245 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:58.775189 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:58:58.793607 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:58.788653 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/979cc0cb-b14a-4c0a-84b1-d4fe1cad4279-kube-api-access-fbhkb" pod "979cc0cb-b14a-4c0a-84b1-d4fe1cad4279" (UID: "979cc0cb-b14a-4c0a-84b1-d4fe1cad4279"). InnerVolumeSpecName "kube-api-access-fbhkb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:58:58.793607 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:58.793515 5057 generic.go:356] "Generic (PLEG): container finished" podID="979cc0cb-b14a-4c0a-84b1-d4fe1cad4279" containerID="590d522f203a0a01b1257aea23379cde3af0fea2bf8446b01135fb731a460798" exitCode=0 Oct 29 22:58:58.793826 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:58.793805 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f3092a4d-c79c57cf-wzjtm" event={"ID":"979cc0cb-b14a-4c0a-84b1-d4fe1cad4279","Type":"ContainerDied","Data":"590d522f203a0a01b1257aea23379cde3af0fea2bf8446b01135fb731a460798"} Oct 29 22:58:58.793903 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:58.793868 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f3092a4d-c79c57cf-wzjtm" event={"ID":"979cc0cb-b14a-4c0a-84b1-d4fe1cad4279","Type":"ContainerDied","Data":"e3a2e1fb495542b831201f69c12ab945dac6abecddc98e628fb8204f7363b26b"} Oct 29 22:58:58.793962 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:58.793900 5057 scope.go:122] "RemoveContainer" containerID="590d522f203a0a01b1257aea23379cde3af0fea2bf8446b01135fb731a460798" Oct 29 22:58:58.796379 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:58.794055 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f3092a4d-c79c57cf-wzjtm" Oct 29 22:58:58.799088 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:58.798759 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/979cc0cb-b14a-4c0a-84b1-d4fe1cad4279-maas-api-tls" pod "979cc0cb-b14a-4c0a-84b1-d4fe1cad4279" (UID: "979cc0cb-b14a-4c0a-84b1-d4fe1cad4279"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:58:58.804985 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:58.804953 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/979cc0cb-b14a-4c0a-84b1-d4fe1cad4279-metrics-certs" pod "979cc0cb-b14a-4c0a-84b1-d4fe1cad4279" (UID: "979cc0cb-b14a-4c0a-84b1-d4fe1cad4279"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:58:58.827195 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:58.827169 5057 scope.go:122] "RemoveContainer" containerID="590d522f203a0a01b1257aea23379cde3af0fea2bf8446b01135fb731a460798" Oct 29 22:58:58.827928 ip-10-0-140-74 kubenswrapper[5057]: E1029 22:58:58.827850 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"590d522f203a0a01b1257aea23379cde3af0fea2bf8446b01135fb731a460798\": container with ID starting with 590d522f203a0a01b1257aea23379cde3af0fea2bf8446b01135fb731a460798 not found: ID does not exist" containerID="590d522f203a0a01b1257aea23379cde3af0fea2bf8446b01135fb731a460798" Oct 29 22:58:58.827928 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:58.827889 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"590d522f203a0a01b1257aea23379cde3af0fea2bf8446b01135fb731a460798"} err="failed to get container status \"590d522f203a0a01b1257aea23379cde3af0fea2bf8446b01135fb731a460798\": rpc error: code = NotFound desc = could not find container \"590d522f203a0a01b1257aea23379cde3af0fea2bf8446b01135fb731a460798\": container with ID starting with 590d522f203a0a01b1257aea23379cde3af0fea2bf8446b01135fb731a460798 not found: ID does not exist" Oct 29 22:58:58.868415 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:58.867463 5057 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/979cc0cb-b14a-4c0a-84b1-d4fe1cad4279-maas-api-tls\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 22:58:58.868415 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:58.867501 5057 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/979cc0cb-b14a-4c0a-84b1-d4fe1cad4279-metrics-certs\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 22:58:58.868415 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:58.867516 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fbhkb\" (UniqueName: \"kubernetes.io/projected/979cc0cb-b14a-4c0a-84b1-d4fe1cad4279-kube-api-access-fbhkb\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 22:58:58.902186 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:58.902150 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a1d1da-7fd76d4dcf-bnwgx"] Oct 29 22:58:58.931126 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:58.931077 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:58:59.047540 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:59.047503 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:58:59.144813 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:59.144718 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f3092a4d-c79c57cf-wzjtm"] Oct 29 22:58:59.150121 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:59.150093 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f3092a4d-c79c57cf-wzjtm"] Oct 29 22:58:59.169003 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:59.168966 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:58:59.298011 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:59.297958 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:58:59.358414 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:58:59.358379 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="979cc0cb-b14a-4c0a-84b1-d4fe1cad4279" path="/var/lib/kubelet/pods/979cc0cb-b14a-4c0a-84b1-d4fe1cad4279/volumes" Oct 29 22:59:00.011356 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:00.008251 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:59:02.289691 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:02.289633 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:59:03.401690 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:03.401662 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:59:03.703588 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:03.703562 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-h7w59" Oct 29 22:59:03.744415 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:03.744382 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c92c699-743e-4118-9887-d6ecc1568b88-istio-token\" (UniqueName: \"kubernetes.io/projected/4c92c699-743e-4118-9887-d6ecc1568b88-istio-token\") pod \"4c92c699-743e-4118-9887-d6ecc1568b88\" (UID: \"4c92c699-743e-4118-9887-d6ecc1568b88\") " Oct 29 22:59:03.744553 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:03.744450 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c92c699-743e-4118-9887-d6ecc1568b88-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c92c699-743e-4118-9887-d6ecc1568b88-workload-certs\") pod \"4c92c699-743e-4118-9887-d6ecc1568b88\" (UID: \"4c92c699-743e-4118-9887-d6ecc1568b88\") " Oct 29 22:59:03.744553 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:03.744521 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c92c699-743e-4118-9887-d6ecc1568b88-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c92c699-743e-4118-9887-d6ecc1568b88-credential-socket\") pod \"4c92c699-743e-4118-9887-d6ecc1568b88\" (UID: \"4c92c699-743e-4118-9887-d6ecc1568b88\") " Oct 29 22:59:03.744658 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:03.744644 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c92c699-743e-4118-9887-d6ecc1568b88-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c92c699-743e-4118-9887-d6ecc1568b88-istiod-ca-cert\") pod \"4c92c699-743e-4118-9887-d6ecc1568b88\" (UID: \"4c92c699-743e-4118-9887-d6ecc1568b88\") " Oct 29 22:59:03.744764 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:03.744749 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c92c699-743e-4118-9887-d6ecc1568b88-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c92c699-743e-4118-9887-d6ecc1568b88-istio-data\") pod \"4c92c699-743e-4118-9887-d6ecc1568b88\" (UID: \"4c92c699-743e-4118-9887-d6ecc1568b88\") " Oct 29 22:59:03.744832 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:03.744745 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c92c699-743e-4118-9887-d6ecc1568b88-workload-certs" pod "4c92c699-743e-4118-9887-d6ecc1568b88" (UID: "4c92c699-743e-4118-9887-d6ecc1568b88"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:59:03.744832 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:03.744794 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4c92c699-743e-4118-9887-d6ecc1568b88-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c92c699-743e-4118-9887-d6ecc1568b88-istio-podinfo\") pod \"4c92c699-743e-4118-9887-d6ecc1568b88\" (UID: \"4c92c699-743e-4118-9887-d6ecc1568b88\") " Oct 29 22:59:03.744942 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:03.744789 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c92c699-743e-4118-9887-d6ecc1568b88-credential-socket" pod "4c92c699-743e-4118-9887-d6ecc1568b88" (UID: "4c92c699-743e-4118-9887-d6ecc1568b88"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:59:03.744994 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:03.744934 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c92c699-743e-4118-9887-d6ecc1568b88-istio-data" pod "4c92c699-743e-4118-9887-d6ecc1568b88" (UID: "4c92c699-743e-4118-9887-d6ecc1568b88"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:59:03.744994 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:03.744972 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c92c699-743e-4118-9887-d6ecc1568b88-kube-api-access-4wxtn\" (UniqueName: \"kubernetes.io/projected/4c92c699-743e-4118-9887-d6ecc1568b88-kube-api-access-4wxtn\") pod \"4c92c699-743e-4118-9887-d6ecc1568b88\" (UID: \"4c92c699-743e-4118-9887-d6ecc1568b88\") " Oct 29 22:59:03.745099 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:03.745015 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c92c699-743e-4118-9887-d6ecc1568b88-istiod-ca-cert" pod "4c92c699-743e-4118-9887-d6ecc1568b88" (UID: "4c92c699-743e-4118-9887-d6ecc1568b88"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:59:03.745099 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:03.745030 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c92c699-743e-4118-9887-d6ecc1568b88-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c92c699-743e-4118-9887-d6ecc1568b88-workload-socket\") pod \"4c92c699-743e-4118-9887-d6ecc1568b88\" (UID: \"4c92c699-743e-4118-9887-d6ecc1568b88\") " Oct 29 22:59:03.745099 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:03.745068 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c92c699-743e-4118-9887-d6ecc1568b88-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c92c699-743e-4118-9887-d6ecc1568b88-istio-envoy\") pod \"4c92c699-743e-4118-9887-d6ecc1568b88\" (UID: \"4c92c699-743e-4118-9887-d6ecc1568b88\") " Oct 29 22:59:03.745300 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:03.745279 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c92c699-743e-4118-9887-d6ecc1568b88-workload-socket" pod "4c92c699-743e-4118-9887-d6ecc1568b88" (UID: "4c92c699-743e-4118-9887-d6ecc1568b88"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:59:03.745934 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:03.745909 5057 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c92c699-743e-4118-9887-d6ecc1568b88-workload-socket\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 22:59:03.746062 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:03.745941 5057 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c92c699-743e-4118-9887-d6ecc1568b88-workload-certs\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 22:59:03.746062 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:03.745959 5057 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c92c699-743e-4118-9887-d6ecc1568b88-credential-socket\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 22:59:03.746062 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:03.745976 5057 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c92c699-743e-4118-9887-d6ecc1568b88-istiod-ca-cert\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 22:59:03.746062 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:03.745990 5057 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c92c699-743e-4118-9887-d6ecc1568b88-istio-data\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 22:59:03.746889 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:03.746862 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c92c699-743e-4118-9887-d6ecc1568b88-istio-token" pod "4c92c699-743e-4118-9887-d6ecc1568b88" (UID: "4c92c699-743e-4118-9887-d6ecc1568b88"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:59:03.747301 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:03.747283 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4c92c699-743e-4118-9887-d6ecc1568b88-istio-podinfo" pod "4c92c699-743e-4118-9887-d6ecc1568b88" (UID: "4c92c699-743e-4118-9887-d6ecc1568b88"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:59:03.747394 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:03.747297 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c92c699-743e-4118-9887-d6ecc1568b88-istio-envoy" pod "4c92c699-743e-4118-9887-d6ecc1568b88" (UID: "4c92c699-743e-4118-9887-d6ecc1568b88"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:59:03.747660 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:03.747632 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c92c699-743e-4118-9887-d6ecc1568b88-kube-api-access-4wxtn" pod "4c92c699-743e-4118-9887-d6ecc1568b88" (UID: "4c92c699-743e-4118-9887-d6ecc1568b88"). InnerVolumeSpecName "kube-api-access-4wxtn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:59:03.847426 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:03.847395 5057 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c92c699-743e-4118-9887-d6ecc1568b88-istio-podinfo\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 22:59:03.847426 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:03.847423 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4wxtn\" (UniqueName: \"kubernetes.io/projected/4c92c699-743e-4118-9887-d6ecc1568b88-kube-api-access-4wxtn\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 22:59:03.847426 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:03.847433 5057 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c92c699-743e-4118-9887-d6ecc1568b88-istio-envoy\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 22:59:03.847696 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:03.847442 5057 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c92c699-743e-4118-9887-d6ecc1568b88-istio-token\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 22:59:03.863992 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:03.863964 5057 generic.go:356] "Generic (PLEG): container finished" podID="4c92c699-743e-4118-9887-d6ecc1568b88" containerID="aab1b00be86f3010ff586c6b1140eecbc0b4cf03f3ef596c066765576440edda" exitCode=0 Oct 29 22:59:03.864161 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:03.864024 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-h7w59" Oct 29 22:59:03.864161 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:03.864052 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-h7w59" event={"ID":"4c92c699-743e-4118-9887-d6ecc1568b88","Type":"ContainerDied","Data":"aab1b00be86f3010ff586c6b1140eecbc0b4cf03f3ef596c066765576440edda"} Oct 29 22:59:03.864161 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:03.864116 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-h7w59" event={"ID":"4c92c699-743e-4118-9887-d6ecc1568b88","Type":"ContainerDied","Data":"3881ad7b4d1045e525acf78ed8ce1712bac6a967496ff96a9ba43a25446bf954"} Oct 29 22:59:03.864161 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:03.864139 5057 scope.go:122] "RemoveContainer" containerID="aab1b00be86f3010ff586c6b1140eecbc0b4cf03f3ef596c066765576440edda" Oct 29 22:59:03.877154 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:03.877131 5057 scope.go:122] "RemoveContainer" containerID="aab1b00be86f3010ff586c6b1140eecbc0b4cf03f3ef596c066765576440edda" Oct 29 22:59:03.877489 ip-10-0-140-74 kubenswrapper[5057]: E1029 22:59:03.877462 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aab1b00be86f3010ff586c6b1140eecbc0b4cf03f3ef596c066765576440edda\": container with ID starting with aab1b00be86f3010ff586c6b1140eecbc0b4cf03f3ef596c066765576440edda not found: ID does not exist" containerID="aab1b00be86f3010ff586c6b1140eecbc0b4cf03f3ef596c066765576440edda" Oct 29 22:59:03.877573 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:03.877497 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aab1b00be86f3010ff586c6b1140eecbc0b4cf03f3ef596c066765576440edda"} err="failed to get container status \"aab1b00be86f3010ff586c6b1140eecbc0b4cf03f3ef596c066765576440edda\": rpc error: code = NotFound desc = could not find container \"aab1b00be86f3010ff586c6b1140eecbc0b4cf03f3ef596c066765576440edda\": container with ID starting with aab1b00be86f3010ff586c6b1140eecbc0b4cf03f3ef596c066765576440edda not found: ID does not exist" Oct 29 22:59:03.903479 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:03.903439 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a1d1da-7fd76d4dcf-bnwgx" podUID="dd8950b3-7cd5-4759-866e-1352ef928f64" containerName="payload-pre-processing" containerID="cri-o://2e0224f4770b4db0f1572e69f086e2b0b4b0297850fabf753a710f036849c376" gracePeriod=25 Oct 29 22:59:03.917654 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:03.917615 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-h7w59"] Oct 29 22:59:03.922484 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:03.922458 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-h7w59"] Oct 29 22:59:04.188851 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:04.188826 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a1d1da-7fd76d4dcf-bnwgx" Oct 29 22:59:04.282790 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:04.282748 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd8950b3-7cd5-4759-866e-1352ef928f64-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd8950b3-7cd5-4759-866e-1352ef928f64-plugins-config-volume\") pod \"dd8950b3-7cd5-4759-866e-1352ef928f64\" (UID: \"dd8950b3-7cd5-4759-866e-1352ef928f64\") " Oct 29 22:59:04.282981 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:04.282808 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd8950b3-7cd5-4759-866e-1352ef928f64-kube-api-access-n6lqh\" (UniqueName: \"kubernetes.io/projected/dd8950b3-7cd5-4759-866e-1352ef928f64-kube-api-access-n6lqh\") pod \"dd8950b3-7cd5-4759-866e-1352ef928f64\" (UID: \"dd8950b3-7cd5-4759-866e-1352ef928f64\") " Oct 29 22:59:04.284473 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:04.284440 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd8950b3-7cd5-4759-866e-1352ef928f64-plugins-config-volume" pod "dd8950b3-7cd5-4759-866e-1352ef928f64" (UID: "dd8950b3-7cd5-4759-866e-1352ef928f64"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:59:04.286635 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:04.286602 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd8950b3-7cd5-4759-866e-1352ef928f64-kube-api-access-n6lqh" pod "dd8950b3-7cd5-4759-866e-1352ef928f64" (UID: "dd8950b3-7cd5-4759-866e-1352ef928f64"). InnerVolumeSpecName "kube-api-access-n6lqh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:59:04.357079 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:04.357037 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:59:04.386812 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:04.386780 5057 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd8950b3-7cd5-4759-866e-1352ef928f64-plugins-config-volume\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 22:59:04.386812 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:04.386811 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n6lqh\" (UniqueName: \"kubernetes.io/projected/dd8950b3-7cd5-4759-866e-1352ef928f64-kube-api-access-n6lqh\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 22:59:04.878686 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:04.878646 5057 generic.go:356] "Generic (PLEG): container finished" podID="dd8950b3-7cd5-4759-866e-1352ef928f64" containerID="2e0224f4770b4db0f1572e69f086e2b0b4b0297850fabf753a710f036849c376" exitCode=0 Oct 29 22:59:04.879188 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:04.878731 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a1d1da-7fd76d4dcf-bnwgx" event={"ID":"dd8950b3-7cd5-4759-866e-1352ef928f64","Type":"ContainerDied","Data":"2e0224f4770b4db0f1572e69f086e2b0b4b0297850fabf753a710f036849c376"} Oct 29 22:59:04.879188 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:04.878829 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a1d1da-7fd76d4dcf-bnwgx" event={"ID":"dd8950b3-7cd5-4759-866e-1352ef928f64","Type":"ContainerDied","Data":"32d7033642bee43a4b0c7bd147738fd17959563524de7815b0c432e5a6d00232"} Oct 29 22:59:04.879188 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:04.878863 5057 scope.go:122] "RemoveContainer" containerID="2e0224f4770b4db0f1572e69f086e2b0b4b0297850fabf753a710f036849c376" Oct 29 22:59:04.879478 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:04.879458 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a1d1da-7fd76d4dcf-bnwgx" Oct 29 22:59:04.897958 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:04.897936 5057 scope.go:122] "RemoveContainer" containerID="2e0224f4770b4db0f1572e69f086e2b0b4b0297850fabf753a710f036849c376" Oct 29 22:59:04.898414 ip-10-0-140-74 kubenswrapper[5057]: E1029 22:59:04.898382 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e0224f4770b4db0f1572e69f086e2b0b4b0297850fabf753a710f036849c376\": container with ID starting with 2e0224f4770b4db0f1572e69f086e2b0b4b0297850fabf753a710f036849c376 not found: ID does not exist" containerID="2e0224f4770b4db0f1572e69f086e2b0b4b0297850fabf753a710f036849c376" Oct 29 22:59:04.898540 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:04.898424 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e0224f4770b4db0f1572e69f086e2b0b4b0297850fabf753a710f036849c376"} err="failed to get container status \"2e0224f4770b4db0f1572e69f086e2b0b4b0297850fabf753a710f036849c376\": rpc error: code = NotFound desc = could not find container \"2e0224f4770b4db0f1572e69f086e2b0b4b0297850fabf753a710f036849c376\": container with ID starting with 2e0224f4770b4db0f1572e69f086e2b0b4b0297850fabf753a710f036849c376 not found: ID does not exist" Oct 29 22:59:04.932932 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:04.932905 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a1d1da-7fd76d4dcf-bnwgx"] Oct 29 22:59:04.941849 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:04.941815 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a1d1da-7fd76d4dcf-bnwgx"] Oct 29 22:59:05.331159 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:05.331129 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c92c699-743e-4118-9887-d6ecc1568b88" path="/var/lib/kubelet/pods/4c92c699-743e-4118-9887-d6ecc1568b88/volumes" Oct 29 22:59:05.331627 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:05.331612 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd8950b3-7cd5-4759-866e-1352ef928f64" path="/var/lib/kubelet/pods/dd8950b3-7cd5-4759-866e-1352ef928f64/volumes" Oct 29 22:59:06.118409 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:06.118311 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:59:07.815253 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:07.815214 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:59:09.309810 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:09.309763 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:59:10.204534 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:10.204495 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:59:10.319105 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:10.319027 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:59:10.428993 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:10.428956 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:59:10.662134 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:10.662060 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:59:10.778281 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:10.778240 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:59:10.982466 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:10.982429 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:59:11.232363 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:11.222835 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:59:11.536158 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:11.536117 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:59:12.325265 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:12.325208 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:59:12.438668 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:12.438635 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:59:13.741270 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:13.741233 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:59:14.081550 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:14.081465 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:59:15.522632 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:15.522597 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:59:15.642591 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:15.642323 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:59:15.757782 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:15.757747 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:59:16.187658 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:16.187612 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:59:16.657148 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:16.657118 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:59:17.383047 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:17.383006 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:59:18.072244 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:18.072207 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:59:18.194468 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:18.194431 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:59:18.479080 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:18.479045 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:59:18.604247 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:18.604213 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:59:18.657522 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:18.657489 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-49dbb5-c95485467-m2cbg"] Oct 29 22:59:18.658432 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:18.658412 5057 cpu_manager.go:414] "containerMap: removing container" podUID="979cc0cb-b14a-4c0a-84b1-d4fe1cad4279" containerName="maas-api" Oct 29 22:59:18.658488 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:18.658433 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="979cc0cb-b14a-4c0a-84b1-d4fe1cad4279" containerName="maas-api" Oct 29 22:59:18.658488 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:18.658450 5057 cpu_manager.go:414] "containerMap: removing container" podUID="ccfbe3ea-306d-4c9f-be5c-19317c09847d" containerName="payload-pre-processing" Oct 29 22:59:18.658488 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:18.658456 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ccfbe3ea-306d-4c9f-be5c-19317c09847d" containerName="payload-pre-processing" Oct 29 22:59:18.658488 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:18.658477 5057 cpu_manager.go:414] "containerMap: removing container" podUID="dd8950b3-7cd5-4759-866e-1352ef928f64" containerName="payload-pre-processing" Oct 29 22:59:18.658488 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:18.658482 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd8950b3-7cd5-4759-866e-1352ef928f64" containerName="payload-pre-processing" Oct 29 22:59:18.658643 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:18.658524 5057 cpu_manager.go:414] "containerMap: removing container" podUID="4c92c699-743e-4118-9887-d6ecc1568b88" containerName="istio-proxy" Oct 29 22:59:18.658643 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:18.658530 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c92c699-743e-4118-9887-d6ecc1568b88" containerName="istio-proxy" Oct 29 22:59:18.658643 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:18.658554 5057 cpu_manager.go:414] "containerMap: removing container" podUID="50643c46-b0b4-4a44-9c51-539844e01e44" containerName="maas-api" Oct 29 22:59:18.658643 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:18.658560 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50643c46-b0b4-4a44-9c51-539844e01e44" containerName="maas-api" Oct 29 22:59:18.658643 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:18.658578 5057 cpu_manager.go:414] "containerMap: removing container" podUID="e4b2f080-3a05-48e3-b667-ba1e4f93c5d5" containerName="payload-processing" Oct 29 22:59:18.658643 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:18.658583 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4b2f080-3a05-48e3-b667-ba1e4f93c5d5" containerName="payload-processing" Oct 29 22:59:18.658829 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:18.658715 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c92c699-743e-4118-9887-d6ecc1568b88" containerName="istio-proxy" Oct 29 22:59:18.658829 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:18.658728 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="50643c46-b0b4-4a44-9c51-539844e01e44" containerName="maas-api" Oct 29 22:59:18.658829 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:18.658738 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4b2f080-3a05-48e3-b667-ba1e4f93c5d5" containerName="payload-processing" Oct 29 22:59:18.658829 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:18.658747 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="979cc0cb-b14a-4c0a-84b1-d4fe1cad4279" containerName="maas-api" Oct 29 22:59:18.658829 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:18.658754 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="ccfbe3ea-306d-4c9f-be5c-19317c09847d" containerName="payload-pre-processing" Oct 29 22:59:18.658829 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:18.658761 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd8950b3-7cd5-4759-866e-1352ef928f64" containerName="payload-pre-processing" Oct 29 22:59:18.664831 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:18.664810 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-49dbb5-c95485467-m2cbg" Oct 29 22:59:18.682231 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:18.682204 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-49dbb5-c95485467-m2cbg"] Oct 29 22:59:18.685021 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:18.685002 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-49dbb5-dockercfg-rlxdv\"" Oct 29 22:59:18.685141 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:18.685035 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-49dbb5\"" Oct 29 22:59:18.732153 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:18.732069 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-49dbb5-5777bd86cd-jfgz7"] Oct 29 22:59:18.755191 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:18.753768 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-49dbb5-5777bd86cd-jfgz7" Oct 29 22:59:18.838486 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:18.838402 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-49dbb5-5777bd86cd-jfgz7"] Oct 29 22:59:18.873431 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:18.873388 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mnblf\" (UniqueName: \"kubernetes.io/projected/b289af8a-e8b1-4f7e-a875-e78b4d8e89a8-kube-api-access-mnblf\") pod \"payload-processing-e2e-api-a-49dbb5-5777bd86cd-jfgz7\" (UID: \"b289af8a-e8b1-4f7e-a875-e78b4d8e89a8\") " pod="openshift-ingress/payload-processing-e2e-api-a-49dbb5-5777bd86cd-jfgz7" Oct 29 22:59:18.873628 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:18.873511 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqm28\" (UniqueName: \"kubernetes.io/projected/e76f405f-e6b4-4645-97ec-a90a9428c826-kube-api-access-kqm28\") pod \"payload-pre-processing-e2e-api-a-49dbb5-c95485467-m2cbg\" (UID: \"e76f405f-e6b4-4645-97ec-a90a9428c826\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-49dbb5-c95485467-m2cbg" Oct 29 22:59:18.873628 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:18.873537 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b289af8a-e8b1-4f7e-a875-e78b4d8e89a8-plugins-config-volume\") pod \"payload-processing-e2e-api-a-49dbb5-5777bd86cd-jfgz7\" (UID: \"b289af8a-e8b1-4f7e-a875-e78b4d8e89a8\") " pod="openshift-ingress/payload-processing-e2e-api-a-49dbb5-5777bd86cd-jfgz7" Oct 29 22:59:18.873810 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:18.873787 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e76f405f-e6b4-4645-97ec-a90a9428c826-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-49dbb5-c95485467-m2cbg\" (UID: \"e76f405f-e6b4-4645-97ec-a90a9428c826\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-49dbb5-c95485467-m2cbg" Oct 29 22:59:18.975436 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:18.975398 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnblf\" (UniqueName: \"kubernetes.io/projected/b289af8a-e8b1-4f7e-a875-e78b4d8e89a8-kube-api-access-mnblf\") pod \"payload-processing-e2e-api-a-49dbb5-5777bd86cd-jfgz7\" (UID: \"b289af8a-e8b1-4f7e-a875-e78b4d8e89a8\") " pod="openshift-ingress/payload-processing-e2e-api-a-49dbb5-5777bd86cd-jfgz7" Oct 29 22:59:18.975615 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:18.975527 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqm28\" (UniqueName: \"kubernetes.io/projected/e76f405f-e6b4-4645-97ec-a90a9428c826-kube-api-access-kqm28\") pod \"payload-pre-processing-e2e-api-a-49dbb5-c95485467-m2cbg\" (UID: \"e76f405f-e6b4-4645-97ec-a90a9428c826\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-49dbb5-c95485467-m2cbg" Oct 29 22:59:18.975615 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:18.975555 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b289af8a-e8b1-4f7e-a875-e78b4d8e89a8-plugins-config-volume\") pod \"payload-processing-e2e-api-a-49dbb5-5777bd86cd-jfgz7\" (UID: \"b289af8a-e8b1-4f7e-a875-e78b4d8e89a8\") " pod="openshift-ingress/payload-processing-e2e-api-a-49dbb5-5777bd86cd-jfgz7" Oct 29 22:59:18.975790 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:18.975774 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e76f405f-e6b4-4645-97ec-a90a9428c826-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-49dbb5-c95485467-m2cbg\" (UID: \"e76f405f-e6b4-4645-97ec-a90a9428c826\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-49dbb5-c95485467-m2cbg" Oct 29 22:59:18.976408 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:18.976384 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b289af8a-e8b1-4f7e-a875-e78b4d8e89a8-plugins-config-volume\") pod \"payload-processing-e2e-api-a-49dbb5-5777bd86cd-jfgz7\" (UID: \"b289af8a-e8b1-4f7e-a875-e78b4d8e89a8\") " pod="openshift-ingress/payload-processing-e2e-api-a-49dbb5-5777bd86cd-jfgz7" Oct 29 22:59:18.976685 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:18.976663 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e76f405f-e6b4-4645-97ec-a90a9428c826-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-49dbb5-c95485467-m2cbg\" (UID: \"e76f405f-e6b4-4645-97ec-a90a9428c826\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-49dbb5-c95485467-m2cbg" Oct 29 22:59:18.998060 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:18.997991 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqm28\" (UniqueName: \"kubernetes.io/projected/e76f405f-e6b4-4645-97ec-a90a9428c826-kube-api-access-kqm28\") pod \"payload-pre-processing-e2e-api-a-49dbb5-c95485467-m2cbg\" (UID: \"e76f405f-e6b4-4645-97ec-a90a9428c826\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-49dbb5-c95485467-m2cbg" Oct 29 22:59:18.998475 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:18.998456 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mnblf\" (UniqueName: \"kubernetes.io/projected/b289af8a-e8b1-4f7e-a875-e78b4d8e89a8-kube-api-access-mnblf\") pod \"payload-processing-e2e-api-a-49dbb5-5777bd86cd-jfgz7\" (UID: \"b289af8a-e8b1-4f7e-a875-e78b4d8e89a8\") " pod="openshift-ingress/payload-processing-e2e-api-a-49dbb5-5777bd86cd-jfgz7" Oct 29 22:59:19.041679 ip-10-0-140-74 kubenswrapper[5057]: E1029 22:59:19.041636 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d2dnn" containerName="extractor" Oct 29 22:59:19.139674 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:19.139636 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-49dbb5-5777bd86cd-jfgz7" Oct 29 22:59:19.288369 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:19.287346 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-49dbb5-c95485467-m2cbg" Oct 29 22:59:19.473223 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:19.472191 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-49dbb5-5777bd86cd-jfgz7"] Oct 29 22:59:19.635643 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:19.635612 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-49dbb5-c95485467-m2cbg"] Oct 29 22:59:19.635841 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:59:19.635816 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode76f405f_e6b4_4645_97ec_a90a9428c826.slice/crio-fb2f73455b468c0c3558f2c23a5ba55c9534d3e58833d64a493cf09a35fc96b9 WatchSource:0}: Error finding container fb2f73455b468c0c3558f2c23a5ba55c9534d3e58833d64a493cf09a35fc96b9: Status 404 returned error can't find the container with id fb2f73455b468c0c3558f2c23a5ba55c9534d3e58833d64a493cf09a35fc96b9 Oct 29 22:59:19.896491 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:19.896399 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:59:20.092353 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:20.091445 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:59:20.104539 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:20.097196 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-49dbb5-5777bd86cd-jfgz7" event={"ID":"b289af8a-e8b1-4f7e-a875-e78b4d8e89a8","Type":"ContainerStarted","Data":"49df3010ab7d00babd135a7d939d6c908db16ab501fe194944f746a9ad294ca4"} Oct 29 22:59:20.104539 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:20.097261 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-49dbb5-5777bd86cd-jfgz7" event={"ID":"b289af8a-e8b1-4f7e-a875-e78b4d8e89a8","Type":"ContainerStarted","Data":"fc094c865028164bfd6d4a4eb134fb5e08b8492ea1792a482ad4f3a9464e7369"} Oct 29 22:59:20.104539 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:20.101302 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-49dbb5-5777bd86cd-jfgz7" Oct 29 22:59:20.104539 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:20.103678 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-49dbb5-c95485467-m2cbg" event={"ID":"e76f405f-e6b4-4645-97ec-a90a9428c826","Type":"ContainerStarted","Data":"a32e51688864616aa0b81a66ea7e5f9a1fe4304aef168ea8d53317c06968b22b"} Oct 29 22:59:20.104539 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:20.103724 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-49dbb5-c95485467-m2cbg" event={"ID":"e76f405f-e6b4-4645-97ec-a90a9428c826","Type":"ContainerStarted","Data":"fb2f73455b468c0c3558f2c23a5ba55c9534d3e58833d64a493cf09a35fc96b9"} Oct 29 22:59:20.104539 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:20.104471 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-49dbb5-c95485467-m2cbg" Oct 29 22:59:20.143576 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:20.143522 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-49dbb5-5777bd86cd-jfgz7" podStartSLOduration=2.1435042539999998 podStartE2EDuration="2.143504254s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:59:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:59:20.139435715 +0000 UTC m=+1540.922649152" watchObservedRunningTime="2026-10-29 22:59:20.143504254 +0000 UTC m=+1540.926717679" Oct 29 22:59:20.181705 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:20.181595 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-49dbb5-c95485467-m2cbg" podStartSLOduration=2.181573134 podStartE2EDuration="2.181573134s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:59:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:59:20.176276293 +0000 UTC m=+1540.959489719" watchObservedRunningTime="2026-10-29 22:59:20.181573134 +0000 UTC m=+1540.964786561" Oct 29 22:59:20.339353 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:20.338960 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-36befd-65b4658f87-7xrg8"] Oct 29 22:59:20.347067 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:20.347034 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-36befd-65b4658f87-7xrg8" Oct 29 22:59:20.362990 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:20.362942 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-36befd\"" Oct 29 22:59:20.383347 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:20.378080 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-36befd-65b4658f87-7xrg8"] Oct 29 22:59:20.415355 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:20.414701 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85gqt\" (UniqueName: \"kubernetes.io/projected/767849e9-8c9c-45d1-91e6-0d8fa0ab2945-kube-api-access-85gqt\") pod \"payload-pre-processing-e2e-ipp-a-36befd-65b4658f87-7xrg8\" (UID: \"767849e9-8c9c-45d1-91e6-0d8fa0ab2945\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-36befd-65b4658f87-7xrg8" Oct 29 22:59:20.415355 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:20.414938 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/767849e9-8c9c-45d1-91e6-0d8fa0ab2945-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-36befd-65b4658f87-7xrg8\" (UID: \"767849e9-8c9c-45d1-91e6-0d8fa0ab2945\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-36befd-65b4658f87-7xrg8" Oct 29 22:59:20.495310 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:20.495258 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-36befd-97f97445d-vmwlw"] Oct 29 22:59:20.505409 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:20.502622 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-36befd-97f97445d-vmwlw" Oct 29 22:59:20.511600 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:20.511572 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-36befd-97f97445d-vmwlw"] Oct 29 22:59:20.520726 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:20.517376 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85gqt\" (UniqueName: \"kubernetes.io/projected/767849e9-8c9c-45d1-91e6-0d8fa0ab2945-kube-api-access-85gqt\") pod \"payload-pre-processing-e2e-ipp-a-36befd-65b4658f87-7xrg8\" (UID: \"767849e9-8c9c-45d1-91e6-0d8fa0ab2945\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-36befd-65b4658f87-7xrg8" Oct 29 22:59:20.520726 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:20.517939 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f774785c-ad04-4414-80b3-dcbe4d3f75e0-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-36befd-97f97445d-vmwlw\" (UID: \"f774785c-ad04-4414-80b3-dcbe4d3f75e0\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-36befd-97f97445d-vmwlw" Oct 29 22:59:20.520726 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:20.518130 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbs6f\" (UniqueName: \"kubernetes.io/projected/f774785c-ad04-4414-80b3-dcbe4d3f75e0-kube-api-access-mbs6f\") pod \"payload-processing-e2e-ipp-a-36befd-97f97445d-vmwlw\" (UID: \"f774785c-ad04-4414-80b3-dcbe4d3f75e0\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-36befd-97f97445d-vmwlw" Oct 29 22:59:20.520726 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:20.518165 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/767849e9-8c9c-45d1-91e6-0d8fa0ab2945-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-36befd-65b4658f87-7xrg8\" (UID: \"767849e9-8c9c-45d1-91e6-0d8fa0ab2945\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-36befd-65b4658f87-7xrg8" Oct 29 22:59:20.520726 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:20.519628 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/767849e9-8c9c-45d1-91e6-0d8fa0ab2945-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-36befd-65b4658f87-7xrg8\" (UID: \"767849e9-8c9c-45d1-91e6-0d8fa0ab2945\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-36befd-65b4658f87-7xrg8" Oct 29 22:59:20.520726 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:20.520529 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-36befd-dockercfg-wwhhc\"" Oct 29 22:59:20.541894 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:20.541862 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-85gqt\" (UniqueName: \"kubernetes.io/projected/767849e9-8c9c-45d1-91e6-0d8fa0ab2945-kube-api-access-85gqt\") pod \"payload-pre-processing-e2e-ipp-a-36befd-65b4658f87-7xrg8\" (UID: \"767849e9-8c9c-45d1-91e6-0d8fa0ab2945\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-36befd-65b4658f87-7xrg8" Oct 29 22:59:20.620164 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:20.620119 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbs6f\" (UniqueName: \"kubernetes.io/projected/f774785c-ad04-4414-80b3-dcbe4d3f75e0-kube-api-access-mbs6f\") pod \"payload-processing-e2e-ipp-a-36befd-97f97445d-vmwlw\" (UID: \"f774785c-ad04-4414-80b3-dcbe4d3f75e0\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-36befd-97f97445d-vmwlw" Oct 29 22:59:20.620449 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:20.620432 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f774785c-ad04-4414-80b3-dcbe4d3f75e0-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-36befd-97f97445d-vmwlw\" (UID: \"f774785c-ad04-4414-80b3-dcbe4d3f75e0\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-36befd-97f97445d-vmwlw" Oct 29 22:59:20.621142 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:20.621120 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f774785c-ad04-4414-80b3-dcbe4d3f75e0-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-36befd-97f97445d-vmwlw\" (UID: \"f774785c-ad04-4414-80b3-dcbe4d3f75e0\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-36befd-97f97445d-vmwlw" Oct 29 22:59:20.655348 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:20.642699 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbs6f\" (UniqueName: \"kubernetes.io/projected/f774785c-ad04-4414-80b3-dcbe4d3f75e0-kube-api-access-mbs6f\") pod \"payload-processing-e2e-ipp-a-36befd-97f97445d-vmwlw\" (UID: \"f774785c-ad04-4414-80b3-dcbe4d3f75e0\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-36befd-97f97445d-vmwlw" Oct 29 22:59:20.684507 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:20.684212 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-36befd-65b4658f87-7xrg8" Oct 29 22:59:20.833934 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:20.833899 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-36befd-97f97445d-vmwlw" Oct 29 22:59:21.063377 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:59:21.062289 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod767849e9_8c9c_45d1_91e6_0d8fa0ab2945.slice/crio-710d5b028d3e11bfd23557f68d59fd654edb7aa839c8c56748832e101ef53908 WatchSource:0}: Error finding container 710d5b028d3e11bfd23557f68d59fd654edb7aa839c8c56748832e101ef53908: Status 404 returned error can't find the container with id 710d5b028d3e11bfd23557f68d59fd654edb7aa839c8c56748832e101ef53908 Oct 29 22:59:21.063377 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:21.062453 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-36befd-65b4658f87-7xrg8"] Oct 29 22:59:21.114660 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:21.114624 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-36befd-65b4658f87-7xrg8" event={"ID":"767849e9-8c9c-45d1-91e6-0d8fa0ab2945","Type":"ContainerStarted","Data":"710d5b028d3e11bfd23557f68d59fd654edb7aa839c8c56748832e101ef53908"} Oct 29 22:59:21.189270 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:59:21.189226 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf774785c_ad04_4414_80b3_dcbe4d3f75e0.slice/crio-770ff25c1266e2e65ec045077cffc32d50e21af63396808bd733e01f25f02b34 WatchSource:0}: Error finding container 770ff25c1266e2e65ec045077cffc32d50e21af63396808bd733e01f25f02b34: Status 404 returned error can't find the container with id 770ff25c1266e2e65ec045077cffc32d50e21af63396808bd733e01f25f02b34 Oct 29 22:59:21.189743 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:21.189489 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-36befd-97f97445d-vmwlw"] Oct 29 22:59:21.668314 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:21.668278 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:59:22.138965 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:22.138927 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-36befd-97f97445d-vmwlw" event={"ID":"f774785c-ad04-4414-80b3-dcbe4d3f75e0","Type":"ContainerStarted","Data":"770ff25c1266e2e65ec045077cffc32d50e21af63396808bd733e01f25f02b34"} Oct 29 22:59:23.589965 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:23.589918 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:59:24.159392 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:24.159347 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-36befd-65b4658f87-7xrg8" event={"ID":"767849e9-8c9c-45d1-91e6-0d8fa0ab2945","Type":"ContainerStarted","Data":"86cb77e25b23b76e275c796c4006ce3a1658188be6b4b7c3cf4ec959d4745947"} Oct 29 22:59:24.159618 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:24.159423 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-36befd-65b4658f87-7xrg8" Oct 29 22:59:24.161409 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:24.161378 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-36befd-97f97445d-vmwlw" event={"ID":"f774785c-ad04-4414-80b3-dcbe4d3f75e0","Type":"ContainerStarted","Data":"3ae53834965d08d05446e1ca580cbce1983e403da96b045218a49c60c762d360"} Oct 29 22:59:24.161527 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:24.161438 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-36befd-97f97445d-vmwlw" Oct 29 22:59:24.202842 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:24.202790 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-36befd-65b4658f87-7xrg8" podStartSLOduration=2.102438547 podStartE2EDuration="4.202776309s" totalImagesPullingTime="2.100337762s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:59:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:59:24.199153987 +0000 UTC m=+1544.982367415" watchObservedRunningTime="2026-10-29 22:59:24.202776309 +0000 UTC m=+1544.985989771" Oct 29 22:59:24.240697 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:24.240639 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-36befd-97f97445d-vmwlw" podStartSLOduration=2.261041005 podStartE2EDuration="4.240620602s" totalImagesPullingTime="1.979579597s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:59:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:59:24.237531374 +0000 UTC m=+1545.020744799" watchObservedRunningTime="2026-10-29 22:59:24.240620602 +0000 UTC m=+1545.023834029" Oct 29 22:59:24.654226 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:24.654185 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:59:24.775848 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:24.775812 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:59:25.019353 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:25.018655 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:59:25.259392 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:25.258525 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:59:25.413412 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:25.412546 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:59:25.548755 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:25.548534 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:59:25.795352 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:25.778412 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a1d1da-6f68d967db-9njsv"] Oct 29 22:59:25.827402 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:25.825615 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a1d1da-6f68d967db-9njsv"] Oct 29 22:59:25.827402 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:25.825772 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-a1d1da-6f68d967db-9njsv" Oct 29 22:59:25.839446 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:25.839414 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-a1d1da-dockercfg-dwnn9\"" Oct 29 22:59:25.839581 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:25.839477 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-a1d1da\"" Oct 29 22:59:25.931631 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:25.931593 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvwsn\" (UniqueName: \"kubernetes.io/projected/94c6a587-bd68-4f2f-8bc0-38261350e01c-kube-api-access-rvwsn\") pod \"payload-processing-e2e-ipp-a-a1d1da-6f68d967db-9njsv\" (UID: \"94c6a587-bd68-4f2f-8bc0-38261350e01c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a1d1da-6f68d967db-9njsv" Oct 29 22:59:25.931969 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:25.931951 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94c6a587-bd68-4f2f-8bc0-38261350e01c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-a1d1da-6f68d967db-9njsv\" (UID: \"94c6a587-bd68-4f2f-8bc0-38261350e01c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a1d1da-6f68d967db-9njsv" Oct 29 22:59:25.988394 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:25.988359 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a1d1da-5475458f5f-h9gjm"] Oct 29 22:59:25.993780 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:25.993751 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a1d1da-5475458f5f-h9gjm" Oct 29 22:59:26.014797 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:26.014765 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a1d1da-5475458f5f-h9gjm"] Oct 29 22:59:26.081547 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:26.070125 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94c6a587-bd68-4f2f-8bc0-38261350e01c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-a1d1da-6f68d967db-9njsv\" (UID: \"94c6a587-bd68-4f2f-8bc0-38261350e01c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a1d1da-6f68d967db-9njsv" Oct 29 22:59:26.081547 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:26.070219 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvwsn\" (UniqueName: \"kubernetes.io/projected/94c6a587-bd68-4f2f-8bc0-38261350e01c-kube-api-access-rvwsn\") pod \"payload-processing-e2e-ipp-a-a1d1da-6f68d967db-9njsv\" (UID: \"94c6a587-bd68-4f2f-8bc0-38261350e01c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a1d1da-6f68d967db-9njsv" Oct 29 22:59:26.081547 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:26.070301 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cb4qn\" (UniqueName: \"kubernetes.io/projected/5f36ee07-3a0b-4db0-af36-46db93217079-kube-api-access-cb4qn\") pod \"payload-pre-processing-e2e-ipp-a-a1d1da-5475458f5f-h9gjm\" (UID: \"5f36ee07-3a0b-4db0-af36-46db93217079\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a1d1da-5475458f5f-h9gjm" Oct 29 22:59:26.081547 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:26.070425 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f36ee07-3a0b-4db0-af36-46db93217079-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-a1d1da-5475458f5f-h9gjm\" (UID: \"5f36ee07-3a0b-4db0-af36-46db93217079\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a1d1da-5475458f5f-h9gjm" Oct 29 22:59:26.086160 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:26.086087 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94c6a587-bd68-4f2f-8bc0-38261350e01c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-a1d1da-6f68d967db-9njsv\" (UID: \"94c6a587-bd68-4f2f-8bc0-38261350e01c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a1d1da-6f68d967db-9njsv" Oct 29 22:59:26.167863 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:26.167830 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvwsn\" (UniqueName: \"kubernetes.io/projected/94c6a587-bd68-4f2f-8bc0-38261350e01c-kube-api-access-rvwsn\") pod \"payload-processing-e2e-ipp-a-a1d1da-6f68d967db-9njsv\" (UID: \"94c6a587-bd68-4f2f-8bc0-38261350e01c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a1d1da-6f68d967db-9njsv" Oct 29 22:59:26.174521 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:26.174488 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f36ee07-3a0b-4db0-af36-46db93217079-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-a1d1da-5475458f5f-h9gjm\" (UID: \"5f36ee07-3a0b-4db0-af36-46db93217079\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a1d1da-5475458f5f-h9gjm" Oct 29 22:59:26.176150 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:26.176123 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cb4qn\" (UniqueName: \"kubernetes.io/projected/5f36ee07-3a0b-4db0-af36-46db93217079-kube-api-access-cb4qn\") pod \"payload-pre-processing-e2e-ipp-a-a1d1da-5475458f5f-h9gjm\" (UID: \"5f36ee07-3a0b-4db0-af36-46db93217079\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a1d1da-5475458f5f-h9gjm" Oct 29 22:59:26.181315 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:26.175662 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f36ee07-3a0b-4db0-af36-46db93217079-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-a1d1da-5475458f5f-h9gjm\" (UID: \"5f36ee07-3a0b-4db0-af36-46db93217079\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a1d1da-5475458f5f-h9gjm" Oct 29 22:59:26.238496 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:26.238461 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cb4qn\" (UniqueName: \"kubernetes.io/projected/5f36ee07-3a0b-4db0-af36-46db93217079-kube-api-access-cb4qn\") pod \"payload-pre-processing-e2e-ipp-a-a1d1da-5475458f5f-h9gjm\" (UID: \"5f36ee07-3a0b-4db0-af36-46db93217079\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a1d1da-5475458f5f-h9gjm" Oct 29 22:59:26.332585 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:26.319246 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a1d1da-5475458f5f-h9gjm" Oct 29 22:59:26.448308 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:26.448280 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-a1d1da-6f68d967db-9njsv" Oct 29 22:59:26.975094 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:59:26.975059 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f36ee07_3a0b_4db0_af36_46db93217079.slice/crio-96b37ddea955de99831e58c1aaf2883a2b99d40f1c3752e8b30be6ea60eac208 WatchSource:0}: Error finding container 96b37ddea955de99831e58c1aaf2883a2b99d40f1c3752e8b30be6ea60eac208: Status 404 returned error can't find the container with id 96b37ddea955de99831e58c1aaf2883a2b99d40f1c3752e8b30be6ea60eac208 Oct 29 22:59:26.976197 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:26.976171 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a1d1da-5475458f5f-h9gjm"] Oct 29 22:59:27.014184 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:27.014018 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a1d1da-6f68d967db-9njsv"] Oct 29 22:59:27.018922 ip-10-0-140-74 kubenswrapper[5057]: W1029 22:59:27.018891 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94c6a587_bd68_4f2f_8bc0_38261350e01c.slice/crio-c0d0df305d53646abf645ccbd08ec53d500171367894cce8e51ac1707427ec4b WatchSource:0}: Error finding container c0d0df305d53646abf645ccbd08ec53d500171367894cce8e51ac1707427ec4b: Status 404 returned error can't find the container with id c0d0df305d53646abf645ccbd08ec53d500171367894cce8e51ac1707427ec4b Oct 29 22:59:27.286415 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:27.284301 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:59:27.369422 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:27.366838 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a1d1da-5475458f5f-h9gjm" Oct 29 22:59:27.369422 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:27.366908 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-a1d1da-6f68d967db-9njsv" event={"ID":"94c6a587-bd68-4f2f-8bc0-38261350e01c","Type":"ContainerStarted","Data":"a3837e24b3140cd250d33f80b9dfce39ce727a25a3d936250eaff01d38b89676"} Oct 29 22:59:27.369422 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:27.366942 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-a1d1da-6f68d967db-9njsv" Oct 29 22:59:27.369422 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:27.366978 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-a1d1da-6f68d967db-9njsv" event={"ID":"94c6a587-bd68-4f2f-8bc0-38261350e01c","Type":"ContainerStarted","Data":"c0d0df305d53646abf645ccbd08ec53d500171367894cce8e51ac1707427ec4b"} Oct 29 22:59:27.369422 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:27.367002 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a1d1da-5475458f5f-h9gjm" event={"ID":"5f36ee07-3a0b-4db0-af36-46db93217079","Type":"ContainerStarted","Data":"c1d27951a8d62b6da23dfe0794834187ea5a7afa6a5c936830fef28dcbec48af"} Oct 29 22:59:27.369422 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:27.367029 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a1d1da-5475458f5f-h9gjm" event={"ID":"5f36ee07-3a0b-4db0-af36-46db93217079","Type":"ContainerStarted","Data":"96b37ddea955de99831e58c1aaf2883a2b99d40f1c3752e8b30be6ea60eac208"} Oct 29 22:59:27.405399 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:27.404822 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a1d1da-5475458f5f-h9gjm" podStartSLOduration=2.404803979 podStartE2EDuration="2.404803979s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:59:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:59:27.404610416 +0000 UTC m=+1548.187823844" watchObservedRunningTime="2026-10-29 22:59:27.404803979 +0000 UTC m=+1548.188017409" Oct 29 22:59:27.452391 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:27.452177 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-a1d1da-6f68d967db-9njsv" podStartSLOduration=2.452158977 podStartE2EDuration="2.452158977s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:59:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:59:27.447089849 +0000 UTC m=+1548.230303299" watchObservedRunningTime="2026-10-29 22:59:27.452158977 +0000 UTC m=+1548.235372404" Oct 29 22:59:29.005550 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:29.005509 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:59:30.527841 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:30.527807 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:59:30.635256 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:30.635194 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:59:30.743427 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:30.743389 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:59:32.031293 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:32.031248 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:59:32.142599 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:32.142566 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-49dbb5-c95485467-m2cbg" Oct 29 22:59:32.142797 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:32.142655 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-49dbb5-5777bd86cd-jfgz7" Oct 29 22:59:32.278187 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:32.278134 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:59:33.882758 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:33.882716 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:59:35.295266 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:35.295233 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-36befd-65b4658f87-7xrg8" Oct 29 22:59:35.295742 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:35.295310 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-36befd-97f97445d-vmwlw" Oct 29 22:59:35.523702 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:35.523662 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:59:36.380684 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:36.380548 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:59:36.486993 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:36.486959 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:59:37.140728 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:37.140697 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:59:38.134924 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:38.134883 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:59:38.372679 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:38.372652 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-a1d1da-6f68d967db-9njsv" Oct 29 22:59:38.372840 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:38.372717 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a1d1da-5475458f5f-h9gjm" Oct 29 22:59:38.718179 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:38.718114 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:59:38.833283 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:38.833245 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:59:40.263942 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:40.263907 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:59:41.407204 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:41.407164 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:59:41.840168 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:41.840131 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v2tfm"] Oct 29 22:59:41.898130 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:41.898099 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:59:42.043371 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:42.040429 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:59:42.151887 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:42.150049 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:59:42.829363 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:42.828242 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:59:42.948243 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:42.948211 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:59:44.429780 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:44.429749 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:59:44.549846 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:44.549720 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:59:44.666579 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:44.666542 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:59:44.784003 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:44.783956 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:59:45.999036 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:45.999001 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:59:46.115260 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:46.115225 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:59:46.911496 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:46.911464 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:59:48.207312 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:48.207277 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:59:48.333009 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:48.332975 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:59:49.034934 ip-10-0-140-74 kubenswrapper[5057]: E1029 22:59:49.034898 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d2dnn" containerName="extractor" Oct 29 22:59:49.402999 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:49.402965 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:59:51.105146 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:51.105063 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:59:52.954071 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:52.954032 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:59:54.194559 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:54.194526 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:59:54.852881 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:54.852813 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:59:55.032845 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:55.032800 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:59:55.422716 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:55.422671 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:59:55.628353 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:55.625432 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:59:57.411939 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:57.411899 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 22:59:59.186483 ip-10-0-140-74 kubenswrapper[5057]: I1029 22:59:59.186439 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:00:00.225370 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:00.225299 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-0df9b0-29888580-ss4kb"] Oct 29 23:00:00.229833 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:00.229813 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-0df9b0-29888580-ss4kb" Oct 29 23:00:00.256261 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:00.256229 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-0df9b0-29888580-ss4kb"] Oct 29 23:00:00.290399 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:00.290363 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dmx57\"" Oct 29 23:00:00.332734 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:00.332703 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxg5h\" (UniqueName: \"kubernetes.io/projected/33a3c7b1-abf7-4cce-b13a-073c96d05d9f-kube-api-access-rxg5h\") pod \"maas-api-key-cleanup-e2e-api-b-0df9b0-29888580-ss4kb\" (UID: \"33a3c7b1-abf7-4cce-b13a-073c96d05d9f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-0df9b0-29888580-ss4kb" Oct 29 23:00:00.332913 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:00.332776 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/33a3c7b1-abf7-4cce-b13a-073c96d05d9f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-0df9b0-29888580-ss4kb\" (UID: \"33a3c7b1-abf7-4cce-b13a-073c96d05d9f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-0df9b0-29888580-ss4kb" Oct 29 23:00:00.434450 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:00.434417 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxg5h\" (UniqueName: \"kubernetes.io/projected/33a3c7b1-abf7-4cce-b13a-073c96d05d9f-kube-api-access-rxg5h\") pod \"maas-api-key-cleanup-e2e-api-b-0df9b0-29888580-ss4kb\" (UID: \"33a3c7b1-abf7-4cce-b13a-073c96d05d9f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-0df9b0-29888580-ss4kb" Oct 29 23:00:00.434648 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:00.434597 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/33a3c7b1-abf7-4cce-b13a-073c96d05d9f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-0df9b0-29888580-ss4kb\" (UID: \"33a3c7b1-abf7-4cce-b13a-073c96d05d9f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-0df9b0-29888580-ss4kb" Oct 29 23:00:00.435527 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:00.435492 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/33a3c7b1-abf7-4cce-b13a-073c96d05d9f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-0df9b0-29888580-ss4kb\" (UID: \"33a3c7b1-abf7-4cce-b13a-073c96d05d9f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-0df9b0-29888580-ss4kb" Oct 29 23:00:00.456342 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:00.456308 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxg5h\" (UniqueName: \"kubernetes.io/projected/33a3c7b1-abf7-4cce-b13a-073c96d05d9f-kube-api-access-rxg5h\") pod \"maas-api-key-cleanup-e2e-api-b-0df9b0-29888580-ss4kb\" (UID: \"33a3c7b1-abf7-4cce-b13a-073c96d05d9f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-0df9b0-29888580-ss4kb" Oct 29 23:00:00.552660 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:00.552591 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-0df9b0-29888580-ss4kb" Oct 29 23:00:00.817756 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:00.817668 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:00:00.864166 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:00.864131 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-0df9b0-29888580-ss4kb"] Oct 29 23:00:00.865094 ip-10-0-140-74 kubenswrapper[5057]: W1029 23:00:00.865064 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33a3c7b1_abf7_4cce_b13a_073c96d05d9f.slice/crio-550b380d9c366cc0d850f85881f04ca2cfccd1a16d2a60dee945aedfd0391303 WatchSource:0}: Error finding container 550b380d9c366cc0d850f85881f04ca2cfccd1a16d2a60dee945aedfd0391303: Status 404 returned error can't find the container with id 550b380d9c366cc0d850f85881f04ca2cfccd1a16d2a60dee945aedfd0391303 Oct 29 23:00:00.903115 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:00.902955 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-0df9b0-29888580-ss4kb" event={"ID":"33a3c7b1-abf7-4cce-b13a-073c96d05d9f","Type":"ContainerStarted","Data":"550b380d9c366cc0d850f85881f04ca2cfccd1a16d2a60dee945aedfd0391303"} Oct 29 23:00:02.386787 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:02.386696 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:00:03.135346 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:03.135301 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:00:03.269944 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:03.269909 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:00:03.501395 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:03.476514 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-27d16a-openshift-default-869755f997-ws2s6"] Oct 29 23:00:03.501395 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:03.477258 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-27d16a-openshift-default-869755f997-ws2s6" podUID="e6af2d78-ba9f-4f94-a985-47985aa30dd3" containerName="istio-proxy" containerID="cri-o://0148cf646e079db62e52d31a8ddc05db1e5e580fbd3a9b7063aeaf4ecec39bb2" gracePeriod=30 Oct 29 23:00:03.516385 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:03.508948 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:00:03.866613 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:03.866047 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:00:03.917919 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:03.917611 5057 patch_prober.go:29] interesting pod/e2e-api-b-27d16a-openshift-default-869755f997-ws2s6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 23:00:03.917919 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:03.917666 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-27d16a-openshift-default-869755f997-ws2s6" podUID="e6af2d78-ba9f-4f94-a985-47985aa30dd3" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 23:00:04.605410 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:04.605365 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:00:04.782815 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:04.782779 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:00:04.895761 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:04.895679 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:00:05.004081 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:05.003130 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-0df9b0-29888580-ss4kb" event={"ID":"33a3c7b1-abf7-4cce-b13a-073c96d05d9f","Type":"ContainerStarted","Data":"42b2b45ae412f7bf3875a4f092eb372f8def38f5c966a9e31d8b2ce0dacf1f81"} Oct 29 23:00:05.015357 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:05.014788 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:00:05.342637 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:05.342424 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:00:05.581262 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:05.576248 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:00:05.773614 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:05.773574 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:00:06.016757 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:06.016712 5057 generic.go:356] "Generic (PLEG): container finished" podID="33a3c7b1-abf7-4cce-b13a-073c96d05d9f" containerID="42b2b45ae412f7bf3875a4f092eb372f8def38f5c966a9e31d8b2ce0dacf1f81" exitCode=60 Oct 29 23:00:06.017125 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:06.017096 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-0df9b0-29888580-ss4kb" event={"ID":"33a3c7b1-abf7-4cce-b13a-073c96d05d9f","Type":"ContainerDied","Data":"42b2b45ae412f7bf3875a4f092eb372f8def38f5c966a9e31d8b2ce0dacf1f81"} Oct 29 23:00:06.017722 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:06.017694 5057 scope.go:122] "RemoveContainer" containerID="42b2b45ae412f7bf3875a4f092eb372f8def38f5c966a9e31d8b2ce0dacf1f81" Oct 29 23:00:07.035267 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:07.035233 5057 generic.go:356] "Generic (PLEG): container finished" podID="33a3c7b1-abf7-4cce-b13a-073c96d05d9f" containerID="652bd7e293937c9cfbd235415cb6a91f6b80534aae6342499eedd28fa3bd6778" exitCode=60 Oct 29 23:00:07.035851 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:07.035309 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-0df9b0-29888580-ss4kb" event={"ID":"33a3c7b1-abf7-4cce-b13a-073c96d05d9f","Type":"ContainerDied","Data":"652bd7e293937c9cfbd235415cb6a91f6b80534aae6342499eedd28fa3bd6778"} Oct 29 23:00:07.035851 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:07.035446 5057 scope.go:122] "RemoveContainer" containerID="42b2b45ae412f7bf3875a4f092eb372f8def38f5c966a9e31d8b2ce0dacf1f81" Oct 29 23:00:07.035851 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:07.035817 5057 scope.go:122] "RemoveContainer" containerID="652bd7e293937c9cfbd235415cb6a91f6b80534aae6342499eedd28fa3bd6778" Oct 29 23:00:07.036251 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:00:07.036231 5057 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-0df9b0-29888580-ss4kb_odh-ai-gateway-infra(33a3c7b1-abf7-4cce-b13a-073c96d05d9f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-0df9b0-29888580-ss4kb" podUID="33a3c7b1-abf7-4cce-b13a-073c96d05d9f" Oct 29 23:00:07.518908 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:07.518866 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:00:08.045753 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:08.045727 5057 scope.go:122] "RemoveContainer" containerID="652bd7e293937c9cfbd235415cb6a91f6b80534aae6342499eedd28fa3bd6778" Oct 29 23:00:08.046140 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:00:08.046025 5057 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-0df9b0-29888580-ss4kb_odh-ai-gateway-infra(33a3c7b1-abf7-4cce-b13a-073c96d05d9f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-0df9b0-29888580-ss4kb" podUID="33a3c7b1-abf7-4cce-b13a-073c96d05d9f" Oct 29 23:00:09.003315 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:09.002745 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:00:09.058186 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:09.058152 5057 generic.go:356] "Generic (PLEG): container finished" podID="e6af2d78-ba9f-4f94-a985-47985aa30dd3" containerID="0148cf646e079db62e52d31a8ddc05db1e5e580fbd3a9b7063aeaf4ecec39bb2" exitCode=0 Oct 29 23:00:09.058585 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:09.058251 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-27d16a-openshift-default-869755f997-ws2s6" event={"ID":"e6af2d78-ba9f-4f94-a985-47985aa30dd3","Type":"ContainerDied","Data":"0148cf646e079db62e52d31a8ddc05db1e5e580fbd3a9b7063aeaf4ecec39bb2"} Oct 29 23:00:09.123972 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:09.123934 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:00:09.221534 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:09.220778 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-27d16a-openshift-default-869755f997-ws2s6" Oct 29 23:00:09.291019 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:09.290895 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:00:09.341608 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:09.341579 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6af2d78-ba9f-4f94-a985-47985aa30dd3-istio-token\" (UniqueName: \"kubernetes.io/projected/e6af2d78-ba9f-4f94-a985-47985aa30dd3-istio-token\") pod \"e6af2d78-ba9f-4f94-a985-47985aa30dd3\" (UID: \"e6af2d78-ba9f-4f94-a985-47985aa30dd3\") " Oct 29 23:00:09.341764 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:09.341631 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6af2d78-ba9f-4f94-a985-47985aa30dd3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6af2d78-ba9f-4f94-a985-47985aa30dd3-istio-data\") pod \"e6af2d78-ba9f-4f94-a985-47985aa30dd3\" (UID: \"e6af2d78-ba9f-4f94-a985-47985aa30dd3\") " Oct 29 23:00:09.341764 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:09.341726 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6af2d78-ba9f-4f94-a985-47985aa30dd3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6af2d78-ba9f-4f94-a985-47985aa30dd3-workload-socket\") pod \"e6af2d78-ba9f-4f94-a985-47985aa30dd3\" (UID: \"e6af2d78-ba9f-4f94-a985-47985aa30dd3\") " Oct 29 23:00:09.341906 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:09.341786 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6af2d78-ba9f-4f94-a985-47985aa30dd3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6af2d78-ba9f-4f94-a985-47985aa30dd3-credential-socket\") pod \"e6af2d78-ba9f-4f94-a985-47985aa30dd3\" (UID: \"e6af2d78-ba9f-4f94-a985-47985aa30dd3\") " Oct 29 23:00:09.341906 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:09.341823 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6af2d78-ba9f-4f94-a985-47985aa30dd3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6af2d78-ba9f-4f94-a985-47985aa30dd3-workload-certs\") pod \"e6af2d78-ba9f-4f94-a985-47985aa30dd3\" (UID: \"e6af2d78-ba9f-4f94-a985-47985aa30dd3\") " Oct 29 23:00:09.341906 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:09.341899 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6af2d78-ba9f-4f94-a985-47985aa30dd3-istio-data" pod "e6af2d78-ba9f-4f94-a985-47985aa30dd3" (UID: "e6af2d78-ba9f-4f94-a985-47985aa30dd3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:00:09.342067 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:09.341960 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e6af2d78-ba9f-4f94-a985-47985aa30dd3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6af2d78-ba9f-4f94-a985-47985aa30dd3-istio-podinfo\") pod \"e6af2d78-ba9f-4f94-a985-47985aa30dd3\" (UID: \"e6af2d78-ba9f-4f94-a985-47985aa30dd3\") " Oct 29 23:00:09.342067 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:09.341998 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e6af2d78-ba9f-4f94-a985-47985aa30dd3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6af2d78-ba9f-4f94-a985-47985aa30dd3-istiod-ca-cert\") pod \"e6af2d78-ba9f-4f94-a985-47985aa30dd3\" (UID: \"e6af2d78-ba9f-4f94-a985-47985aa30dd3\") " Oct 29 23:00:09.342164 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:09.342091 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6af2d78-ba9f-4f94-a985-47985aa30dd3-kube-api-access-9dkpt\" (UniqueName: \"kubernetes.io/projected/e6af2d78-ba9f-4f94-a985-47985aa30dd3-kube-api-access-9dkpt\") pod \"e6af2d78-ba9f-4f94-a985-47985aa30dd3\" (UID: \"e6af2d78-ba9f-4f94-a985-47985aa30dd3\") " Oct 29 23:00:09.342164 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:09.342097 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6af2d78-ba9f-4f94-a985-47985aa30dd3-workload-socket" pod "e6af2d78-ba9f-4f94-a985-47985aa30dd3" (UID: "e6af2d78-ba9f-4f94-a985-47985aa30dd3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:00:09.342610 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:09.342255 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6af2d78-ba9f-4f94-a985-47985aa30dd3-workload-certs" pod "e6af2d78-ba9f-4f94-a985-47985aa30dd3" (UID: "e6af2d78-ba9f-4f94-a985-47985aa30dd3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:00:09.342610 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:09.342429 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6af2d78-ba9f-4f94-a985-47985aa30dd3-credential-socket" pod "e6af2d78-ba9f-4f94-a985-47985aa30dd3" (UID: "e6af2d78-ba9f-4f94-a985-47985aa30dd3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:00:09.342610 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:09.342488 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6af2d78-ba9f-4f94-a985-47985aa30dd3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6af2d78-ba9f-4f94-a985-47985aa30dd3-istio-envoy\") pod \"e6af2d78-ba9f-4f94-a985-47985aa30dd3\" (UID: \"e6af2d78-ba9f-4f94-a985-47985aa30dd3\") " Oct 29 23:00:09.342899 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:09.342876 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6af2d78-ba9f-4f94-a985-47985aa30dd3-istiod-ca-cert" pod "e6af2d78-ba9f-4f94-a985-47985aa30dd3" (UID: "e6af2d78-ba9f-4f94-a985-47985aa30dd3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:00:09.343235 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:09.343212 5057 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6af2d78-ba9f-4f94-a985-47985aa30dd3-istio-data\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:00:09.343315 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:09.343240 5057 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6af2d78-ba9f-4f94-a985-47985aa30dd3-workload-socket\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:00:09.343315 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:09.343255 5057 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6af2d78-ba9f-4f94-a985-47985aa30dd3-credential-socket\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:00:09.343315 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:09.343268 5057 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6af2d78-ba9f-4f94-a985-47985aa30dd3-workload-certs\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:00:09.343315 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:09.343282 5057 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6af2d78-ba9f-4f94-a985-47985aa30dd3-istiod-ca-cert\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:00:09.345210 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:09.345189 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6af2d78-ba9f-4f94-a985-47985aa30dd3-istio-token" pod "e6af2d78-ba9f-4f94-a985-47985aa30dd3" (UID: "e6af2d78-ba9f-4f94-a985-47985aa30dd3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:00:09.348520 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:09.348491 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6af2d78-ba9f-4f94-a985-47985aa30dd3-istio-envoy" pod "e6af2d78-ba9f-4f94-a985-47985aa30dd3" (UID: "e6af2d78-ba9f-4f94-a985-47985aa30dd3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:00:09.349752 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:09.349721 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e6af2d78-ba9f-4f94-a985-47985aa30dd3-istio-podinfo" pod "e6af2d78-ba9f-4f94-a985-47985aa30dd3" (UID: "e6af2d78-ba9f-4f94-a985-47985aa30dd3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:00:09.349752 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:09.349739 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6af2d78-ba9f-4f94-a985-47985aa30dd3-kube-api-access-9dkpt" pod "e6af2d78-ba9f-4f94-a985-47985aa30dd3" (UID: "e6af2d78-ba9f-4f94-a985-47985aa30dd3"). InnerVolumeSpecName "kube-api-access-9dkpt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:00:09.430440 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:09.428224 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:00:09.448547 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:09.448397 5057 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6af2d78-ba9f-4f94-a985-47985aa30dd3-istio-token\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:00:09.448547 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:09.448442 5057 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6af2d78-ba9f-4f94-a985-47985aa30dd3-istio-podinfo\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:00:09.448547 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:09.448458 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9dkpt\" (UniqueName: \"kubernetes.io/projected/e6af2d78-ba9f-4f94-a985-47985aa30dd3-kube-api-access-9dkpt\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:00:09.448547 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:09.448472 5057 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6af2d78-ba9f-4f94-a985-47985aa30dd3-istio-envoy\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:00:10.068668 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:10.068639 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-27d16a-openshift-default-869755f997-ws2s6" Oct 29 23:00:10.068668 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:10.068652 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-27d16a-openshift-default-869755f997-ws2s6" event={"ID":"e6af2d78-ba9f-4f94-a985-47985aa30dd3","Type":"ContainerDied","Data":"ccbd0be5546ace2f2768c2f9581ce0e4b895121b24c2998b7215bcf0b1d0f834"} Oct 29 23:00:10.069130 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:10.068715 5057 scope.go:122] "RemoveContainer" containerID="0148cf646e079db62e52d31a8ddc05db1e5e580fbd3a9b7063aeaf4ecec39bb2" Oct 29 23:00:10.132890 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:10.132851 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-27d16a-openshift-default-869755f997-ws2s6"] Oct 29 23:00:10.143564 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:10.143536 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-27d16a-openshift-default-869755f997-ws2s6"] Oct 29 23:00:10.757134 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:10.757088 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:00:10.870292 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:10.870244 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:00:11.342668 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:11.342635 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e6af2d78-ba9f-4f94-a985-47985aa30dd3" path="/var/lib/kubelet/pods/e6af2d78-ba9f-4f94-a985-47985aa30dd3/volumes" Oct 29 23:00:11.587509 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:11.587473 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:00:11.704492 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:11.704438 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:00:11.812379 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:11.811324 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:00:11.932354 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:11.927934 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-cd525f-openshift-default-5567b55479-lhtg7"] Oct 29 23:00:11.932354 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:11.928154 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-cd525f-openshift-default-5567b55479-lhtg7" podUID="3616f622-b298-42ce-bf47-80e53935b0f0" containerName="istio-proxy" containerID="cri-o://3dfbe1f9f3c510824304b205b4edc2da21024f49a1107319eb6078094b500f34" gracePeriod=30 Oct 29 23:00:11.966101 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:11.965989 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:00:13.760612 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:13.760578 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:00:15.790579 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:15.790542 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:00:15.917151 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:15.917108 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:00:16.028667 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:16.028632 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:00:16.774658 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:16.774621 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:00:17.211383 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:17.211348 5057 generic.go:356] "Generic (PLEG): container finished" podID="3616f622-b298-42ce-bf47-80e53935b0f0" containerID="3dfbe1f9f3c510824304b205b4edc2da21024f49a1107319eb6078094b500f34" exitCode=0 Oct 29 23:00:17.211815 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:17.211605 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-cd525f-openshift-default-5567b55479-lhtg7" event={"ID":"3616f622-b298-42ce-bf47-80e53935b0f0","Type":"ContainerDied","Data":"3dfbe1f9f3c510824304b205b4edc2da21024f49a1107319eb6078094b500f34"} Oct 29 23:00:17.271009 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:17.270657 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-cd525f-openshift-default-5567b55479-lhtg7" Oct 29 23:00:17.368973 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:17.368881 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3616f622-b298-42ce-bf47-80e53935b0f0-istio-token\" (UniqueName: \"kubernetes.io/projected/3616f622-b298-42ce-bf47-80e53935b0f0-istio-token\") pod \"3616f622-b298-42ce-bf47-80e53935b0f0\" (UID: \"3616f622-b298-42ce-bf47-80e53935b0f0\") " Oct 29 23:00:17.369152 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:17.369001 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3616f622-b298-42ce-bf47-80e53935b0f0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3616f622-b298-42ce-bf47-80e53935b0f0-credential-socket\") pod \"3616f622-b298-42ce-bf47-80e53935b0f0\" (UID: \"3616f622-b298-42ce-bf47-80e53935b0f0\") " Oct 29 23:00:17.369152 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:17.369055 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3616f622-b298-42ce-bf47-80e53935b0f0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3616f622-b298-42ce-bf47-80e53935b0f0-workload-socket\") pod \"3616f622-b298-42ce-bf47-80e53935b0f0\" (UID: \"3616f622-b298-42ce-bf47-80e53935b0f0\") " Oct 29 23:00:17.369152 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:17.369137 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3616f622-b298-42ce-bf47-80e53935b0f0-kube-api-access-rzfb9\" (UniqueName: \"kubernetes.io/projected/3616f622-b298-42ce-bf47-80e53935b0f0-kube-api-access-rzfb9\") pod \"3616f622-b298-42ce-bf47-80e53935b0f0\" (UID: \"3616f622-b298-42ce-bf47-80e53935b0f0\") " Oct 29 23:00:17.369375 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:17.369208 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3616f622-b298-42ce-bf47-80e53935b0f0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3616f622-b298-42ce-bf47-80e53935b0f0-istiod-ca-cert\") pod \"3616f622-b298-42ce-bf47-80e53935b0f0\" (UID: \"3616f622-b298-42ce-bf47-80e53935b0f0\") " Oct 29 23:00:17.369375 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:17.369269 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3616f622-b298-42ce-bf47-80e53935b0f0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3616f622-b298-42ce-bf47-80e53935b0f0-istio-envoy\") pod \"3616f622-b298-42ce-bf47-80e53935b0f0\" (UID: \"3616f622-b298-42ce-bf47-80e53935b0f0\") " Oct 29 23:00:17.369375 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:17.369312 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3616f622-b298-42ce-bf47-80e53935b0f0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3616f622-b298-42ce-bf47-80e53935b0f0-istio-podinfo\") pod \"3616f622-b298-42ce-bf47-80e53935b0f0\" (UID: \"3616f622-b298-42ce-bf47-80e53935b0f0\") " Oct 29 23:00:17.369550 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:17.369424 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3616f622-b298-42ce-bf47-80e53935b0f0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3616f622-b298-42ce-bf47-80e53935b0f0-workload-certs\") pod \"3616f622-b298-42ce-bf47-80e53935b0f0\" (UID: \"3616f622-b298-42ce-bf47-80e53935b0f0\") " Oct 29 23:00:17.369647 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:17.369629 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3616f622-b298-42ce-bf47-80e53935b0f0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3616f622-b298-42ce-bf47-80e53935b0f0-istio-data\") pod \"3616f622-b298-42ce-bf47-80e53935b0f0\" (UID: \"3616f622-b298-42ce-bf47-80e53935b0f0\") " Oct 29 23:00:17.373469 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:17.373137 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3616f622-b298-42ce-bf47-80e53935b0f0-istio-data" pod "3616f622-b298-42ce-bf47-80e53935b0f0" (UID: "3616f622-b298-42ce-bf47-80e53935b0f0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:00:17.374485 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:17.374191 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3616f622-b298-42ce-bf47-80e53935b0f0-credential-socket" pod "3616f622-b298-42ce-bf47-80e53935b0f0" (UID: "3616f622-b298-42ce-bf47-80e53935b0f0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:00:17.406351 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:17.402412 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3616f622-b298-42ce-bf47-80e53935b0f0-workload-socket" pod "3616f622-b298-42ce-bf47-80e53935b0f0" (UID: "3616f622-b298-42ce-bf47-80e53935b0f0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:00:17.406351 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:17.402444 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3616f622-b298-42ce-bf47-80e53935b0f0-workload-certs" pod "3616f622-b298-42ce-bf47-80e53935b0f0" (UID: "3616f622-b298-42ce-bf47-80e53935b0f0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:00:17.439473 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:17.406831 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3616f622-b298-42ce-bf47-80e53935b0f0-istiod-ca-cert" pod "3616f622-b298-42ce-bf47-80e53935b0f0" (UID: "3616f622-b298-42ce-bf47-80e53935b0f0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:00:17.486356 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:17.470348 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3616f622-b298-42ce-bf47-80e53935b0f0-istio-envoy" pod "3616f622-b298-42ce-bf47-80e53935b0f0" (UID: "3616f622-b298-42ce-bf47-80e53935b0f0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:00:17.486356 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:17.472442 5057 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3616f622-b298-42ce-bf47-80e53935b0f0-credential-socket\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:00:17.486356 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:17.472470 5057 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3616f622-b298-42ce-bf47-80e53935b0f0-workload-socket\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:00:17.486356 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:17.472484 5057 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3616f622-b298-42ce-bf47-80e53935b0f0-istiod-ca-cert\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:00:17.486356 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:17.472498 5057 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3616f622-b298-42ce-bf47-80e53935b0f0-istio-envoy\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:00:17.486356 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:17.472509 5057 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3616f622-b298-42ce-bf47-80e53935b0f0-workload-certs\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:00:17.486356 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:17.472522 5057 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3616f622-b298-42ce-bf47-80e53935b0f0-istio-data\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:00:17.486356 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:17.479020 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:00:17.512977 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:17.512921 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3616f622-b298-42ce-bf47-80e53935b0f0-istio-podinfo" pod "3616f622-b298-42ce-bf47-80e53935b0f0" (UID: "3616f622-b298-42ce-bf47-80e53935b0f0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:00:17.513273 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:17.513246 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3616f622-b298-42ce-bf47-80e53935b0f0-kube-api-access-rzfb9" pod "3616f622-b298-42ce-bf47-80e53935b0f0" (UID: "3616f622-b298-42ce-bf47-80e53935b0f0"). InnerVolumeSpecName "kube-api-access-rzfb9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:00:17.559354 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:17.552604 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3616f622-b298-42ce-bf47-80e53935b0f0-istio-token" pod "3616f622-b298-42ce-bf47-80e53935b0f0" (UID: "3616f622-b298-42ce-bf47-80e53935b0f0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:00:17.574194 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:17.574153 5057 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3616f622-b298-42ce-bf47-80e53935b0f0-istio-token\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:00:17.574390 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:17.574218 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rzfb9\" (UniqueName: \"kubernetes.io/projected/3616f622-b298-42ce-bf47-80e53935b0f0-kube-api-access-rzfb9\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:00:17.574390 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:17.574233 5057 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3616f622-b298-42ce-bf47-80e53935b0f0-istio-podinfo\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:00:18.241433 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:18.232260 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-cd525f-openshift-default-5567b55479-lhtg7" event={"ID":"3616f622-b298-42ce-bf47-80e53935b0f0","Type":"ContainerDied","Data":"5dbcedba39e1debea8d2c66aa06574f5448167c71b5bebca5cf42ecd6e743ff2"} Oct 29 23:00:18.241433 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:18.232358 5057 scope.go:122] "RemoveContainer" containerID="3dfbe1f9f3c510824304b205b4edc2da21024f49a1107319eb6078094b500f34" Oct 29 23:00:18.241433 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:18.232569 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-cd525f-openshift-default-5567b55479-lhtg7" Oct 29 23:00:18.287414 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:18.287384 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:00:18.312372 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:18.312298 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-cd525f-openshift-default-5567b55479-lhtg7"] Oct 29 23:00:18.319981 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:18.319789 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-cd525f-openshift-default-5567b55479-lhtg7"] Oct 29 23:00:18.497306 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:18.497222 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:00:18.990089 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:18.990042 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:00:19.067405 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:00:19.067369 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d2dnn" containerName="extractor" Oct 29 23:00:19.110025 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:19.109977 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:00:19.247792 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:19.247712 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:00:19.333876 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:19.333826 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3616f622-b298-42ce-bf47-80e53935b0f0" path="/var/lib/kubelet/pods/3616f622-b298-42ce-bf47-80e53935b0f0/volumes" Oct 29 23:00:21.320724 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:21.320689 5057 scope.go:122] "RemoveContainer" containerID="652bd7e293937c9cfbd235415cb6a91f6b80534aae6342499eedd28fa3bd6778" Oct 29 23:00:21.718632 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:21.718600 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:00:21.843831 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:21.843796 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:00:22.109894 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:22.109856 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:00:22.285366 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:22.285315 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-0df9b0-29888580-ss4kb" event={"ID":"33a3c7b1-abf7-4cce-b13a-073c96d05d9f","Type":"ContainerStarted","Data":"7b25fb3757a911d1bc0f02ffeeb18630f9e006adca30c343c492ba6807126597"} Oct 29 23:00:22.326300 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:22.326251 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-0df9b0-29888580-ss4kb" podStartSLOduration=19.000500484 podStartE2EDuration="22.32623722s" totalImagesPullingTime="3.325736736s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:00:05.049072748 +0000 UTC m=+1585.832286178" watchObservedRunningTime="2026-10-29 23:00:22.32623722 +0000 UTC m=+1603.109450646" Oct 29 23:00:23.295701 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:23.295667 5057 generic.go:356] "Generic (PLEG): container finished" podID="33a3c7b1-abf7-4cce-b13a-073c96d05d9f" containerID="7b25fb3757a911d1bc0f02ffeeb18630f9e006adca30c343c492ba6807126597" exitCode=60 Oct 29 23:00:23.295895 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:23.295718 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-0df9b0-29888580-ss4kb" event={"ID":"33a3c7b1-abf7-4cce-b13a-073c96d05d9f","Type":"ContainerDied","Data":"7b25fb3757a911d1bc0f02ffeeb18630f9e006adca30c343c492ba6807126597"} Oct 29 23:00:23.295895 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:23.295773 5057 scope.go:122] "RemoveContainer" containerID="652bd7e293937c9cfbd235415cb6a91f6b80534aae6342499eedd28fa3bd6778" Oct 29 23:00:23.296200 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:23.296172 5057 scope.go:122] "RemoveContainer" containerID="7b25fb3757a911d1bc0f02ffeeb18630f9e006adca30c343c492ba6807126597" Oct 29 23:00:23.296618 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:00:23.296595 5057 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-0df9b0-29888580-ss4kb_odh-ai-gateway-infra(33a3c7b1-abf7-4cce-b13a-073c96d05d9f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-0df9b0-29888580-ss4kb" podUID="33a3c7b1-abf7-4cce-b13a-073c96d05d9f" Oct 29 23:00:23.337694 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:23.337666 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-0df9b0-29888580-ss4kb"] Oct 29 23:00:23.462032 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:23.461995 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:00:23.571673 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:23.571585 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:00:23.759528 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:23.759470 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:00:24.157406 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:24.153953 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:00:24.264002 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:24.263968 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:00:24.583928 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:24.583898 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-0df9b0-29888580-ss4kb" Oct 29 23:00:24.655233 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:24.655194 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33a3c7b1-abf7-4cce-b13a-073c96d05d9f-kube-api-access-rxg5h\" (UniqueName: \"kubernetes.io/projected/33a3c7b1-abf7-4cce-b13a-073c96d05d9f-kube-api-access-rxg5h\") pod \"33a3c7b1-abf7-4cce-b13a-073c96d05d9f\" (UID: \"33a3c7b1-abf7-4cce-b13a-073c96d05d9f\") " Oct 29 23:00:24.655466 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:24.655263 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33a3c7b1-abf7-4cce-b13a-073c96d05d9f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/33a3c7b1-abf7-4cce-b13a-073c96d05d9f-maas-api-service-ca\") pod \"33a3c7b1-abf7-4cce-b13a-073c96d05d9f\" (UID: \"33a3c7b1-abf7-4cce-b13a-073c96d05d9f\") " Oct 29 23:00:24.656363 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:24.656308 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33a3c7b1-abf7-4cce-b13a-073c96d05d9f-maas-api-service-ca" pod "33a3c7b1-abf7-4cce-b13a-073c96d05d9f" (UID: "33a3c7b1-abf7-4cce-b13a-073c96d05d9f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:00:24.656520 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:24.656457 5057 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/33a3c7b1-abf7-4cce-b13a-073c96d05d9f-maas-api-service-ca\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:00:24.658951 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:24.658920 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33a3c7b1-abf7-4cce-b13a-073c96d05d9f-kube-api-access-rxg5h" pod "33a3c7b1-abf7-4cce-b13a-073c96d05d9f" (UID: "33a3c7b1-abf7-4cce-b13a-073c96d05d9f"). InnerVolumeSpecName "kube-api-access-rxg5h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:00:24.758137 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:24.758102 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rxg5h\" (UniqueName: \"kubernetes.io/projected/33a3c7b1-abf7-4cce-b13a-073c96d05d9f-kube-api-access-rxg5h\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:00:25.322355 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:25.320251 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-0df9b0-29888580-ss4kb" Oct 29 23:00:25.341975 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:25.341931 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-0df9b0-29888580-ss4kb" event={"ID":"33a3c7b1-abf7-4cce-b13a-073c96d05d9f","Type":"ContainerDied","Data":"550b380d9c366cc0d850f85881f04ca2cfccd1a16d2a60dee945aedfd0391303"} Oct 29 23:00:25.341975 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:25.341978 5057 scope.go:122] "RemoveContainer" containerID="7b25fb3757a911d1bc0f02ffeeb18630f9e006adca30c343c492ba6807126597" Oct 29 23:00:25.378508 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:25.378478 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-0df9b0-29888580-ss4kb"] Oct 29 23:00:25.383053 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:25.383029 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-0df9b0-29888580-ss4kb"] Oct 29 23:00:26.108530 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:26.108482 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:00:27.334853 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:27.334814 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33a3c7b1-abf7-4cce-b13a-073c96d05d9f" path="/var/lib/kubelet/pods/33a3c7b1-abf7-4cce-b13a-073c96d05d9f/volumes" Oct 29 23:00:27.408434 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:27.408395 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-a0dbf5-84c95d7876-qghzq"] Oct 29 23:00:27.408666 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:27.408639 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a0dbf5-84c95d7876-qghzq" podUID="70be02c2-cd2f-43dc-b689-3e088d9ebe10" containerName="maas-api" containerID="cri-o://e7fd367a9f94477df8aa8f79f20493c3fcff59d46e9ae3a5ac0addfa6425e131" gracePeriod=30 Oct 29 23:00:27.467447 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:27.467418 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:00:27.823538 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:27.823125 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:00:27.879659 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:27.879577 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a0dbf5-84c95d7876-qghzq" Oct 29 23:00:27.930780 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:27.930747 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70be02c2-cd2f-43dc-b689-3e088d9ebe10-kube-api-access-hs9r2\" (UniqueName: \"kubernetes.io/projected/70be02c2-cd2f-43dc-b689-3e088d9ebe10-kube-api-access-hs9r2\") pod \"70be02c2-cd2f-43dc-b689-3e088d9ebe10\" (UID: \"70be02c2-cd2f-43dc-b689-3e088d9ebe10\") " Oct 29 23:00:27.930975 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:27.930860 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/70be02c2-cd2f-43dc-b689-3e088d9ebe10-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/70be02c2-cd2f-43dc-b689-3e088d9ebe10-openshift-service-ca\") pod \"70be02c2-cd2f-43dc-b689-3e088d9ebe10\" (UID: \"70be02c2-cd2f-43dc-b689-3e088d9ebe10\") " Oct 29 23:00:27.930975 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:27.930919 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/70be02c2-cd2f-43dc-b689-3e088d9ebe10-metrics-certs\" (UniqueName: \"kubernetes.io/secret/70be02c2-cd2f-43dc-b689-3e088d9ebe10-metrics-certs\") pod \"70be02c2-cd2f-43dc-b689-3e088d9ebe10\" (UID: \"70be02c2-cd2f-43dc-b689-3e088d9ebe10\") " Oct 29 23:00:27.931263 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:27.931244 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/70be02c2-cd2f-43dc-b689-3e088d9ebe10-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/70be02c2-cd2f-43dc-b689-3e088d9ebe10-maas-api-tls\") pod \"70be02c2-cd2f-43dc-b689-3e088d9ebe10\" (UID: \"70be02c2-cd2f-43dc-b689-3e088d9ebe10\") " Oct 29 23:00:27.933621 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:27.933592 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70be02c2-cd2f-43dc-b689-3e088d9ebe10-openshift-service-ca" pod "70be02c2-cd2f-43dc-b689-3e088d9ebe10" (UID: "70be02c2-cd2f-43dc-b689-3e088d9ebe10"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:00:27.935103 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:27.935073 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70be02c2-cd2f-43dc-b689-3e088d9ebe10-metrics-certs" pod "70be02c2-cd2f-43dc-b689-3e088d9ebe10" (UID: "70be02c2-cd2f-43dc-b689-3e088d9ebe10"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:00:27.935479 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:27.935433 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70be02c2-cd2f-43dc-b689-3e088d9ebe10-kube-api-access-hs9r2" pod "70be02c2-cd2f-43dc-b689-3e088d9ebe10" (UID: "70be02c2-cd2f-43dc-b689-3e088d9ebe10"). InnerVolumeSpecName "kube-api-access-hs9r2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:00:27.935479 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:27.935440 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70be02c2-cd2f-43dc-b689-3e088d9ebe10-maas-api-tls" pod "70be02c2-cd2f-43dc-b689-3e088d9ebe10" (UID: "70be02c2-cd2f-43dc-b689-3e088d9ebe10"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:00:28.036571 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:28.036028 5057 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/70be02c2-cd2f-43dc-b689-3e088d9ebe10-maas-api-tls\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:00:28.036571 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:28.036070 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hs9r2\" (UniqueName: \"kubernetes.io/projected/70be02c2-cd2f-43dc-b689-3e088d9ebe10-kube-api-access-hs9r2\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:00:28.036571 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:28.036086 5057 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/70be02c2-cd2f-43dc-b689-3e088d9ebe10-openshift-service-ca\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:00:28.036571 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:28.036100 5057 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70be02c2-cd2f-43dc-b689-3e088d9ebe10-metrics-certs\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:00:28.062409 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:28.062370 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:00:28.173301 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:28.173210 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:00:28.353441 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:28.353403 5057 generic.go:356] "Generic (PLEG): container finished" podID="70be02c2-cd2f-43dc-b689-3e088d9ebe10" containerID="e7fd367a9f94477df8aa8f79f20493c3fcff59d46e9ae3a5ac0addfa6425e131" exitCode=0 Oct 29 23:00:28.353441 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:28.353446 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a0dbf5-84c95d7876-qghzq" Oct 29 23:00:28.353958 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:28.353493 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a0dbf5-84c95d7876-qghzq" event={"ID":"70be02c2-cd2f-43dc-b689-3e088d9ebe10","Type":"ContainerDied","Data":"e7fd367a9f94477df8aa8f79f20493c3fcff59d46e9ae3a5ac0addfa6425e131"} Oct 29 23:00:28.353958 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:28.353551 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a0dbf5-84c95d7876-qghzq" event={"ID":"70be02c2-cd2f-43dc-b689-3e088d9ebe10","Type":"ContainerDied","Data":"f5f36edca6af26be61bdbc7c4a4cd16b341ab195a2c69eecde23f08b2b680c7f"} Oct 29 23:00:28.353958 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:28.353581 5057 scope.go:122] "RemoveContainer" containerID="e7fd367a9f94477df8aa8f79f20493c3fcff59d46e9ae3a5ac0addfa6425e131" Oct 29 23:00:28.366358 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:28.366325 5057 scope.go:122] "RemoveContainer" containerID="e7fd367a9f94477df8aa8f79f20493c3fcff59d46e9ae3a5ac0addfa6425e131" Oct 29 23:00:28.366696 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:00:28.366673 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7fd367a9f94477df8aa8f79f20493c3fcff59d46e9ae3a5ac0addfa6425e131\": container with ID starting with e7fd367a9f94477df8aa8f79f20493c3fcff59d46e9ae3a5ac0addfa6425e131 not found: ID does not exist" containerID="e7fd367a9f94477df8aa8f79f20493c3fcff59d46e9ae3a5ac0addfa6425e131" Oct 29 23:00:28.366815 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:28.366704 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7fd367a9f94477df8aa8f79f20493c3fcff59d46e9ae3a5ac0addfa6425e131"} err="failed to get container status \"e7fd367a9f94477df8aa8f79f20493c3fcff59d46e9ae3a5ac0addfa6425e131\": rpc error: code = NotFound desc = could not find container \"e7fd367a9f94477df8aa8f79f20493c3fcff59d46e9ae3a5ac0addfa6425e131\": container with ID starting with e7fd367a9f94477df8aa8f79f20493c3fcff59d46e9ae3a5ac0addfa6425e131 not found: ID does not exist" Oct 29 23:00:28.406023 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:28.405938 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-a0dbf5-84c95d7876-qghzq"] Oct 29 23:00:28.415102 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:28.415073 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-a0dbf5-84c95d7876-qghzq"] Oct 29 23:00:28.685443 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:28.685405 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:00:29.141778 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:29.141733 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:00:29.272701 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:29.272670 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:00:29.332968 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:29.332933 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="70be02c2-cd2f-43dc-b689-3e088d9ebe10" path="/var/lib/kubelet/pods/70be02c2-cd2f-43dc-b689-3e088d9ebe10/volumes" Oct 29 23:00:29.382383 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:29.379915 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:00:29.799768 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:29.799723 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:00:30.990724 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:30.990666 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:00:32.177497 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:32.177462 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:00:32.484969 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:32.484935 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:00:33.909393 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:33.909353 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:00:34.811506 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:34.811472 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:00:34.920390 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:34.920316 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:00:35.146878 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:35.146793 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:00:35.879132 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:35.879095 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:00:36.127633 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:36.127604 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:00:36.619985 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:36.619903 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:00:36.748236 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:36.748204 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:00:37.740259 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:37.740223 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:00:37.869602 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:37.869561 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:00:38.230276 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:38.230235 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:00:38.867036 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:38.866999 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v2tfm"] Oct 29 23:00:39.799033 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:39.798988 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:00:41.102090 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:41.102051 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:00:41.311228 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:41.311191 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:00:41.433434 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:41.433263 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:00:42.970744 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:42.970707 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:00:44.143793 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:44.143756 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:00:44.273472 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:44.273437 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:00:44.382696 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:44.382656 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:00:44.456842 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:44.456809 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7d18f7-fbbf8944c-wjdkg"] Oct 29 23:00:44.457770 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:44.457752 5057 cpu_manager.go:414] "containerMap: removing container" podUID="33a3c7b1-abf7-4cce-b13a-073c96d05d9f" containerName="cleanup" Oct 29 23:00:44.457825 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:44.457773 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33a3c7b1-abf7-4cce-b13a-073c96d05d9f" containerName="cleanup" Oct 29 23:00:44.457825 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:44.457799 5057 cpu_manager.go:414] "containerMap: removing container" podUID="70be02c2-cd2f-43dc-b689-3e088d9ebe10" containerName="maas-api" Oct 29 23:00:44.457825 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:44.457804 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70be02c2-cd2f-43dc-b689-3e088d9ebe10" containerName="maas-api" Oct 29 23:00:44.457825 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:44.457820 5057 cpu_manager.go:414] "containerMap: removing container" podUID="3616f622-b298-42ce-bf47-80e53935b0f0" containerName="istio-proxy" Oct 29 23:00:44.457825 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:44.457826 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3616f622-b298-42ce-bf47-80e53935b0f0" containerName="istio-proxy" Oct 29 23:00:44.457984 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:44.457842 5057 cpu_manager.go:414] "containerMap: removing container" podUID="e6af2d78-ba9f-4f94-a985-47985aa30dd3" containerName="istio-proxy" Oct 29 23:00:44.457984 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:44.457846 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6af2d78-ba9f-4f94-a985-47985aa30dd3" containerName="istio-proxy" Oct 29 23:00:44.457984 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:44.457888 5057 cpu_manager.go:414] "containerMap: removing container" podUID="33a3c7b1-abf7-4cce-b13a-073c96d05d9f" containerName="cleanup" Oct 29 23:00:44.457984 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:44.457894 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33a3c7b1-abf7-4cce-b13a-073c96d05d9f" containerName="cleanup" Oct 29 23:00:44.458103 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:44.458019 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="70be02c2-cd2f-43dc-b689-3e088d9ebe10" containerName="maas-api" Oct 29 23:00:44.458103 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:44.458030 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="3616f622-b298-42ce-bf47-80e53935b0f0" containerName="istio-proxy" Oct 29 23:00:44.458103 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:44.458039 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6af2d78-ba9f-4f94-a985-47985aa30dd3" containerName="istio-proxy" Oct 29 23:00:44.458103 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:44.458045 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="33a3c7b1-abf7-4cce-b13a-073c96d05d9f" containerName="cleanup" Oct 29 23:00:44.458103 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:44.458083 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="33a3c7b1-abf7-4cce-b13a-073c96d05d9f" containerName="cleanup" Oct 29 23:00:44.466382 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:44.466355 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7d18f7-fbbf8944c-wjdkg" Oct 29 23:00:44.485489 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:44.484865 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7d18f7-fbbf8944c-wjdkg"] Oct 29 23:00:44.485489 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:44.484971 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-7d18f7-dockercfg-twz9f\"" Oct 29 23:00:44.485489 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:44.484904 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-7d18f7\"" Oct 29 23:00:44.600587 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:44.598739 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5gm8\" (UniqueName: \"kubernetes.io/projected/53fb3a29-fc2d-4454-ac89-8b94c4d065a3-kube-api-access-w5gm8\") pod \"payload-pre-processing-e2e-ipp-a-7d18f7-fbbf8944c-wjdkg\" (UID: \"53fb3a29-fc2d-4454-ac89-8b94c4d065a3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7d18f7-fbbf8944c-wjdkg" Oct 29 23:00:44.600587 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:44.599056 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53fb3a29-fc2d-4454-ac89-8b94c4d065a3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-7d18f7-fbbf8944c-wjdkg\" (UID: \"53fb3a29-fc2d-4454-ac89-8b94c4d065a3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7d18f7-fbbf8944c-wjdkg" Oct 29 23:00:44.624771 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:44.624737 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7d18f7-f8648fdc4-5wqsx"] Oct 29 23:00:44.625909 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:44.625886 5057 cpu_manager.go:414] "containerMap: removing container" podUID="33a3c7b1-abf7-4cce-b13a-073c96d05d9f" containerName="cleanup" Oct 29 23:00:44.625909 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:44.625908 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33a3c7b1-abf7-4cce-b13a-073c96d05d9f" containerName="cleanup" Oct 29 23:00:44.626172 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:44.626156 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="33a3c7b1-abf7-4cce-b13a-073c96d05d9f" containerName="cleanup" Oct 29 23:00:44.629322 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:44.629298 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-7d18f7-f8648fdc4-5wqsx" Oct 29 23:00:44.641611 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:44.640702 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7d18f7-f8648fdc4-5wqsx"] Oct 29 23:00:44.700946 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:44.700905 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5gm8\" (UniqueName: \"kubernetes.io/projected/53fb3a29-fc2d-4454-ac89-8b94c4d065a3-kube-api-access-w5gm8\") pod \"payload-pre-processing-e2e-ipp-a-7d18f7-fbbf8944c-wjdkg\" (UID: \"53fb3a29-fc2d-4454-ac89-8b94c4d065a3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7d18f7-fbbf8944c-wjdkg" Oct 29 23:00:44.701494 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:44.701471 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53fb3a29-fc2d-4454-ac89-8b94c4d065a3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-7d18f7-fbbf8944c-wjdkg\" (UID: \"53fb3a29-fc2d-4454-ac89-8b94c4d065a3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7d18f7-fbbf8944c-wjdkg" Oct 29 23:00:44.703529 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:44.703497 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53fb3a29-fc2d-4454-ac89-8b94c4d065a3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-7d18f7-fbbf8944c-wjdkg\" (UID: \"53fb3a29-fc2d-4454-ac89-8b94c4d065a3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7d18f7-fbbf8944c-wjdkg" Oct 29 23:00:44.727042 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:44.726981 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5gm8\" (UniqueName: \"kubernetes.io/projected/53fb3a29-fc2d-4454-ac89-8b94c4d065a3-kube-api-access-w5gm8\") pod \"payload-pre-processing-e2e-ipp-a-7d18f7-fbbf8944c-wjdkg\" (UID: \"53fb3a29-fc2d-4454-ac89-8b94c4d065a3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7d18f7-fbbf8944c-wjdkg" Oct 29 23:00:44.788093 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:44.788064 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7d18f7-fbbf8944c-wjdkg" Oct 29 23:00:44.803373 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:44.803322 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gggnv\" (UniqueName: \"kubernetes.io/projected/d88241f0-29ac-41ef-bb25-c68e6e3b55e7-kube-api-access-gggnv\") pod \"payload-processing-e2e-ipp-a-7d18f7-f8648fdc4-5wqsx\" (UID: \"d88241f0-29ac-41ef-bb25-c68e6e3b55e7\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7d18f7-f8648fdc4-5wqsx" Oct 29 23:00:44.803611 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:44.803584 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d88241f0-29ac-41ef-bb25-c68e6e3b55e7-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-7d18f7-f8648fdc4-5wqsx\" (UID: \"d88241f0-29ac-41ef-bb25-c68e6e3b55e7\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7d18f7-f8648fdc4-5wqsx" Oct 29 23:00:44.911355 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:44.906178 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gggnv\" (UniqueName: \"kubernetes.io/projected/d88241f0-29ac-41ef-bb25-c68e6e3b55e7-kube-api-access-gggnv\") pod \"payload-processing-e2e-ipp-a-7d18f7-f8648fdc4-5wqsx\" (UID: \"d88241f0-29ac-41ef-bb25-c68e6e3b55e7\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7d18f7-f8648fdc4-5wqsx" Oct 29 23:00:44.911355 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:44.906409 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d88241f0-29ac-41ef-bb25-c68e6e3b55e7-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-7d18f7-f8648fdc4-5wqsx\" (UID: \"d88241f0-29ac-41ef-bb25-c68e6e3b55e7\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7d18f7-f8648fdc4-5wqsx" Oct 29 23:00:44.911355 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:44.908380 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d88241f0-29ac-41ef-bb25-c68e6e3b55e7-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-7d18f7-f8648fdc4-5wqsx\" (UID: \"d88241f0-29ac-41ef-bb25-c68e6e3b55e7\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7d18f7-f8648fdc4-5wqsx" Oct 29 23:00:44.946415 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:44.912082 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:00:45.001756 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:45.001092 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gggnv\" (UniqueName: \"kubernetes.io/projected/d88241f0-29ac-41ef-bb25-c68e6e3b55e7-kube-api-access-gggnv\") pod \"payload-processing-e2e-ipp-a-7d18f7-f8648fdc4-5wqsx\" (UID: \"d88241f0-29ac-41ef-bb25-c68e6e3b55e7\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7d18f7-f8648fdc4-5wqsx" Oct 29 23:00:45.032133 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:45.032093 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:00:45.174633 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:45.174594 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7d18f7-fbbf8944c-wjdkg"] Oct 29 23:00:45.249354 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:45.249194 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-7d18f7-f8648fdc4-5wqsx" Oct 29 23:00:45.581997 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:45.581968 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7d18f7-f8648fdc4-5wqsx"] Oct 29 23:00:45.584714 ip-10-0-140-74 kubenswrapper[5057]: W1029 23:00:45.584684 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd88241f0_29ac_41ef_bb25_c68e6e3b55e7.slice/crio-d4a29eddaa5f19267b07611085dac224df36bce3bed2c531642e97646c04369c WatchSource:0}: Error finding container d4a29eddaa5f19267b07611085dac224df36bce3bed2c531642e97646c04369c: Status 404 returned error can't find the container with id d4a29eddaa5f19267b07611085dac224df36bce3bed2c531642e97646c04369c Oct 29 23:00:45.647383 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:45.645206 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-7d18f7-f8648fdc4-5wqsx" event={"ID":"d88241f0-29ac-41ef-bb25-c68e6e3b55e7","Type":"ContainerStarted","Data":"d4a29eddaa5f19267b07611085dac224df36bce3bed2c531642e97646c04369c"} Oct 29 23:00:45.647547 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:45.647407 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7d18f7-fbbf8944c-wjdkg" event={"ID":"53fb3a29-fc2d-4454-ac89-8b94c4d065a3","Type":"ContainerStarted","Data":"6ee15104fd0c24f4d70a8e5ff3dd7deea8f5cbda87042bbb222e4975b19fe304"} Oct 29 23:00:45.647547 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:45.647457 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7d18f7-fbbf8944c-wjdkg" event={"ID":"53fb3a29-fc2d-4454-ac89-8b94c4d065a3","Type":"ContainerStarted","Data":"76ace6e4a8e4db6460b76cfe9026f268191f20909ff7c40f0a283f3da8775077"} Oct 29 23:00:45.647884 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:45.647857 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7d18f7-fbbf8944c-wjdkg" Oct 29 23:00:45.665497 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:45.665467 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:00:45.690318 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:45.690269 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7d18f7-fbbf8944c-wjdkg" podStartSLOduration=1.690252375 podStartE2EDuration="1.690252375s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:00:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:00:45.685674431 +0000 UTC m=+1626.468887858" watchObservedRunningTime="2026-10-29 23:00:45.690252375 +0000 UTC m=+1626.473465802" Oct 29 23:00:46.657962 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:46.657925 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-7d18f7-f8648fdc4-5wqsx" event={"ID":"d88241f0-29ac-41ef-bb25-c68e6e3b55e7","Type":"ContainerStarted","Data":"39adbb610bbed03f3133397953bad9357f2ceeb1457c93af26fb8b53547b0544"} Oct 29 23:00:46.658467 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:46.658255 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-7d18f7-f8648fdc4-5wqsx" Oct 29 23:00:46.697413 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:46.697358 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-7d18f7-f8648fdc4-5wqsx" podStartSLOduration=2.69732378 podStartE2EDuration="2.69732378s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:00:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:00:46.69528651 +0000 UTC m=+1627.478499949" watchObservedRunningTime="2026-10-29 23:00:46.69732378 +0000 UTC m=+1627.480537206" Oct 29 23:00:47.566308 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:47.566269 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:00:49.070472 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:00:49.070429 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d2dnn" containerName="extractor" Oct 29 23:00:49.266446 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:49.266354 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:00:51.703520 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:51.703481 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:00:51.813541 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:51.813500 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:00:52.608925 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:52.608874 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:00:52.729554 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:52.729515 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:00:52.846111 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:52.846073 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:00:54.326090 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:54.326054 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:00:55.942058 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:55.942021 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:00:56.662050 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:56.662023 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7d18f7-fbbf8944c-wjdkg" Oct 29 23:00:57.157364 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:57.157320 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-13aa41-openshift-default-59db78b859-tk95q"] Oct 29 23:00:57.161804 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:57.161784 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-13aa41-openshift-default-59db78b859-tk95q" Oct 29 23:00:57.203469 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:57.203433 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-13aa41-openshift-default-59db78b859-tk95q"] Oct 29 23:00:57.226893 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:57.226850 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc933bed-efee-4aad-9d65-8ee1d04f846b-workload-socket\") pod \"e2e-ipp-b-13aa41-openshift-default-59db78b859-tk95q\" (UID: \"cc933bed-efee-4aad-9d65-8ee1d04f846b\") " pod="openshift-ingress/e2e-ipp-b-13aa41-openshift-default-59db78b859-tk95q" Oct 29 23:00:57.227076 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:57.226921 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc933bed-efee-4aad-9d65-8ee1d04f846b-istiod-ca-cert\") pod \"e2e-ipp-b-13aa41-openshift-default-59db78b859-tk95q\" (UID: \"cc933bed-efee-4aad-9d65-8ee1d04f846b\") " pod="openshift-ingress/e2e-ipp-b-13aa41-openshift-default-59db78b859-tk95q" Oct 29 23:00:57.227155 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:57.227062 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4gs6\" (UniqueName: \"kubernetes.io/projected/cc933bed-efee-4aad-9d65-8ee1d04f846b-kube-api-access-b4gs6\") pod \"e2e-ipp-b-13aa41-openshift-default-59db78b859-tk95q\" (UID: \"cc933bed-efee-4aad-9d65-8ee1d04f846b\") " pod="openshift-ingress/e2e-ipp-b-13aa41-openshift-default-59db78b859-tk95q" Oct 29 23:00:57.227155 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:57.227143 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc933bed-efee-4aad-9d65-8ee1d04f846b-istio-podinfo\") pod \"e2e-ipp-b-13aa41-openshift-default-59db78b859-tk95q\" (UID: \"cc933bed-efee-4aad-9d65-8ee1d04f846b\") " pod="openshift-ingress/e2e-ipp-b-13aa41-openshift-default-59db78b859-tk95q" Oct 29 23:00:57.227307 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:57.227284 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc933bed-efee-4aad-9d65-8ee1d04f846b-istio-token\") pod \"e2e-ipp-b-13aa41-openshift-default-59db78b859-tk95q\" (UID: \"cc933bed-efee-4aad-9d65-8ee1d04f846b\") " pod="openshift-ingress/e2e-ipp-b-13aa41-openshift-default-59db78b859-tk95q" Oct 29 23:00:57.227418 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:57.227392 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc933bed-efee-4aad-9d65-8ee1d04f846b-istio-envoy\") pod \"e2e-ipp-b-13aa41-openshift-default-59db78b859-tk95q\" (UID: \"cc933bed-efee-4aad-9d65-8ee1d04f846b\") " pod="openshift-ingress/e2e-ipp-b-13aa41-openshift-default-59db78b859-tk95q" Oct 29 23:00:57.227480 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:57.227471 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc933bed-efee-4aad-9d65-8ee1d04f846b-istio-data\") pod \"e2e-ipp-b-13aa41-openshift-default-59db78b859-tk95q\" (UID: \"cc933bed-efee-4aad-9d65-8ee1d04f846b\") " pod="openshift-ingress/e2e-ipp-b-13aa41-openshift-default-59db78b859-tk95q" Oct 29 23:00:57.227543 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:57.227512 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc933bed-efee-4aad-9d65-8ee1d04f846b-workload-certs\") pod \"e2e-ipp-b-13aa41-openshift-default-59db78b859-tk95q\" (UID: \"cc933bed-efee-4aad-9d65-8ee1d04f846b\") " pod="openshift-ingress/e2e-ipp-b-13aa41-openshift-default-59db78b859-tk95q" Oct 29 23:00:57.227601 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:57.227565 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc933bed-efee-4aad-9d65-8ee1d04f846b-credential-socket\") pod \"e2e-ipp-b-13aa41-openshift-default-59db78b859-tk95q\" (UID: \"cc933bed-efee-4aad-9d65-8ee1d04f846b\") " pod="openshift-ingress/e2e-ipp-b-13aa41-openshift-default-59db78b859-tk95q" Oct 29 23:00:57.257251 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:57.257221 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-13aa41-openshift-default-dockercfg-2zxdv\"" Oct 29 23:00:57.329683 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:57.329646 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc933bed-efee-4aad-9d65-8ee1d04f846b-workload-socket\") pod \"e2e-ipp-b-13aa41-openshift-default-59db78b859-tk95q\" (UID: \"cc933bed-efee-4aad-9d65-8ee1d04f846b\") " pod="openshift-ingress/e2e-ipp-b-13aa41-openshift-default-59db78b859-tk95q" Oct 29 23:00:57.329865 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:57.329796 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc933bed-efee-4aad-9d65-8ee1d04f846b-istiod-ca-cert\") pod \"e2e-ipp-b-13aa41-openshift-default-59db78b859-tk95q\" (UID: \"cc933bed-efee-4aad-9d65-8ee1d04f846b\") " pod="openshift-ingress/e2e-ipp-b-13aa41-openshift-default-59db78b859-tk95q" Oct 29 23:00:57.329941 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:57.329863 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4gs6\" (UniqueName: \"kubernetes.io/projected/cc933bed-efee-4aad-9d65-8ee1d04f846b-kube-api-access-b4gs6\") pod \"e2e-ipp-b-13aa41-openshift-default-59db78b859-tk95q\" (UID: \"cc933bed-efee-4aad-9d65-8ee1d04f846b\") " pod="openshift-ingress/e2e-ipp-b-13aa41-openshift-default-59db78b859-tk95q" Oct 29 23:00:57.329941 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:57.329919 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc933bed-efee-4aad-9d65-8ee1d04f846b-istio-podinfo\") pod \"e2e-ipp-b-13aa41-openshift-default-59db78b859-tk95q\" (UID: \"cc933bed-efee-4aad-9d65-8ee1d04f846b\") " pod="openshift-ingress/e2e-ipp-b-13aa41-openshift-default-59db78b859-tk95q" Oct 29 23:00:57.330060 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:57.329974 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc933bed-efee-4aad-9d65-8ee1d04f846b-istio-token\") pod \"e2e-ipp-b-13aa41-openshift-default-59db78b859-tk95q\" (UID: \"cc933bed-efee-4aad-9d65-8ee1d04f846b\") " pod="openshift-ingress/e2e-ipp-b-13aa41-openshift-default-59db78b859-tk95q" Oct 29 23:00:57.330060 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:57.330032 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc933bed-efee-4aad-9d65-8ee1d04f846b-istio-envoy\") pod \"e2e-ipp-b-13aa41-openshift-default-59db78b859-tk95q\" (UID: \"cc933bed-efee-4aad-9d65-8ee1d04f846b\") " pod="openshift-ingress/e2e-ipp-b-13aa41-openshift-default-59db78b859-tk95q" Oct 29 23:00:57.330175 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:57.330080 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc933bed-efee-4aad-9d65-8ee1d04f846b-workload-socket\") pod \"e2e-ipp-b-13aa41-openshift-default-59db78b859-tk95q\" (UID: \"cc933bed-efee-4aad-9d65-8ee1d04f846b\") " pod="openshift-ingress/e2e-ipp-b-13aa41-openshift-default-59db78b859-tk95q" Oct 29 23:00:57.330175 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:57.330146 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc933bed-efee-4aad-9d65-8ee1d04f846b-istio-data\") pod \"e2e-ipp-b-13aa41-openshift-default-59db78b859-tk95q\" (UID: \"cc933bed-efee-4aad-9d65-8ee1d04f846b\") " pod="openshift-ingress/e2e-ipp-b-13aa41-openshift-default-59db78b859-tk95q" Oct 29 23:00:57.330284 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:57.330206 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc933bed-efee-4aad-9d65-8ee1d04f846b-workload-certs\") pod \"e2e-ipp-b-13aa41-openshift-default-59db78b859-tk95q\" (UID: \"cc933bed-efee-4aad-9d65-8ee1d04f846b\") " pod="openshift-ingress/e2e-ipp-b-13aa41-openshift-default-59db78b859-tk95q" Oct 29 23:00:57.330378 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:57.330286 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc933bed-efee-4aad-9d65-8ee1d04f846b-credential-socket\") pod \"e2e-ipp-b-13aa41-openshift-default-59db78b859-tk95q\" (UID: \"cc933bed-efee-4aad-9d65-8ee1d04f846b\") " pod="openshift-ingress/e2e-ipp-b-13aa41-openshift-default-59db78b859-tk95q" Oct 29 23:00:57.330562 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:57.330537 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc933bed-efee-4aad-9d65-8ee1d04f846b-istiod-ca-cert\") pod \"e2e-ipp-b-13aa41-openshift-default-59db78b859-tk95q\" (UID: \"cc933bed-efee-4aad-9d65-8ee1d04f846b\") " pod="openshift-ingress/e2e-ipp-b-13aa41-openshift-default-59db78b859-tk95q" Oct 29 23:00:57.330741 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:57.330604 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc933bed-efee-4aad-9d65-8ee1d04f846b-istio-data\") pod \"e2e-ipp-b-13aa41-openshift-default-59db78b859-tk95q\" (UID: \"cc933bed-efee-4aad-9d65-8ee1d04f846b\") " pod="openshift-ingress/e2e-ipp-b-13aa41-openshift-default-59db78b859-tk95q" Oct 29 23:00:57.330960 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:57.330937 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc933bed-efee-4aad-9d65-8ee1d04f846b-workload-certs\") pod \"e2e-ipp-b-13aa41-openshift-default-59db78b859-tk95q\" (UID: \"cc933bed-efee-4aad-9d65-8ee1d04f846b\") " pod="openshift-ingress/e2e-ipp-b-13aa41-openshift-default-59db78b859-tk95q" Oct 29 23:00:57.331140 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:57.331120 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc933bed-efee-4aad-9d65-8ee1d04f846b-credential-socket\") pod \"e2e-ipp-b-13aa41-openshift-default-59db78b859-tk95q\" (UID: \"cc933bed-efee-4aad-9d65-8ee1d04f846b\") " pod="openshift-ingress/e2e-ipp-b-13aa41-openshift-default-59db78b859-tk95q" Oct 29 23:00:57.332488 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:57.332466 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc933bed-efee-4aad-9d65-8ee1d04f846b-istio-envoy\") pod \"e2e-ipp-b-13aa41-openshift-default-59db78b859-tk95q\" (UID: \"cc933bed-efee-4aad-9d65-8ee1d04f846b\") " pod="openshift-ingress/e2e-ipp-b-13aa41-openshift-default-59db78b859-tk95q" Oct 29 23:00:57.332705 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:57.332686 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc933bed-efee-4aad-9d65-8ee1d04f846b-istio-podinfo\") pod \"e2e-ipp-b-13aa41-openshift-default-59db78b859-tk95q\" (UID: \"cc933bed-efee-4aad-9d65-8ee1d04f846b\") " pod="openshift-ingress/e2e-ipp-b-13aa41-openshift-default-59db78b859-tk95q" Oct 29 23:00:57.351707 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:57.351683 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc933bed-efee-4aad-9d65-8ee1d04f846b-istio-token\") pod \"e2e-ipp-b-13aa41-openshift-default-59db78b859-tk95q\" (UID: \"cc933bed-efee-4aad-9d65-8ee1d04f846b\") " pod="openshift-ingress/e2e-ipp-b-13aa41-openshift-default-59db78b859-tk95q" Oct 29 23:00:57.352236 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:57.352213 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4gs6\" (UniqueName: \"kubernetes.io/projected/cc933bed-efee-4aad-9d65-8ee1d04f846b-kube-api-access-b4gs6\") pod \"e2e-ipp-b-13aa41-openshift-default-59db78b859-tk95q\" (UID: \"cc933bed-efee-4aad-9d65-8ee1d04f846b\") " pod="openshift-ingress/e2e-ipp-b-13aa41-openshift-default-59db78b859-tk95q" Oct 29 23:00:57.501812 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:57.501777 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-13aa41-openshift-default-59db78b859-tk95q" Oct 29 23:00:57.670492 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:57.670459 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-7d18f7-f8648fdc4-5wqsx" Oct 29 23:00:57.834932 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:57.834827 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-13aa41-openshift-default-59db78b859-tk95q"] Oct 29 23:00:57.842895 ip-10-0-140-74 kubenswrapper[5057]: W1029 23:00:57.842870 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc933bed_efee_4aad_9d65_8ee1d04f846b.slice/crio-3abc7f2f46c87edaa77f7f9161e64eeb22f752e38d8da4018a3e1d9b1c2b019f WatchSource:0}: Error finding container 3abc7f2f46c87edaa77f7f9161e64eeb22f752e38d8da4018a3e1d9b1c2b019f: Status 404 returned error can't find the container with id 3abc7f2f46c87edaa77f7f9161e64eeb22f752e38d8da4018a3e1d9b1c2b019f Oct 29 23:00:57.845981 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:57.845943 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 23:00:57.846092 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:57.846032 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 23:00:57.846092 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:57.846077 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 23:00:58.211442 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:58.211398 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:00:58.778692 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:58.778652 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-13aa41-openshift-default-59db78b859-tk95q" event={"ID":"cc933bed-efee-4aad-9d65-8ee1d04f846b","Type":"ContainerStarted","Data":"ed28fea70905e1d161ee8c13b156fc5a439646b643dd661c4a7dc6dc56c78747"} Oct 29 23:00:58.778974 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:58.778951 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-13aa41-openshift-default-59db78b859-tk95q" event={"ID":"cc933bed-efee-4aad-9d65-8ee1d04f846b","Type":"ContainerStarted","Data":"3abc7f2f46c87edaa77f7f9161e64eeb22f752e38d8da4018a3e1d9b1c2b019f"} Oct 29 23:00:58.862659 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:58.862593 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-13aa41-openshift-default-59db78b859-tk95q" podStartSLOduration=1.862574912 podStartE2EDuration="1.862574912s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:00:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:00:58.85511254 +0000 UTC m=+1639.638325990" watchObservedRunningTime="2026-10-29 23:00:58.862574912 +0000 UTC m=+1639.645788340" Oct 29 23:00:59.507635 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:59.507582 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-13aa41-openshift-default-59db78b859-tk95q" Oct 29 23:00:59.510299 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:59.510265 5057 patch_prober.go:29] interesting pod/e2e-ipp-b-13aa41-openshift-default-59db78b859-tk95q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body= Oct 29 23:00:59.510481 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:59.510325 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-13aa41-openshift-default-59db78b859-tk95q" podUID="cc933bed-efee-4aad-9d65-8ee1d04f846b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" Oct 29 23:00:59.912645 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:00:59.912528 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:01:00.507357 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:00.505794 5057 patch_prober.go:29] interesting pod/e2e-ipp-b-13aa41-openshift-default-59db78b859-tk95q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body= Oct 29 23:01:00.507357 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:00.505860 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-13aa41-openshift-default-59db78b859-tk95q" podUID="cc933bed-efee-4aad-9d65-8ee1d04f846b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" Oct 29 23:01:01.508499 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:01.508075 5057 patch_prober.go:29] interesting pod/e2e-ipp-b-13aa41-openshift-default-59db78b859-tk95q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body= Oct 29 23:01:01.508499 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:01.508130 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-13aa41-openshift-default-59db78b859-tk95q" podUID="cc933bed-efee-4aad-9d65-8ee1d04f846b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" Oct 29 23:01:01.690846 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:01.690807 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:01:02.503824 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:02.503784 5057 patch_prober.go:29] interesting pod/e2e-ipp-b-13aa41-openshift-default-59db78b859-tk95q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body= Oct 29 23:01:02.504154 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:02.504123 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-13aa41-openshift-default-59db78b859-tk95q" podUID="cc933bed-efee-4aad-9d65-8ee1d04f846b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" Oct 29 23:01:03.502257 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:03.502218 5057 patch_prober.go:29] interesting pod/e2e-ipp-b-13aa41-openshift-default-59db78b859-tk95q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body= Oct 29 23:01:03.502788 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:03.502311 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-13aa41-openshift-default-59db78b859-tk95q" podUID="cc933bed-efee-4aad-9d65-8ee1d04f846b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" Oct 29 23:01:04.008875 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:04.008770 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:01:04.507223 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:04.507187 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-13aa41-openshift-default-59db78b859-tk95q" Oct 29 23:01:04.507735 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:04.507672 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-13aa41-openshift-default-59db78b859-tk95q" Oct 29 23:01:04.508445 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:04.508420 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-13aa41-openshift-default-59db78b859-tk95q" Oct 29 23:01:05.686721 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:05.686683 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:01:07.324377 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:07.324271 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:01:08.921639 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:08.921607 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:01:10.642125 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:10.642087 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:01:10.755269 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:10.755222 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:01:12.552562 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:12.552522 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:01:13.136123 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:13.136085 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:01:13.470814 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:13.470779 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:01:14.319265 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:14.319226 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:01:14.480856 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:14.480672 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:01:14.729495 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:14.729460 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d3e268-6d96fdd5f9-wb2zg"] Oct 29 23:01:14.734449 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:14.734413 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d3e268-6d96fdd5f9-wb2zg" Oct 29 23:01:14.756683 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:14.756652 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d3e268-6d96fdd5f9-wb2zg"] Oct 29 23:01:14.758061 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:14.758034 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-d3e268\"" Oct 29 23:01:14.758311 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:14.758202 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-d3e268-dockercfg-fwvjv\"" Oct 29 23:01:14.832260 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:14.832189 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0bcfdeb7-fb96-4c04-ae29-459ae9d29c65-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-d3e268-6d96fdd5f9-wb2zg\" (UID: \"0bcfdeb7-fb96-4c04-ae29-459ae9d29c65\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d3e268-6d96fdd5f9-wb2zg" Oct 29 23:01:14.833236 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:14.832565 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xh2c9\" (UniqueName: \"kubernetes.io/projected/0bcfdeb7-fb96-4c04-ae29-459ae9d29c65-kube-api-access-xh2c9\") pod \"payload-pre-processing-e2e-ipp-b-d3e268-6d96fdd5f9-wb2zg\" (UID: \"0bcfdeb7-fb96-4c04-ae29-459ae9d29c65\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d3e268-6d96fdd5f9-wb2zg" Oct 29 23:01:14.903768 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:14.903735 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d3e268-9f67b9b8-8d9dg"] Oct 29 23:01:14.907996 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:14.907952 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-d3e268-9f67b9b8-8d9dg" Oct 29 23:01:14.931421 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:14.929851 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d3e268-9f67b9b8-8d9dg"] Oct 29 23:01:14.934614 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:14.934547 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0bcfdeb7-fb96-4c04-ae29-459ae9d29c65-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-d3e268-6d96fdd5f9-wb2zg\" (UID: \"0bcfdeb7-fb96-4c04-ae29-459ae9d29c65\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d3e268-6d96fdd5f9-wb2zg" Oct 29 23:01:14.934762 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:14.934634 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqslp\" (UniqueName: \"kubernetes.io/projected/643465d1-5a26-4f82-ba35-8c9de37291cc-kube-api-access-vqslp\") pod \"payload-processing-e2e-ipp-b-d3e268-9f67b9b8-8d9dg\" (UID: \"643465d1-5a26-4f82-ba35-8c9de37291cc\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d3e268-9f67b9b8-8d9dg" Oct 29 23:01:14.934762 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:14.934687 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xh2c9\" (UniqueName: \"kubernetes.io/projected/0bcfdeb7-fb96-4c04-ae29-459ae9d29c65-kube-api-access-xh2c9\") pod \"payload-pre-processing-e2e-ipp-b-d3e268-6d96fdd5f9-wb2zg\" (UID: \"0bcfdeb7-fb96-4c04-ae29-459ae9d29c65\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d3e268-6d96fdd5f9-wb2zg" Oct 29 23:01:14.934889 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:14.934793 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/643465d1-5a26-4f82-ba35-8c9de37291cc-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-d3e268-9f67b9b8-8d9dg\" (UID: \"643465d1-5a26-4f82-ba35-8c9de37291cc\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d3e268-9f67b9b8-8d9dg" Oct 29 23:01:14.936124 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:14.935769 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0bcfdeb7-fb96-4c04-ae29-459ae9d29c65-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-d3e268-6d96fdd5f9-wb2zg\" (UID: \"0bcfdeb7-fb96-4c04-ae29-459ae9d29c65\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d3e268-6d96fdd5f9-wb2zg" Oct 29 23:01:14.995413 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:14.995273 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xh2c9\" (UniqueName: \"kubernetes.io/projected/0bcfdeb7-fb96-4c04-ae29-459ae9d29c65-kube-api-access-xh2c9\") pod \"payload-pre-processing-e2e-ipp-b-d3e268-6d96fdd5f9-wb2zg\" (UID: \"0bcfdeb7-fb96-4c04-ae29-459ae9d29c65\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d3e268-6d96fdd5f9-wb2zg" Oct 29 23:01:15.040281 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:15.037294 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqslp\" (UniqueName: \"kubernetes.io/projected/643465d1-5a26-4f82-ba35-8c9de37291cc-kube-api-access-vqslp\") pod \"payload-processing-e2e-ipp-b-d3e268-9f67b9b8-8d9dg\" (UID: \"643465d1-5a26-4f82-ba35-8c9de37291cc\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d3e268-9f67b9b8-8d9dg" Oct 29 23:01:15.040281 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:15.037763 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/643465d1-5a26-4f82-ba35-8c9de37291cc-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-d3e268-9f67b9b8-8d9dg\" (UID: \"643465d1-5a26-4f82-ba35-8c9de37291cc\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d3e268-9f67b9b8-8d9dg" Oct 29 23:01:15.040281 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:15.039032 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/643465d1-5a26-4f82-ba35-8c9de37291cc-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-d3e268-9f67b9b8-8d9dg\" (UID: \"643465d1-5a26-4f82-ba35-8c9de37291cc\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d3e268-9f67b9b8-8d9dg" Oct 29 23:01:15.060211 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:15.058300 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d3e268-6d96fdd5f9-wb2zg" Oct 29 23:01:15.084788 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:15.084716 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqslp\" (UniqueName: \"kubernetes.io/projected/643465d1-5a26-4f82-ba35-8c9de37291cc-kube-api-access-vqslp\") pod \"payload-processing-e2e-ipp-b-d3e268-9f67b9b8-8d9dg\" (UID: \"643465d1-5a26-4f82-ba35-8c9de37291cc\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d3e268-9f67b9b8-8d9dg" Oct 29 23:01:15.247434 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:15.245587 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-d3e268-9f67b9b8-8d9dg" Oct 29 23:01:15.340400 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:15.332623 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:01:15.436882 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:15.436815 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d3e268-6d96fdd5f9-wb2zg"] Oct 29 23:01:15.565007 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:15.564588 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:01:15.618196 ip-10-0-140-74 kubenswrapper[5057]: W1029 23:01:15.618156 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod643465d1_5a26_4f82_ba35_8c9de37291cc.slice/crio-0474e5b0446d0a3d8ee2d6d11abadf36f85def398850a224ea28b1213f1cb2fe WatchSource:0}: Error finding container 0474e5b0446d0a3d8ee2d6d11abadf36f85def398850a224ea28b1213f1cb2fe: Status 404 returned error can't find the container with id 0474e5b0446d0a3d8ee2d6d11abadf36f85def398850a224ea28b1213f1cb2fe Oct 29 23:01:15.628910 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:15.628860 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d3e268-9f67b9b8-8d9dg"] Oct 29 23:01:15.682831 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:15.682791 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:01:15.792578 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:15.792488 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:01:15.930355 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:15.930297 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:01:16.067694 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:16.066625 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d3e268-6d96fdd5f9-wb2zg" event={"ID":"0bcfdeb7-fb96-4c04-ae29-459ae9d29c65","Type":"ContainerStarted","Data":"46c293dd7a55f47fea2c024a238b8626a29ceebda7ca7e5c0ac55bc4aaf9ad09"} Oct 29 23:01:16.067694 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:16.066696 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d3e268-6d96fdd5f9-wb2zg" event={"ID":"0bcfdeb7-fb96-4c04-ae29-459ae9d29c65","Type":"ContainerStarted","Data":"bf5c991c1a535d6db4d96bd6b900daadb563f48d748db6d54b23b4b2887b3c3e"} Oct 29 23:01:16.067694 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:16.066987 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d3e268-6d96fdd5f9-wb2zg" Oct 29 23:01:16.069544 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:16.069522 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-d3e268-9f67b9b8-8d9dg" event={"ID":"643465d1-5a26-4f82-ba35-8c9de37291cc","Type":"ContainerStarted","Data":"77f42577484ae3204052ccdb3c6548a79e6c1f61c94179e04145b231e094af97"} Oct 29 23:01:16.070406 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:16.069690 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-d3e268-9f67b9b8-8d9dg" event={"ID":"643465d1-5a26-4f82-ba35-8c9de37291cc","Type":"ContainerStarted","Data":"0474e5b0446d0a3d8ee2d6d11abadf36f85def398850a224ea28b1213f1cb2fe"} Oct 29 23:01:16.070526 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:16.070475 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-d3e268-9f67b9b8-8d9dg" Oct 29 23:01:16.107302 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:16.106573 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d3e268-6d96fdd5f9-wb2zg" podStartSLOduration=2.106553956 podStartE2EDuration="2.106553956s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:01:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:01:16.105007272 +0000 UTC m=+1656.888220698" watchObservedRunningTime="2026-10-29 23:01:16.106553956 +0000 UTC m=+1656.889767383" Oct 29 23:01:16.147958 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:16.147899 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-d3e268-9f67b9b8-8d9dg" podStartSLOduration=2.147883579 podStartE2EDuration="2.147883579s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:01:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:01:16.143140163 +0000 UTC m=+1656.926353589" watchObservedRunningTime="2026-10-29 23:01:16.147883579 +0000 UTC m=+1656.931097003" Oct 29 23:01:16.490096 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:16.490058 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:01:18.382239 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:18.382199 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:01:19.030266 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:01:19.030232 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d2dnn" containerName="extractor" Oct 29 23:01:19.276744 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:19.276710 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:01:19.399528 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:19.399431 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:01:19.643896 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:19.643855 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:01:20.131715 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:20.131675 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:01:20.241035 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:20.240997 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:01:20.424882 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:20.424783 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:01:21.751708 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:21.751670 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:01:21.903036 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:21.903000 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:01:22.162825 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:22.162744 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-13aa41-6ffbb89c47-dbncv"] Oct 29 23:01:22.167796 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:22.167774 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-13aa41-6ffbb89c47-dbncv" Oct 29 23:01:22.215357 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:22.204704 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-13aa41-dockercfg-cf7zz\"" Oct 29 23:01:22.224517 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:22.223725 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-13aa41\"" Oct 29 23:01:22.251621 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:22.251582 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-13aa41-6ffbb89c47-dbncv"] Oct 29 23:01:22.265393 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:22.257046 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bf2jj\" (UniqueName: \"kubernetes.io/projected/9ff674d5-04a8-4852-a166-2dbaac9276f3-kube-api-access-bf2jj\") pod \"payload-pre-processing-e2e-ipp-b-13aa41-6ffbb89c47-dbncv\" (UID: \"9ff674d5-04a8-4852-a166-2dbaac9276f3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-13aa41-6ffbb89c47-dbncv" Oct 29 23:01:22.265393 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:22.257420 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ff674d5-04a8-4852-a166-2dbaac9276f3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-13aa41-6ffbb89c47-dbncv\" (UID: \"9ff674d5-04a8-4852-a166-2dbaac9276f3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-13aa41-6ffbb89c47-dbncv" Oct 29 23:01:22.455381 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:22.444535 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bf2jj\" (UniqueName: \"kubernetes.io/projected/9ff674d5-04a8-4852-a166-2dbaac9276f3-kube-api-access-bf2jj\") pod \"payload-pre-processing-e2e-ipp-b-13aa41-6ffbb89c47-dbncv\" (UID: \"9ff674d5-04a8-4852-a166-2dbaac9276f3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-13aa41-6ffbb89c47-dbncv" Oct 29 23:01:22.457704 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:22.457675 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ff674d5-04a8-4852-a166-2dbaac9276f3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-13aa41-6ffbb89c47-dbncv\" (UID: \"9ff674d5-04a8-4852-a166-2dbaac9276f3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-13aa41-6ffbb89c47-dbncv" Oct 29 23:01:22.459914 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:22.459891 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ff674d5-04a8-4852-a166-2dbaac9276f3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-13aa41-6ffbb89c47-dbncv\" (UID: \"9ff674d5-04a8-4852-a166-2dbaac9276f3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-13aa41-6ffbb89c47-dbncv" Oct 29 23:01:22.475377 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:22.475350 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bf2jj\" (UniqueName: \"kubernetes.io/projected/9ff674d5-04a8-4852-a166-2dbaac9276f3-kube-api-access-bf2jj\") pod \"payload-pre-processing-e2e-ipp-b-13aa41-6ffbb89c47-dbncv\" (UID: \"9ff674d5-04a8-4852-a166-2dbaac9276f3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-13aa41-6ffbb89c47-dbncv" Oct 29 23:01:22.500465 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:22.491188 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-13aa41-85f7584cdb-t2vz5"] Oct 29 23:01:22.508199 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:22.508171 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-13aa41-85f7584cdb-t2vz5"] Oct 29 23:01:22.508504 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:22.508487 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-13aa41-85f7584cdb-t2vz5" Oct 29 23:01:22.568792 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:22.568760 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-13aa41-6ffbb89c47-dbncv" Oct 29 23:01:22.666392 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:22.664132 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ceafdaf2-b0d8-4c58-b0cc-e6d5ae54168f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-13aa41-85f7584cdb-t2vz5\" (UID: \"ceafdaf2-b0d8-4c58-b0cc-e6d5ae54168f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-13aa41-85f7584cdb-t2vz5" Oct 29 23:01:22.666392 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:22.664345 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqh9r\" (UniqueName: \"kubernetes.io/projected/ceafdaf2-b0d8-4c58-b0cc-e6d5ae54168f-kube-api-access-wqh9r\") pod \"payload-processing-e2e-ipp-b-13aa41-85f7584cdb-t2vz5\" (UID: \"ceafdaf2-b0d8-4c58-b0cc-e6d5ae54168f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-13aa41-85f7584cdb-t2vz5" Oct 29 23:01:22.766277 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:22.766240 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ceafdaf2-b0d8-4c58-b0cc-e6d5ae54168f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-13aa41-85f7584cdb-t2vz5\" (UID: \"ceafdaf2-b0d8-4c58-b0cc-e6d5ae54168f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-13aa41-85f7584cdb-t2vz5" Oct 29 23:01:22.766860 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:22.766393 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqh9r\" (UniqueName: \"kubernetes.io/projected/ceafdaf2-b0d8-4c58-b0cc-e6d5ae54168f-kube-api-access-wqh9r\") pod \"payload-processing-e2e-ipp-b-13aa41-85f7584cdb-t2vz5\" (UID: \"ceafdaf2-b0d8-4c58-b0cc-e6d5ae54168f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-13aa41-85f7584cdb-t2vz5" Oct 29 23:01:22.767144 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:22.767119 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ceafdaf2-b0d8-4c58-b0cc-e6d5ae54168f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-13aa41-85f7584cdb-t2vz5\" (UID: \"ceafdaf2-b0d8-4c58-b0cc-e6d5ae54168f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-13aa41-85f7584cdb-t2vz5" Oct 29 23:01:22.788132 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:22.787997 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqh9r\" (UniqueName: \"kubernetes.io/projected/ceafdaf2-b0d8-4c58-b0cc-e6d5ae54168f-kube-api-access-wqh9r\") pod \"payload-processing-e2e-ipp-b-13aa41-85f7584cdb-t2vz5\" (UID: \"ceafdaf2-b0d8-4c58-b0cc-e6d5ae54168f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-13aa41-85f7584cdb-t2vz5" Oct 29 23:01:22.899295 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:22.899261 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-13aa41-85f7584cdb-t2vz5" Oct 29 23:01:22.970970 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:22.970871 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-13aa41-6ffbb89c47-dbncv"] Oct 29 23:01:23.021033 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:23.021001 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:01:23.158203 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:23.158160 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-13aa41-6ffbb89c47-dbncv" event={"ID":"9ff674d5-04a8-4852-a166-2dbaac9276f3","Type":"ContainerStarted","Data":"68355692154c11ac42a465bc9cd3770aefcc94566415aff948d787ef46ece438"} Oct 29 23:01:23.158566 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:23.158539 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-13aa41-6ffbb89c47-dbncv" event={"ID":"9ff674d5-04a8-4852-a166-2dbaac9276f3","Type":"ContainerStarted","Data":"a60349b82534165b75145c5772098ad05d3bd8bad1c2ceb83062355a818797ef"} Oct 29 23:01:23.160415 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:23.160381 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-13aa41-6ffbb89c47-dbncv" Oct 29 23:01:23.202710 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:23.202602 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-13aa41-6ffbb89c47-dbncv" podStartSLOduration=1.202580019 podStartE2EDuration="1.202580019s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:01:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:01:23.196789664 +0000 UTC m=+1663.980003089" watchObservedRunningTime="2026-10-29 23:01:23.202580019 +0000 UTC m=+1663.985793446" Oct 29 23:01:23.214300 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:23.214264 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-13aa41-85f7584cdb-t2vz5"] Oct 29 23:01:23.214535 ip-10-0-140-74 kubenswrapper[5057]: W1029 23:01:23.214504 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podceafdaf2_b0d8_4c58_b0cc_e6d5ae54168f.slice/crio-a6b334a8a175d8674a5acde2ab4b9d0f1e8a74c6ded00a224d1352472d1a0de9 WatchSource:0}: Error finding container a6b334a8a175d8674a5acde2ab4b9d0f1e8a74c6ded00a224d1352472d1a0de9: Status 404 returned error can't find the container with id a6b334a8a175d8674a5acde2ab4b9d0f1e8a74c6ded00a224d1352472d1a0de9 Oct 29 23:01:23.241537 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:23.241478 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:01:23.357208 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:23.357171 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:01:23.590474 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:23.590306 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:01:24.200402 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:24.199325 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-13aa41-85f7584cdb-t2vz5" event={"ID":"ceafdaf2-b0d8-4c58-b0cc-e6d5ae54168f","Type":"ContainerStarted","Data":"3e3777a8bcef48c4f40a09c17bc4239c0b14bf4fac12cd958da3563e7e5ba814"} Oct 29 23:01:24.200402 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:24.199401 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-13aa41-85f7584cdb-t2vz5" event={"ID":"ceafdaf2-b0d8-4c58-b0cc-e6d5ae54168f","Type":"ContainerStarted","Data":"a6b334a8a175d8674a5acde2ab4b9d0f1e8a74c6ded00a224d1352472d1a0de9"} Oct 29 23:01:24.200402 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:24.199460 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-13aa41-85f7584cdb-t2vz5" Oct 29 23:01:24.216901 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:24.216865 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v2tfm"] Oct 29 23:01:24.242399 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:24.240228 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-13aa41-85f7584cdb-t2vz5" podStartSLOduration=2.2402113359999998 podStartE2EDuration="2.240211336s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:01:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:01:24.238869448 +0000 UTC m=+1665.022082946" watchObservedRunningTime="2026-10-29 23:01:24.240211336 +0000 UTC m=+1665.023424761" Oct 29 23:01:25.244890 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:25.244855 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:01:25.453382 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:25.453319 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-4mwxl"] Oct 29 23:01:25.453703 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:25.453578 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-4mwxl" podUID="bd32fc20-abda-413c-88c1-6b3b6156e777" containerName="istio-proxy" containerID="cri-o://8a177953428b4ad55d62c0e777cd59815b6feb1585f41222bc1c5cc94e76d9af" gracePeriod=30 Oct 29 23:01:25.533699 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:25.533615 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:01:25.967625 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:25.967588 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:01:26.287920 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:26.287834 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:01:27.091961 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:27.091934 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d3e268-6d96fdd5f9-wb2zg" Oct 29 23:01:27.605886 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:27.605829 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:01:27.720955 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:27.720908 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:01:28.102407 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:28.102373 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-d3e268-9f67b9b8-8d9dg" Oct 29 23:01:28.207553 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:28.207517 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:01:29.256587 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:29.256550 5057 patch_prober.go:29] interesting pod/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-4mwxl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 23:01:29.256998 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:29.256616 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-4mwxl" podUID="bd32fc20-abda-413c-88c1-6b3b6156e777" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 23:01:29.500643 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:29.500606 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:01:29.700442 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:29.700124 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:01:29.810812 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:29.810775 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:01:30.764923 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:30.764898 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-4mwxl" Oct 29 23:01:30.869960 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:30.869925 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd32fc20-abda-413c-88c1-6b3b6156e777-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd32fc20-abda-413c-88c1-6b3b6156e777-istio-envoy\") pod \"bd32fc20-abda-413c-88c1-6b3b6156e777\" (UID: \"bd32fc20-abda-413c-88c1-6b3b6156e777\") " Oct 29 23:01:30.870142 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:30.870067 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bd32fc20-abda-413c-88c1-6b3b6156e777-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd32fc20-abda-413c-88c1-6b3b6156e777-istio-podinfo\") pod \"bd32fc20-abda-413c-88c1-6b3b6156e777\" (UID: \"bd32fc20-abda-413c-88c1-6b3b6156e777\") " Oct 29 23:01:30.870237 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:30.870222 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd32fc20-abda-413c-88c1-6b3b6156e777-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd32fc20-abda-413c-88c1-6b3b6156e777-credential-socket\") pod \"bd32fc20-abda-413c-88c1-6b3b6156e777\" (UID: \"bd32fc20-abda-413c-88c1-6b3b6156e777\") " Oct 29 23:01:30.870301 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:30.870290 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd32fc20-abda-413c-88c1-6b3b6156e777-istio-token\" (UniqueName: \"kubernetes.io/projected/bd32fc20-abda-413c-88c1-6b3b6156e777-istio-token\") pod \"bd32fc20-abda-413c-88c1-6b3b6156e777\" (UID: \"bd32fc20-abda-413c-88c1-6b3b6156e777\") " Oct 29 23:01:30.872480 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:30.870414 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd32fc20-abda-413c-88c1-6b3b6156e777-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd32fc20-abda-413c-88c1-6b3b6156e777-istio-data\") pod \"bd32fc20-abda-413c-88c1-6b3b6156e777\" (UID: \"bd32fc20-abda-413c-88c1-6b3b6156e777\") " Oct 29 23:01:30.872480 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:30.870508 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd32fc20-abda-413c-88c1-6b3b6156e777-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd32fc20-abda-413c-88c1-6b3b6156e777-istiod-ca-cert\") pod \"bd32fc20-abda-413c-88c1-6b3b6156e777\" (UID: \"bd32fc20-abda-413c-88c1-6b3b6156e777\") " Oct 29 23:01:30.872480 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:30.870551 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd32fc20-abda-413c-88c1-6b3b6156e777-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd32fc20-abda-413c-88c1-6b3b6156e777-workload-socket\") pod \"bd32fc20-abda-413c-88c1-6b3b6156e777\" (UID: \"bd32fc20-abda-413c-88c1-6b3b6156e777\") " Oct 29 23:01:30.872480 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:30.870617 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd32fc20-abda-413c-88c1-6b3b6156e777-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd32fc20-abda-413c-88c1-6b3b6156e777-workload-certs\") pod \"bd32fc20-abda-413c-88c1-6b3b6156e777\" (UID: \"bd32fc20-abda-413c-88c1-6b3b6156e777\") " Oct 29 23:01:30.872480 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:30.870623 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd32fc20-abda-413c-88c1-6b3b6156e777-credential-socket" pod "bd32fc20-abda-413c-88c1-6b3b6156e777" (UID: "bd32fc20-abda-413c-88c1-6b3b6156e777"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:01:30.872480 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:30.870715 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd32fc20-abda-413c-88c1-6b3b6156e777-kube-api-access-5bpfq\" (UniqueName: \"kubernetes.io/projected/bd32fc20-abda-413c-88c1-6b3b6156e777-kube-api-access-5bpfq\") pod \"bd32fc20-abda-413c-88c1-6b3b6156e777\" (UID: \"bd32fc20-abda-413c-88c1-6b3b6156e777\") " Oct 29 23:01:30.872480 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:30.871197 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd32fc20-abda-413c-88c1-6b3b6156e777-istio-data" pod "bd32fc20-abda-413c-88c1-6b3b6156e777" (UID: "bd32fc20-abda-413c-88c1-6b3b6156e777"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:01:30.872480 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:30.872013 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd32fc20-abda-413c-88c1-6b3b6156e777-istiod-ca-cert" pod "bd32fc20-abda-413c-88c1-6b3b6156e777" (UID: "bd32fc20-abda-413c-88c1-6b3b6156e777"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:01:30.872480 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:30.872037 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd32fc20-abda-413c-88c1-6b3b6156e777-workload-socket" pod "bd32fc20-abda-413c-88c1-6b3b6156e777" (UID: "bd32fc20-abda-413c-88c1-6b3b6156e777"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:01:30.872480 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:30.872235 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd32fc20-abda-413c-88c1-6b3b6156e777-workload-certs" pod "bd32fc20-abda-413c-88c1-6b3b6156e777" (UID: "bd32fc20-abda-413c-88c1-6b3b6156e777"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:01:30.872480 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:30.872401 5057 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd32fc20-abda-413c-88c1-6b3b6156e777-credential-socket\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:01:30.872480 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:30.872427 5057 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd32fc20-abda-413c-88c1-6b3b6156e777-istio-data\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:01:30.872480 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:30.872442 5057 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd32fc20-abda-413c-88c1-6b3b6156e777-istiod-ca-cert\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:01:30.872480 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:30.872456 5057 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd32fc20-abda-413c-88c1-6b3b6156e777-workload-socket\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:01:30.874658 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:30.874599 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bd32fc20-abda-413c-88c1-6b3b6156e777-istio-podinfo" pod "bd32fc20-abda-413c-88c1-6b3b6156e777" (UID: "bd32fc20-abda-413c-88c1-6b3b6156e777"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:01:30.875461 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:30.875438 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd32fc20-abda-413c-88c1-6b3b6156e777-kube-api-access-5bpfq" pod "bd32fc20-abda-413c-88c1-6b3b6156e777" (UID: "bd32fc20-abda-413c-88c1-6b3b6156e777"). InnerVolumeSpecName "kube-api-access-5bpfq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:01:30.875772 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:30.875746 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd32fc20-abda-413c-88c1-6b3b6156e777-istio-token" pod "bd32fc20-abda-413c-88c1-6b3b6156e777" (UID: "bd32fc20-abda-413c-88c1-6b3b6156e777"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:01:30.878356 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:30.878303 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd32fc20-abda-413c-88c1-6b3b6156e777-istio-envoy" pod "bd32fc20-abda-413c-88c1-6b3b6156e777" (UID: "bd32fc20-abda-413c-88c1-6b3b6156e777"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:01:30.932044 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:30.931996 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:01:30.974012 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:30.973978 5057 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd32fc20-abda-413c-88c1-6b3b6156e777-workload-certs\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:01:30.974012 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:30.974011 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5bpfq\" (UniqueName: \"kubernetes.io/projected/bd32fc20-abda-413c-88c1-6b3b6156e777-kube-api-access-5bpfq\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:01:30.974210 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:30.974024 5057 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd32fc20-abda-413c-88c1-6b3b6156e777-istio-envoy\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:01:30.974210 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:30.974039 5057 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd32fc20-abda-413c-88c1-6b3b6156e777-istio-podinfo\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:01:30.974210 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:30.974049 5057 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd32fc20-abda-413c-88c1-6b3b6156e777-istio-token\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:01:31.220978 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:31.220932 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:01:31.339840 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:31.339637 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:01:31.360963 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:31.360926 5057 generic.go:356] "Generic (PLEG): container finished" podID="bd32fc20-abda-413c-88c1-6b3b6156e777" containerID="8a177953428b4ad55d62c0e777cd59815b6feb1585f41222bc1c5cc94e76d9af" exitCode=0 Oct 29 23:01:31.361115 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:31.360988 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-4mwxl" Oct 29 23:01:31.361115 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:31.360986 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-4mwxl" event={"ID":"bd32fc20-abda-413c-88c1-6b3b6156e777","Type":"ContainerDied","Data":"8a177953428b4ad55d62c0e777cd59815b6feb1585f41222bc1c5cc94e76d9af"} Oct 29 23:01:31.361115 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:31.361099 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-4mwxl" event={"ID":"bd32fc20-abda-413c-88c1-6b3b6156e777","Type":"ContainerDied","Data":"0f241a5d13dfc20c4a09a4752e0c9a27a55835bdc447a7615a5f4e60d3ad183e"} Oct 29 23:01:31.361291 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:31.361130 5057 scope.go:122] "RemoveContainer" containerID="8a177953428b4ad55d62c0e777cd59815b6feb1585f41222bc1c5cc94e76d9af" Oct 29 23:01:31.373485 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:31.373460 5057 scope.go:122] "RemoveContainer" containerID="8a177953428b4ad55d62c0e777cd59815b6feb1585f41222bc1c5cc94e76d9af" Oct 29 23:01:31.373737 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:01:31.373720 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a177953428b4ad55d62c0e777cd59815b6feb1585f41222bc1c5cc94e76d9af\": container with ID starting with 8a177953428b4ad55d62c0e777cd59815b6feb1585f41222bc1c5cc94e76d9af not found: ID does not exist" containerID="8a177953428b4ad55d62c0e777cd59815b6feb1585f41222bc1c5cc94e76d9af" Oct 29 23:01:31.373807 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:31.373745 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a177953428b4ad55d62c0e777cd59815b6feb1585f41222bc1c5cc94e76d9af"} err="failed to get container status \"8a177953428b4ad55d62c0e777cd59815b6feb1585f41222bc1c5cc94e76d9af\": rpc error: code = NotFound desc = could not find container \"8a177953428b4ad55d62c0e777cd59815b6feb1585f41222bc1c5cc94e76d9af\": container with ID starting with 8a177953428b4ad55d62c0e777cd59815b6feb1585f41222bc1c5cc94e76d9af not found: ID does not exist" Oct 29 23:01:31.468422 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:31.468389 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-4mwxl"] Oct 29 23:01:31.494630 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:31.494553 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-4mwxl"] Oct 29 23:01:32.293673 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:32.293633 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:01:32.617109 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:32.617013 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:01:33.331544 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:33.331511 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd32fc20-abda-413c-88c1-6b3b6156e777" path="/var/lib/kubelet/pods/bd32fc20-abda-413c-88c1-6b3b6156e777/volumes" Oct 29 23:01:34.405950 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:34.405914 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:01:35.294721 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:35.294692 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-13aa41-6ffbb89c47-dbncv" Oct 29 23:01:36.298031 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:36.297919 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:01:36.314409 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:36.314385 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-13aa41-85f7584cdb-t2vz5" Oct 29 23:01:37.844049 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:37.844015 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:01:39.204901 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:39.204862 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-49dbb5-5777bd86cd-jfgz7"] Oct 29 23:01:39.240200 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:39.240170 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-49dbb5-c95485467-m2cbg"] Oct 29 23:01:39.355943 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:39.355852 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:01:39.480769 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:39.480662 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:01:41.458322 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:41.458286 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:01:43.444158 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:43.444123 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:01:44.205643 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:44.205600 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-49dbb5-5777bd86cd-jfgz7" podUID="b289af8a-e8b1-4f7e-a875-e78b4d8e89a8" containerName="payload-processing" containerID="cri-o://49df3010ab7d00babd135a7d939d6c908db16ab501fe194944f746a9ad294ca4" gracePeriod=25 Oct 29 23:01:44.241120 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:44.241061 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-49dbb5-c95485467-m2cbg" podUID="e76f405f-e6b4-4645-97ec-a90a9428c826" containerName="payload-pre-processing" containerID="cri-o://a32e51688864616aa0b81a66ea7e5f9a1fe4304aef168ea8d53317c06968b22b" gracePeriod=25 Oct 29 23:01:44.540696 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:44.540662 5057 generic.go:356] "Generic (PLEG): container finished" podID="b289af8a-e8b1-4f7e-a875-e78b4d8e89a8" containerID="49df3010ab7d00babd135a7d939d6c908db16ab501fe194944f746a9ad294ca4" exitCode=0 Oct 29 23:01:44.541151 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:44.540803 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-49dbb5-5777bd86cd-jfgz7" event={"ID":"b289af8a-e8b1-4f7e-a875-e78b4d8e89a8","Type":"ContainerDied","Data":"49df3010ab7d00babd135a7d939d6c908db16ab501fe194944f746a9ad294ca4"} Oct 29 23:01:44.542739 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:44.542717 5057 generic.go:356] "Generic (PLEG): container finished" podID="e76f405f-e6b4-4645-97ec-a90a9428c826" containerID="a32e51688864616aa0b81a66ea7e5f9a1fe4304aef168ea8d53317c06968b22b" exitCode=0 Oct 29 23:01:44.542874 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:44.542753 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-49dbb5-c95485467-m2cbg" event={"ID":"e76f405f-e6b4-4645-97ec-a90a9428c826","Type":"ContainerDied","Data":"a32e51688864616aa0b81a66ea7e5f9a1fe4304aef168ea8d53317c06968b22b"} Oct 29 23:01:44.631549 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:44.631516 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-49dbb5-5777bd86cd-jfgz7" Oct 29 23:01:44.672280 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:44.672244 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-49dbb5-c95485467-m2cbg" Oct 29 23:01:44.767884 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:44.767807 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b289af8a-e8b1-4f7e-a875-e78b4d8e89a8-kube-api-access-mnblf\" (UniqueName: \"kubernetes.io/projected/b289af8a-e8b1-4f7e-a875-e78b4d8e89a8-kube-api-access-mnblf\") pod \"b289af8a-e8b1-4f7e-a875-e78b4d8e89a8\" (UID: \"b289af8a-e8b1-4f7e-a875-e78b4d8e89a8\") " Oct 29 23:01:44.768070 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:44.767963 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b289af8a-e8b1-4f7e-a875-e78b4d8e89a8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b289af8a-e8b1-4f7e-a875-e78b4d8e89a8-plugins-config-volume\") pod \"b289af8a-e8b1-4f7e-a875-e78b4d8e89a8\" (UID: \"b289af8a-e8b1-4f7e-a875-e78b4d8e89a8\") " Oct 29 23:01:44.768199 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:44.768183 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e76f405f-e6b4-4645-97ec-a90a9428c826-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e76f405f-e6b4-4645-97ec-a90a9428c826-plugins-config-volume\") pod \"e76f405f-e6b4-4645-97ec-a90a9428c826\" (UID: \"e76f405f-e6b4-4645-97ec-a90a9428c826\") " Oct 29 23:01:44.768272 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:44.768250 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e76f405f-e6b4-4645-97ec-a90a9428c826-kube-api-access-kqm28\" (UniqueName: \"kubernetes.io/projected/e76f405f-e6b4-4645-97ec-a90a9428c826-kube-api-access-kqm28\") pod \"e76f405f-e6b4-4645-97ec-a90a9428c826\" (UID: \"e76f405f-e6b4-4645-97ec-a90a9428c826\") " Oct 29 23:01:44.768554 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:44.768526 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b289af8a-e8b1-4f7e-a875-e78b4d8e89a8-plugins-config-volume" pod "b289af8a-e8b1-4f7e-a875-e78b4d8e89a8" (UID: "b289af8a-e8b1-4f7e-a875-e78b4d8e89a8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:01:44.768681 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:44.768575 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e76f405f-e6b4-4645-97ec-a90a9428c826-plugins-config-volume" pod "e76f405f-e6b4-4645-97ec-a90a9428c826" (UID: "e76f405f-e6b4-4645-97ec-a90a9428c826"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:01:44.769071 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:44.769047 5057 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b289af8a-e8b1-4f7e-a875-e78b4d8e89a8-plugins-config-volume\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:01:44.769071 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:44.769075 5057 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e76f405f-e6b4-4645-97ec-a90a9428c826-plugins-config-volume\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:01:44.770203 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:44.770176 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b289af8a-e8b1-4f7e-a875-e78b4d8e89a8-kube-api-access-mnblf" pod "b289af8a-e8b1-4f7e-a875-e78b4d8e89a8" (UID: "b289af8a-e8b1-4f7e-a875-e78b4d8e89a8"). InnerVolumeSpecName "kube-api-access-mnblf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:01:44.770458 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:44.770435 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e76f405f-e6b4-4645-97ec-a90a9428c826-kube-api-access-kqm28" pod "e76f405f-e6b4-4645-97ec-a90a9428c826" (UID: "e76f405f-e6b4-4645-97ec-a90a9428c826"). InnerVolumeSpecName "kube-api-access-kqm28". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:01:44.871088 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:44.871049 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kqm28\" (UniqueName: \"kubernetes.io/projected/e76f405f-e6b4-4645-97ec-a90a9428c826-kube-api-access-kqm28\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:01:44.871088 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:44.871082 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mnblf\" (UniqueName: \"kubernetes.io/projected/b289af8a-e8b1-4f7e-a875-e78b4d8e89a8-kube-api-access-mnblf\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:01:44.875270 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:44.875228 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:01:45.056918 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:45.056839 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:01:45.301922 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:45.301877 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f173b1-7b9d5d9867-52n67"] Oct 29 23:01:45.311404 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:45.303892 5057 cpu_manager.go:414] "containerMap: removing container" podUID="bd32fc20-abda-413c-88c1-6b3b6156e777" containerName="istio-proxy" Oct 29 23:01:45.311404 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:45.303915 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd32fc20-abda-413c-88c1-6b3b6156e777" containerName="istio-proxy" Oct 29 23:01:45.311404 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:45.303939 5057 cpu_manager.go:414] "containerMap: removing container" podUID="b289af8a-e8b1-4f7e-a875-e78b4d8e89a8" containerName="payload-processing" Oct 29 23:01:45.311404 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:45.303945 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b289af8a-e8b1-4f7e-a875-e78b4d8e89a8" containerName="payload-processing" Oct 29 23:01:45.311404 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:45.303982 5057 cpu_manager.go:414] "containerMap: removing container" podUID="e76f405f-e6b4-4645-97ec-a90a9428c826" containerName="payload-pre-processing" Oct 29 23:01:45.311404 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:45.303987 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e76f405f-e6b4-4645-97ec-a90a9428c826" containerName="payload-pre-processing" Oct 29 23:01:45.311404 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:45.304165 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="e76f405f-e6b4-4645-97ec-a90a9428c826" containerName="payload-pre-processing" Oct 29 23:01:45.311404 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:45.304190 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd32fc20-abda-413c-88c1-6b3b6156e777" containerName="istio-proxy" Oct 29 23:01:45.311404 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:45.304203 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="b289af8a-e8b1-4f7e-a875-e78b4d8e89a8" containerName="payload-processing" Oct 29 23:01:45.311404 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:45.308836 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f173b1-7b9d5d9867-52n67" Oct 29 23:01:45.331820 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:45.331776 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-f173b1\"" Oct 29 23:01:45.332074 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:45.332042 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-f173b1-dockercfg-lrlf8\"" Oct 29 23:01:45.401397 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:45.390050 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e3693ec-7e28-479f-a285-9720616a7b53-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-f173b1-7b9d5d9867-52n67\" (UID: \"9e3693ec-7e28-479f-a285-9720616a7b53\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f173b1-7b9d5d9867-52n67" Oct 29 23:01:45.401397 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:45.390176 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzvl7\" (UniqueName: \"kubernetes.io/projected/9e3693ec-7e28-479f-a285-9720616a7b53-kube-api-access-xzvl7\") pod \"payload-pre-processing-e2e-ipp-a-f173b1-7b9d5d9867-52n67\" (UID: \"9e3693ec-7e28-479f-a285-9720616a7b53\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f173b1-7b9d5d9867-52n67" Oct 29 23:01:45.426858 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:45.426819 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f173b1-7b9d5d9867-52n67"] Oct 29 23:01:45.492359 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:45.492302 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e3693ec-7e28-479f-a285-9720616a7b53-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-f173b1-7b9d5d9867-52n67\" (UID: \"9e3693ec-7e28-479f-a285-9720616a7b53\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f173b1-7b9d5d9867-52n67" Oct 29 23:01:45.492545 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:45.492441 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzvl7\" (UniqueName: \"kubernetes.io/projected/9e3693ec-7e28-479f-a285-9720616a7b53-kube-api-access-xzvl7\") pod \"payload-pre-processing-e2e-ipp-a-f173b1-7b9d5d9867-52n67\" (UID: \"9e3693ec-7e28-479f-a285-9720616a7b53\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f173b1-7b9d5d9867-52n67" Oct 29 23:01:45.493462 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:45.493433 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e3693ec-7e28-479f-a285-9720616a7b53-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-f173b1-7b9d5d9867-52n67\" (UID: \"9e3693ec-7e28-479f-a285-9720616a7b53\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f173b1-7b9d5d9867-52n67" Oct 29 23:01:45.514286 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:45.514260 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzvl7\" (UniqueName: \"kubernetes.io/projected/9e3693ec-7e28-479f-a285-9720616a7b53-kube-api-access-xzvl7\") pod \"payload-pre-processing-e2e-ipp-a-f173b1-7b9d5d9867-52n67\" (UID: \"9e3693ec-7e28-479f-a285-9720616a7b53\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f173b1-7b9d5d9867-52n67" Oct 29 23:01:45.569520 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:45.569403 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-49dbb5-5777bd86cd-jfgz7" event={"ID":"b289af8a-e8b1-4f7e-a875-e78b4d8e89a8","Type":"ContainerDied","Data":"fc094c865028164bfd6d4a4eb134fb5e08b8492ea1792a482ad4f3a9464e7369"} Oct 29 23:01:45.569520 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:45.569480 5057 scope.go:122] "RemoveContainer" containerID="49df3010ab7d00babd135a7d939d6c908db16ab501fe194944f746a9ad294ca4" Oct 29 23:01:45.570052 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:45.569634 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-49dbb5-5777bd86cd-jfgz7" Oct 29 23:01:45.577213 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:45.577180 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-49dbb5-c95485467-m2cbg" event={"ID":"e76f405f-e6b4-4645-97ec-a90a9428c826","Type":"ContainerDied","Data":"fb2f73455b468c0c3558f2c23a5ba55c9534d3e58833d64a493cf09a35fc96b9"} Oct 29 23:01:45.577371 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:45.577215 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-49dbb5-c95485467-m2cbg" Oct 29 23:01:45.597952 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:45.597922 5057 scope.go:122] "RemoveContainer" containerID="a32e51688864616aa0b81a66ea7e5f9a1fe4304aef168ea8d53317c06968b22b" Oct 29 23:01:45.650978 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:45.650951 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-49dbb5-5777bd86cd-jfgz7"] Oct 29 23:01:45.656941 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:45.656910 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-49dbb5-5777bd86cd-jfgz7"] Oct 29 23:01:45.665790 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:45.665763 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f173b1-7b9d5d9867-52n67" Oct 29 23:01:45.692884 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:45.692855 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-49dbb5-c95485467-m2cbg"] Oct 29 23:01:45.700392 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:45.700364 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-49dbb5-c95485467-m2cbg"] Oct 29 23:01:45.987276 ip-10-0-140-74 kubenswrapper[5057]: W1029 23:01:45.987246 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e3693ec_7e28_479f_a285_9720616a7b53.slice/crio-494d845b09f1714b4f9d56a19f3acb9bdcecc2887fe74f1acfeffca8410bedd1 WatchSource:0}: Error finding container 494d845b09f1714b4f9d56a19f3acb9bdcecc2887fe74f1acfeffca8410bedd1: Status 404 returned error can't find the container with id 494d845b09f1714b4f9d56a19f3acb9bdcecc2887fe74f1acfeffca8410bedd1 Oct 29 23:01:45.987853 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:45.987832 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f173b1-7b9d5d9867-52n67"] Oct 29 23:01:46.208391 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:46.208167 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:01:46.597135 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:46.597092 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f173b1-7b9d5d9867-52n67" event={"ID":"9e3693ec-7e28-479f-a285-9720616a7b53","Type":"ContainerStarted","Data":"1bc9fe8ec5dc9fc9589c7a1411837caf5b92288e8a4f765272c759ad961f253e"} Oct 29 23:01:46.597788 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:46.597160 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f173b1-7b9d5d9867-52n67" event={"ID":"9e3693ec-7e28-479f-a285-9720616a7b53","Type":"ContainerStarted","Data":"494d845b09f1714b4f9d56a19f3acb9bdcecc2887fe74f1acfeffca8410bedd1"} Oct 29 23:01:46.636136 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:46.636068 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f173b1-7b9d5d9867-52n67" podStartSLOduration=1.636051447 podStartE2EDuration="1.636051447s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:01:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:01:46.634814846 +0000 UTC m=+1687.418028272" watchObservedRunningTime="2026-10-29 23:01:46.636051447 +0000 UTC m=+1687.419264875" Oct 29 23:01:47.347828 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:47.347794 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b289af8a-e8b1-4f7e-a875-e78b4d8e89a8" path="/var/lib/kubelet/pods/b289af8a-e8b1-4f7e-a875-e78b4d8e89a8/volumes" Oct 29 23:01:47.348234 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:47.348220 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e76f405f-e6b4-4645-97ec-a90a9428c826" path="/var/lib/kubelet/pods/e76f405f-e6b4-4645-97ec-a90a9428c826/volumes" Oct 29 23:01:47.609430 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:47.609347 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f173b1-7b9d5d9867-52n67" Oct 29 23:01:48.510206 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:48.510144 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:01:49.029777 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:01:49.029745 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d2dnn" containerName="extractor" Oct 29 23:01:50.979267 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:50.979233 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:01:52.536849 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:52.536808 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:01:52.589059 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:52.589025 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7d18f7-f8648fdc4-5wqsx"] Oct 29 23:01:52.625626 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:52.625590 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7d18f7-fbbf8944c-wjdkg"] Oct 29 23:01:52.735291 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:52.735248 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:01:52.844139 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:52.844037 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:01:52.953789 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:52.953750 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:01:53.163681 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:53.163591 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:01:55.104836 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:55.104804 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-36befd-8479fb9fcd-vkvl2"] Oct 29 23:01:55.105810 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:55.105775 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-36befd-8479fb9fcd-vkvl2" podUID="a921041d-2670-4930-b375-b8bbdef6d573" containerName="maas-api" containerID="cri-o://d873dea07ad065016612b8fec58aa99bc90bd99c6e3276ccae7240f3143a4cdf" gracePeriod=30 Oct 29 23:01:55.183830 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:55.183760 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:01:55.267993 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:55.267961 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-36befd-97f97445d-vmwlw"] Oct 29 23:01:55.268289 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:55.268254 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-36befd-97f97445d-vmwlw" podUID="f774785c-ad04-4414-80b3-dcbe4d3f75e0" containerName="payload-processing" containerID="cri-o://3ae53834965d08d05446e1ca580cbce1983e403da96b045218a49c60c762d360" gracePeriod=30 Oct 29 23:01:55.293899 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:55.293783 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-ipp-a-36befd-97f97445d-vmwlw" podUID="f774785c-ad04-4414-80b3-dcbe4d3f75e0" containerName="payload-processing" probeResult="failure" output="dial tcp 10.132.0.123:9004: connect: connection refused" Oct 29 23:01:55.349801 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:55.349522 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-36befd-65b4658f87-7xrg8"] Oct 29 23:01:55.349801 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:55.349743 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-36befd-65b4658f87-7xrg8" podUID="767849e9-8c9c-45d1-91e6-0d8fa0ab2945" containerName="payload-pre-processing" containerID="cri-o://86cb77e25b23b76e275c796c4006ce3a1658188be6b4b7c3cf4ec959d4745947" gracePeriod=30 Oct 29 23:01:55.353237 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:55.353190 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:01:55.475816 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:55.475756 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:01:55.541917 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:55.541677 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-36befd-8479fb9fcd-vkvl2" Oct 29 23:01:55.621442 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:55.617990 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a921041d-2670-4930-b375-b8bbdef6d573-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a921041d-2670-4930-b375-b8bbdef6d573-maas-api-tls\") pod \"a921041d-2670-4930-b375-b8bbdef6d573\" (UID: \"a921041d-2670-4930-b375-b8bbdef6d573\") " Oct 29 23:01:55.621442 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:55.618060 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a921041d-2670-4930-b375-b8bbdef6d573-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a921041d-2670-4930-b375-b8bbdef6d573-openshift-service-ca\") pod \"a921041d-2670-4930-b375-b8bbdef6d573\" (UID: \"a921041d-2670-4930-b375-b8bbdef6d573\") " Oct 29 23:01:55.687897 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:55.687854 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a921041d-2670-4930-b375-b8bbdef6d573-openshift-service-ca" pod "a921041d-2670-4930-b375-b8bbdef6d573" (UID: "a921041d-2670-4930-b375-b8bbdef6d573"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:01:55.710533 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:55.710383 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:01:55.711725 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:55.711698 5057 generic.go:356] "Generic (PLEG): container finished" podID="767849e9-8c9c-45d1-91e6-0d8fa0ab2945" containerID="86cb77e25b23b76e275c796c4006ce3a1658188be6b4b7c3cf4ec959d4745947" exitCode=0 Oct 29 23:01:55.711974 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:55.711954 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-36befd-65b4658f87-7xrg8" event={"ID":"767849e9-8c9c-45d1-91e6-0d8fa0ab2945","Type":"ContainerDied","Data":"86cb77e25b23b76e275c796c4006ce3a1658188be6b4b7c3cf4ec959d4745947"} Oct 29 23:01:55.714486 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:55.714463 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-36befd-8479fb9fcd-vkvl2" event={"ID":"a921041d-2670-4930-b375-b8bbdef6d573","Type":"ContainerDied","Data":"d873dea07ad065016612b8fec58aa99bc90bd99c6e3276ccae7240f3143a4cdf"} Oct 29 23:01:55.714681 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:55.714667 5057 scope.go:122] "RemoveContainer" containerID="d873dea07ad065016612b8fec58aa99bc90bd99c6e3276ccae7240f3143a4cdf" Oct 29 23:01:55.715000 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:55.714221 5057 generic.go:356] "Generic (PLEG): container finished" podID="a921041d-2670-4930-b375-b8bbdef6d573" containerID="d873dea07ad065016612b8fec58aa99bc90bd99c6e3276ccae7240f3143a4cdf" exitCode=0 Oct 29 23:01:55.716720 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:55.714290 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-36befd-8479fb9fcd-vkvl2" Oct 29 23:01:55.718723 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:55.717546 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a921041d-2670-4930-b375-b8bbdef6d573-maas-api-tls" pod "a921041d-2670-4930-b375-b8bbdef6d573" (UID: "a921041d-2670-4930-b375-b8bbdef6d573"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:01:55.718723 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:55.716807 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-36befd-8479fb9fcd-vkvl2" event={"ID":"a921041d-2670-4930-b375-b8bbdef6d573","Type":"ContainerDied","Data":"cffa743933dec6847a375cf36c502f4ff1b61382aa3fb66edf211cd8a7bde820"} Oct 29 23:01:55.721193 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:55.720010 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a921041d-2670-4930-b375-b8bbdef6d573-kube-api-access-glbrg\" (UniqueName: \"kubernetes.io/projected/a921041d-2670-4930-b375-b8bbdef6d573-kube-api-access-glbrg\") pod \"a921041d-2670-4930-b375-b8bbdef6d573\" (UID: \"a921041d-2670-4930-b375-b8bbdef6d573\") " Oct 29 23:01:55.721193 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:55.720271 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a921041d-2670-4930-b375-b8bbdef6d573-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a921041d-2670-4930-b375-b8bbdef6d573-metrics-certs\") pod \"a921041d-2670-4930-b375-b8bbdef6d573\" (UID: \"a921041d-2670-4930-b375-b8bbdef6d573\") " Oct 29 23:01:55.721911 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:55.721890 5057 generic.go:356] "Generic (PLEG): container finished" podID="f774785c-ad04-4414-80b3-dcbe4d3f75e0" containerID="3ae53834965d08d05446e1ca580cbce1983e403da96b045218a49c60c762d360" exitCode=0 Oct 29 23:01:55.722149 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:55.722132 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-36befd-97f97445d-vmwlw" event={"ID":"f774785c-ad04-4414-80b3-dcbe4d3f75e0","Type":"ContainerDied","Data":"3ae53834965d08d05446e1ca580cbce1983e403da96b045218a49c60c762d360"} Oct 29 23:01:55.724504 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:55.724119 5057 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a921041d-2670-4930-b375-b8bbdef6d573-maas-api-tls\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:01:55.724504 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:55.724160 5057 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a921041d-2670-4930-b375-b8bbdef6d573-openshift-service-ca\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:01:55.771671 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:55.771629 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a921041d-2670-4930-b375-b8bbdef6d573-metrics-certs" pod "a921041d-2670-4930-b375-b8bbdef6d573" (UID: "a921041d-2670-4930-b375-b8bbdef6d573"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:01:55.772307 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:55.772282 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a921041d-2670-4930-b375-b8bbdef6d573-kube-api-access-glbrg" pod "a921041d-2670-4930-b375-b8bbdef6d573" (UID: "a921041d-2670-4930-b375-b8bbdef6d573"). InnerVolumeSpecName "kube-api-access-glbrg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:01:55.807184 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:55.807159 5057 scope.go:122] "RemoveContainer" containerID="d873dea07ad065016612b8fec58aa99bc90bd99c6e3276ccae7240f3143a4cdf" Oct 29 23:01:55.807661 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:01:55.807625 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d873dea07ad065016612b8fec58aa99bc90bd99c6e3276ccae7240f3143a4cdf\": container with ID starting with d873dea07ad065016612b8fec58aa99bc90bd99c6e3276ccae7240f3143a4cdf not found: ID does not exist" containerID="d873dea07ad065016612b8fec58aa99bc90bd99c6e3276ccae7240f3143a4cdf" Oct 29 23:01:55.807778 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:55.807666 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d873dea07ad065016612b8fec58aa99bc90bd99c6e3276ccae7240f3143a4cdf"} err="failed to get container status \"d873dea07ad065016612b8fec58aa99bc90bd99c6e3276ccae7240f3143a4cdf\": rpc error: code = NotFound desc = could not find container \"d873dea07ad065016612b8fec58aa99bc90bd99c6e3276ccae7240f3143a4cdf\": container with ID starting with d873dea07ad065016612b8fec58aa99bc90bd99c6e3276ccae7240f3143a4cdf not found: ID does not exist" Oct 29 23:01:55.828957 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:55.826370 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-glbrg\" (UniqueName: \"kubernetes.io/projected/a921041d-2670-4930-b375-b8bbdef6d573-kube-api-access-glbrg\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:01:55.828957 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:55.826405 5057 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a921041d-2670-4930-b375-b8bbdef6d573-metrics-certs\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:01:55.843607 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:55.843582 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-36befd-97f97445d-vmwlw" Oct 29 23:01:55.884474 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:55.884274 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:01:55.937466 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:55.935827 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f774785c-ad04-4414-80b3-dcbe4d3f75e0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f774785c-ad04-4414-80b3-dcbe4d3f75e0-plugins-config-volume\") pod \"f774785c-ad04-4414-80b3-dcbe4d3f75e0\" (UID: \"f774785c-ad04-4414-80b3-dcbe4d3f75e0\") " Oct 29 23:01:55.937466 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:55.935896 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f774785c-ad04-4414-80b3-dcbe4d3f75e0-kube-api-access-mbs6f\" (UniqueName: \"kubernetes.io/projected/f774785c-ad04-4414-80b3-dcbe4d3f75e0-kube-api-access-mbs6f\") pod \"f774785c-ad04-4414-80b3-dcbe4d3f75e0\" (UID: \"f774785c-ad04-4414-80b3-dcbe4d3f75e0\") " Oct 29 23:01:55.943365 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:55.938910 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f774785c-ad04-4414-80b3-dcbe4d3f75e0-plugins-config-volume" pod "f774785c-ad04-4414-80b3-dcbe4d3f75e0" (UID: "f774785c-ad04-4414-80b3-dcbe4d3f75e0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:01:55.943365 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:55.941760 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f774785c-ad04-4414-80b3-dcbe4d3f75e0-kube-api-access-mbs6f" pod "f774785c-ad04-4414-80b3-dcbe4d3f75e0" (UID: "f774785c-ad04-4414-80b3-dcbe4d3f75e0"). InnerVolumeSpecName "kube-api-access-mbs6f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:01:55.956091 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:55.956066 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-36befd-65b4658f87-7xrg8" Oct 29 23:01:56.038705 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:56.038664 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/767849e9-8c9c-45d1-91e6-0d8fa0ab2945-kube-api-access-85gqt\" (UniqueName: \"kubernetes.io/projected/767849e9-8c9c-45d1-91e6-0d8fa0ab2945-kube-api-access-85gqt\") pod \"767849e9-8c9c-45d1-91e6-0d8fa0ab2945\" (UID: \"767849e9-8c9c-45d1-91e6-0d8fa0ab2945\") " Oct 29 23:01:56.038849 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:56.038756 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/767849e9-8c9c-45d1-91e6-0d8fa0ab2945-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/767849e9-8c9c-45d1-91e6-0d8fa0ab2945-plugins-config-volume\") pod \"767849e9-8c9c-45d1-91e6-0d8fa0ab2945\" (UID: \"767849e9-8c9c-45d1-91e6-0d8fa0ab2945\") " Oct 29 23:01:56.039613 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:56.039589 5057 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f774785c-ad04-4414-80b3-dcbe4d3f75e0-plugins-config-volume\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:01:56.039613 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:56.039618 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mbs6f\" (UniqueName: \"kubernetes.io/projected/f774785c-ad04-4414-80b3-dcbe4d3f75e0-kube-api-access-mbs6f\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:01:56.040008 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:56.039985 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/767849e9-8c9c-45d1-91e6-0d8fa0ab2945-plugins-config-volume" pod "767849e9-8c9c-45d1-91e6-0d8fa0ab2945" (UID: "767849e9-8c9c-45d1-91e6-0d8fa0ab2945"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:01:56.043861 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:56.043817 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/767849e9-8c9c-45d1-91e6-0d8fa0ab2945-kube-api-access-85gqt" pod "767849e9-8c9c-45d1-91e6-0d8fa0ab2945" (UID: "767849e9-8c9c-45d1-91e6-0d8fa0ab2945"). InnerVolumeSpecName "kube-api-access-85gqt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:01:56.075207 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:56.075173 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-36befd-8479fb9fcd-vkvl2"] Oct 29 23:01:56.079909 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:56.079878 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-36befd-8479fb9fcd-vkvl2"] Oct 29 23:01:56.142373 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:56.142277 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-85gqt\" (UniqueName: \"kubernetes.io/projected/767849e9-8c9c-45d1-91e6-0d8fa0ab2945-kube-api-access-85gqt\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:01:56.142373 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:56.142313 5057 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/767849e9-8c9c-45d1-91e6-0d8fa0ab2945-plugins-config-volume\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:01:56.274955 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:56.274916 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:01:56.498264 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:56.498227 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:01:56.732302 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:56.732270 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-36befd-97f97445d-vmwlw" Oct 29 23:01:56.732302 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:56.732295 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-36befd-97f97445d-vmwlw" event={"ID":"f774785c-ad04-4414-80b3-dcbe4d3f75e0","Type":"ContainerDied","Data":"770ff25c1266e2e65ec045077cffc32d50e21af63396808bd733e01f25f02b34"} Oct 29 23:01:56.732563 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:56.732372 5057 scope.go:122] "RemoveContainer" containerID="3ae53834965d08d05446e1ca580cbce1983e403da96b045218a49c60c762d360" Oct 29 23:01:56.734112 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:56.734092 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-36befd-65b4658f87-7xrg8" Oct 29 23:01:56.734215 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:56.734092 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-36befd-65b4658f87-7xrg8" event={"ID":"767849e9-8c9c-45d1-91e6-0d8fa0ab2945","Type":"ContainerDied","Data":"710d5b028d3e11bfd23557f68d59fd654edb7aa839c8c56748832e101ef53908"} Oct 29 23:01:56.743760 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:56.743744 5057 scope.go:122] "RemoveContainer" containerID="86cb77e25b23b76e275c796c4006ce3a1658188be6b4b7c3cf4ec959d4745947" Oct 29 23:01:56.814637 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:56.814606 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-36befd-65b4658f87-7xrg8"] Oct 29 23:01:56.824571 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:56.824541 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-36befd-65b4658f87-7xrg8"] Oct 29 23:01:56.850500 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:56.850471 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-36befd-97f97445d-vmwlw"] Oct 29 23:01:56.854638 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:56.854610 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-36befd-97f97445d-vmwlw"] Oct 29 23:01:57.332380 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:57.332350 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="767849e9-8c9c-45d1-91e6-0d8fa0ab2945" path="/var/lib/kubelet/pods/767849e9-8c9c-45d1-91e6-0d8fa0ab2945/volumes" Oct 29 23:01:57.332859 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:57.332841 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a921041d-2670-4930-b375-b8bbdef6d573" path="/var/lib/kubelet/pods/a921041d-2670-4930-b375-b8bbdef6d573/volumes" Oct 29 23:01:57.333280 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:57.333266 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f774785c-ad04-4414-80b3-dcbe4d3f75e0" path="/var/lib/kubelet/pods/f774785c-ad04-4414-80b3-dcbe4d3f75e0/volumes" Oct 29 23:01:57.590251 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:57.590146 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-7d18f7-f8648fdc4-5wqsx" podUID="d88241f0-29ac-41ef-bb25-c68e6e3b55e7" containerName="payload-processing" containerID="cri-o://39adbb610bbed03f3133397953bad9357f2ceeb1457c93af26fb8b53547b0544" gracePeriod=25 Oct 29 23:01:57.627040 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:57.627002 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7d18f7-fbbf8944c-wjdkg" podUID="53fb3a29-fc2d-4454-ac89-8b94c4d065a3" containerName="payload-pre-processing" containerID="cri-o://6ee15104fd0c24f4d70a8e5ff3dd7deea8f5cbda87042bbb222e4975b19fe304" gracePeriod=25 Oct 29 23:01:57.668777 ip-10-0-140-74 kubenswrapper[5057]: W1029 23:01:57.668755 5057 logging.go:55] [core] [Channel #1562 SubChannel #1563]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.128:50052", ServerName: "10.132.0.128:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.128:50052: connect: connection refused" Oct 29 23:01:57.750114 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:57.750080 5057 generic.go:356] "Generic (PLEG): container finished" podID="d88241f0-29ac-41ef-bb25-c68e6e3b55e7" containerID="39adbb610bbed03f3133397953bad9357f2ceeb1457c93af26fb8b53547b0544" exitCode=0 Oct 29 23:01:57.750205 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:57.750151 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-7d18f7-f8648fdc4-5wqsx" event={"ID":"d88241f0-29ac-41ef-bb25-c68e6e3b55e7","Type":"ContainerDied","Data":"39adbb610bbed03f3133397953bad9357f2ceeb1457c93af26fb8b53547b0544"} Oct 29 23:01:57.752141 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:57.752120 5057 generic.go:356] "Generic (PLEG): container finished" podID="53fb3a29-fc2d-4454-ac89-8b94c4d065a3" containerID="6ee15104fd0c24f4d70a8e5ff3dd7deea8f5cbda87042bbb222e4975b19fe304" exitCode=0 Oct 29 23:01:57.752269 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:57.752177 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7d18f7-fbbf8944c-wjdkg" event={"ID":"53fb3a29-fc2d-4454-ac89-8b94c4d065a3","Type":"ContainerDied","Data":"6ee15104fd0c24f4d70a8e5ff3dd7deea8f5cbda87042bbb222e4975b19fe304"} Oct 29 23:01:57.955609 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:57.955582 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-7d18f7-f8648fdc4-5wqsx" Oct 29 23:01:58.008069 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:58.008039 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7d18f7-fbbf8944c-wjdkg" Oct 29 23:01:58.074545 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:58.074514 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53fb3a29-fc2d-4454-ac89-8b94c4d065a3-kube-api-access-w5gm8\" (UniqueName: \"kubernetes.io/projected/53fb3a29-fc2d-4454-ac89-8b94c4d065a3-kube-api-access-w5gm8\") pod \"53fb3a29-fc2d-4454-ac89-8b94c4d065a3\" (UID: \"53fb3a29-fc2d-4454-ac89-8b94c4d065a3\") " Oct 29 23:01:58.074732 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:58.074591 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53fb3a29-fc2d-4454-ac89-8b94c4d065a3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53fb3a29-fc2d-4454-ac89-8b94c4d065a3-plugins-config-volume\") pod \"53fb3a29-fc2d-4454-ac89-8b94c4d065a3\" (UID: \"53fb3a29-fc2d-4454-ac89-8b94c4d065a3\") " Oct 29 23:01:58.074840 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:58.074825 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d88241f0-29ac-41ef-bb25-c68e6e3b55e7-kube-api-access-gggnv\" (UniqueName: \"kubernetes.io/projected/d88241f0-29ac-41ef-bb25-c68e6e3b55e7-kube-api-access-gggnv\") pod \"d88241f0-29ac-41ef-bb25-c68e6e3b55e7\" (UID: \"d88241f0-29ac-41ef-bb25-c68e6e3b55e7\") " Oct 29 23:01:58.074964 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:58.074947 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d88241f0-29ac-41ef-bb25-c68e6e3b55e7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d88241f0-29ac-41ef-bb25-c68e6e3b55e7-plugins-config-volume\") pod \"d88241f0-29ac-41ef-bb25-c68e6e3b55e7\" (UID: \"d88241f0-29ac-41ef-bb25-c68e6e3b55e7\") " Oct 29 23:01:58.075072 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:58.075041 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53fb3a29-fc2d-4454-ac89-8b94c4d065a3-plugins-config-volume" pod "53fb3a29-fc2d-4454-ac89-8b94c4d065a3" (UID: "53fb3a29-fc2d-4454-ac89-8b94c4d065a3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:01:58.075293 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:58.075264 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d88241f0-29ac-41ef-bb25-c68e6e3b55e7-plugins-config-volume" pod "d88241f0-29ac-41ef-bb25-c68e6e3b55e7" (UID: "d88241f0-29ac-41ef-bb25-c68e6e3b55e7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:01:58.075740 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:58.075719 5057 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53fb3a29-fc2d-4454-ac89-8b94c4d065a3-plugins-config-volume\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:01:58.075821 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:58.075744 5057 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d88241f0-29ac-41ef-bb25-c68e6e3b55e7-plugins-config-volume\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:01:58.077438 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:58.077410 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d88241f0-29ac-41ef-bb25-c68e6e3b55e7-kube-api-access-gggnv" pod "d88241f0-29ac-41ef-bb25-c68e6e3b55e7" (UID: "d88241f0-29ac-41ef-bb25-c68e6e3b55e7"). InnerVolumeSpecName "kube-api-access-gggnv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:01:58.077675 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:58.077655 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53fb3a29-fc2d-4454-ac89-8b94c4d065a3-kube-api-access-w5gm8" pod "53fb3a29-fc2d-4454-ac89-8b94c4d065a3" (UID: "53fb3a29-fc2d-4454-ac89-8b94c4d065a3"). InnerVolumeSpecName "kube-api-access-w5gm8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:01:58.143768 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:58.143655 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:01:58.177623 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:58.177596 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w5gm8\" (UniqueName: \"kubernetes.io/projected/53fb3a29-fc2d-4454-ac89-8b94c4d065a3-kube-api-access-w5gm8\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:01:58.177748 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:58.177635 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gggnv\" (UniqueName: \"kubernetes.io/projected/d88241f0-29ac-41ef-bb25-c68e6e3b55e7-kube-api-access-gggnv\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:01:58.622733 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:58.622703 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f173b1-7b9d5d9867-52n67" Oct 29 23:01:58.667742 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:58.667708 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-ipp-a-7d18f7-f8648fdc4-5wqsx" podUID="d88241f0-29ac-41ef-bb25-c68e6e3b55e7" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.128:50052\" within 1s: context deadline exceeded" Oct 29 23:01:58.762194 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:58.762165 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-7d18f7-f8648fdc4-5wqsx" Oct 29 23:01:58.762408 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:58.762206 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-7d18f7-f8648fdc4-5wqsx" event={"ID":"d88241f0-29ac-41ef-bb25-c68e6e3b55e7","Type":"ContainerDied","Data":"d4a29eddaa5f19267b07611085dac224df36bce3bed2c531642e97646c04369c"} Oct 29 23:01:58.762408 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:58.762264 5057 scope.go:122] "RemoveContainer" containerID="39adbb610bbed03f3133397953bad9357f2ceeb1457c93af26fb8b53547b0544" Oct 29 23:01:58.763808 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:58.763787 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7d18f7-fbbf8944c-wjdkg" Oct 29 23:01:58.763808 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:58.763799 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7d18f7-fbbf8944c-wjdkg" event={"ID":"53fb3a29-fc2d-4454-ac89-8b94c4d065a3","Type":"ContainerDied","Data":"76ace6e4a8e4db6460b76cfe9026f268191f20909ff7c40f0a283f3da8775077"} Oct 29 23:01:58.775911 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:58.775896 5057 scope.go:122] "RemoveContainer" containerID="6ee15104fd0c24f4d70a8e5ff3dd7deea8f5cbda87042bbb222e4975b19fe304" Oct 29 23:01:58.829361 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:58.829292 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7d18f7-fbbf8944c-wjdkg"] Oct 29 23:01:58.834298 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:58.834275 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7d18f7-fbbf8944c-wjdkg"] Oct 29 23:01:58.860459 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:58.860432 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7d18f7-f8648fdc4-5wqsx"] Oct 29 23:01:58.864815 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:58.864759 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7d18f7-f8648fdc4-5wqsx"] Oct 29 23:01:59.143982 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:59.143897 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:01:59.331439 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:59.331410 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53fb3a29-fc2d-4454-ac89-8b94c4d065a3" path="/var/lib/kubelet/pods/53fb3a29-fc2d-4454-ac89-8b94c4d065a3/volumes" Oct 29 23:01:59.331919 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:59.331904 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d88241f0-29ac-41ef-bb25-c68e6e3b55e7" path="/var/lib/kubelet/pods/d88241f0-29ac-41ef-bb25-c68e6e3b55e7/volumes" Oct 29 23:01:59.840552 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:59.840513 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:01:59.948077 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:01:59.948047 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:02:01.548899 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:01.548863 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:02:01.609737 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:01.609701 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a1d1da-6f68d967db-9njsv"] Oct 29 23:02:01.610001 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:01.609974 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-a1d1da-6f68d967db-9njsv" podUID="94c6a587-bd68-4f2f-8bc0-38261350e01c" containerName="payload-processing" containerID="cri-o://a3837e24b3140cd250d33f80b9dfce39ce727a25a3d936250eaff01d38b89676" gracePeriod=30 Oct 29 23:02:01.632031 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:01.631988 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a1d1da-5475458f5f-h9gjm"] Oct 29 23:02:01.632305 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:01.632278 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a1d1da-5475458f5f-h9gjm" podUID="5f36ee07-3a0b-4db0-af36-46db93217079" containerName="payload-pre-processing" containerID="cri-o://c1d27951a8d62b6da23dfe0794834187ea5a7afa6a5c936830fef28dcbec48af" gracePeriod=30 Oct 29 23:02:01.707092 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:01.706797 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:02:01.803102 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:01.801997 5057 generic.go:356] "Generic (PLEG): container finished" podID="5f36ee07-3a0b-4db0-af36-46db93217079" containerID="c1d27951a8d62b6da23dfe0794834187ea5a7afa6a5c936830fef28dcbec48af" exitCode=0 Oct 29 23:02:01.803102 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:01.802067 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a1d1da-5475458f5f-h9gjm" event={"ID":"5f36ee07-3a0b-4db0-af36-46db93217079","Type":"ContainerDied","Data":"c1d27951a8d62b6da23dfe0794834187ea5a7afa6a5c936830fef28dcbec48af"} Oct 29 23:02:01.804632 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:01.804525 5057 generic.go:356] "Generic (PLEG): container finished" podID="94c6a587-bd68-4f2f-8bc0-38261350e01c" containerID="a3837e24b3140cd250d33f80b9dfce39ce727a25a3d936250eaff01d38b89676" exitCode=0 Oct 29 23:02:01.804632 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:01.804589 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-a1d1da-6f68d967db-9njsv" event={"ID":"94c6a587-bd68-4f2f-8bc0-38261350e01c","Type":"ContainerDied","Data":"a3837e24b3140cd250d33f80b9dfce39ce727a25a3d936250eaff01d38b89676"} Oct 29 23:02:01.824371 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:01.824241 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:02:02.059558 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:02.056984 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:02:02.194054 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:02.194016 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-a1d1da-6f68d967db-9njsv" Oct 29 23:02:02.220575 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:02.220548 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a1d1da-5475458f5f-h9gjm" Oct 29 23:02:02.266236 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:02.266199 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:02:02.352923 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:02.352845 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f36ee07-3a0b-4db0-af36-46db93217079-kube-api-access-cb4qn\" (UniqueName: \"kubernetes.io/projected/5f36ee07-3a0b-4db0-af36-46db93217079-kube-api-access-cb4qn\") pod \"5f36ee07-3a0b-4db0-af36-46db93217079\" (UID: \"5f36ee07-3a0b-4db0-af36-46db93217079\") " Oct 29 23:02:02.352923 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:02.352921 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94c6a587-bd68-4f2f-8bc0-38261350e01c-kube-api-access-rvwsn\" (UniqueName: \"kubernetes.io/projected/94c6a587-bd68-4f2f-8bc0-38261350e01c-kube-api-access-rvwsn\") pod \"94c6a587-bd68-4f2f-8bc0-38261350e01c\" (UID: \"94c6a587-bd68-4f2f-8bc0-38261350e01c\") " Oct 29 23:02:02.353142 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:02.353021 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f36ee07-3a0b-4db0-af36-46db93217079-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f36ee07-3a0b-4db0-af36-46db93217079-plugins-config-volume\") pod \"5f36ee07-3a0b-4db0-af36-46db93217079\" (UID: \"5f36ee07-3a0b-4db0-af36-46db93217079\") " Oct 29 23:02:02.353821 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:02.353259 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/94c6a587-bd68-4f2f-8bc0-38261350e01c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94c6a587-bd68-4f2f-8bc0-38261350e01c-plugins-config-volume\") pod \"94c6a587-bd68-4f2f-8bc0-38261350e01c\" (UID: \"94c6a587-bd68-4f2f-8bc0-38261350e01c\") " Oct 29 23:02:02.353821 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:02.353423 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f36ee07-3a0b-4db0-af36-46db93217079-plugins-config-volume" pod "5f36ee07-3a0b-4db0-af36-46db93217079" (UID: "5f36ee07-3a0b-4db0-af36-46db93217079"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:02:02.353821 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:02.353682 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94c6a587-bd68-4f2f-8bc0-38261350e01c-plugins-config-volume" pod "94c6a587-bd68-4f2f-8bc0-38261350e01c" (UID: "94c6a587-bd68-4f2f-8bc0-38261350e01c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:02:02.354096 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:02.354078 5057 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f36ee07-3a0b-4db0-af36-46db93217079-plugins-config-volume\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:02:02.354141 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:02.354099 5057 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94c6a587-bd68-4f2f-8bc0-38261350e01c-plugins-config-volume\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:02:02.355031 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:02.355009 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94c6a587-bd68-4f2f-8bc0-38261350e01c-kube-api-access-rvwsn" pod "94c6a587-bd68-4f2f-8bc0-38261350e01c" (UID: "94c6a587-bd68-4f2f-8bc0-38261350e01c"). InnerVolumeSpecName "kube-api-access-rvwsn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:02:02.355093 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:02.355032 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f36ee07-3a0b-4db0-af36-46db93217079-kube-api-access-cb4qn" pod "5f36ee07-3a0b-4db0-af36-46db93217079" (UID: "5f36ee07-3a0b-4db0-af36-46db93217079"). InnerVolumeSpecName "kube-api-access-cb4qn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:02:02.377590 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:02.377556 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:02:02.456082 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:02.456014 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cb4qn\" (UniqueName: \"kubernetes.io/projected/5f36ee07-3a0b-4db0-af36-46db93217079-kube-api-access-cb4qn\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:02:02.456082 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:02.456053 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rvwsn\" (UniqueName: \"kubernetes.io/projected/94c6a587-bd68-4f2f-8bc0-38261350e01c-kube-api-access-rvwsn\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:02:02.814827 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:02.814794 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-a1d1da-6f68d967db-9njsv" Oct 29 23:02:02.815292 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:02.814848 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-a1d1da-6f68d967db-9njsv" event={"ID":"94c6a587-bd68-4f2f-8bc0-38261350e01c","Type":"ContainerDied","Data":"c0d0df305d53646abf645ccbd08ec53d500171367894cce8e51ac1707427ec4b"} Oct 29 23:02:02.815292 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:02.814919 5057 scope.go:122] "RemoveContainer" containerID="a3837e24b3140cd250d33f80b9dfce39ce727a25a3d936250eaff01d38b89676" Oct 29 23:02:02.817302 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:02.817276 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a1d1da-5475458f5f-h9gjm" Oct 29 23:02:02.817456 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:02.817301 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a1d1da-5475458f5f-h9gjm" event={"ID":"5f36ee07-3a0b-4db0-af36-46db93217079","Type":"ContainerDied","Data":"96b37ddea955de99831e58c1aaf2883a2b99d40f1c3752e8b30be6ea60eac208"} Oct 29 23:02:02.829416 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:02.829390 5057 scope.go:122] "RemoveContainer" containerID="c1d27951a8d62b6da23dfe0794834187ea5a7afa6a5c936830fef28dcbec48af" Oct 29 23:02:02.838385 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:02.838348 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:02:02.877739 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:02.877711 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a1d1da-5475458f5f-h9gjm"] Oct 29 23:02:02.885395 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:02.885339 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a1d1da-5475458f5f-h9gjm"] Oct 29 23:02:02.915526 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:02.915499 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a1d1da-6f68d967db-9njsv"] Oct 29 23:02:02.920542 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:02.920517 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a1d1da-6f68d967db-9njsv"] Oct 29 23:02:03.132118 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:03.132087 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:02:03.331812 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:03.331770 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f36ee07-3a0b-4db0-af36-46db93217079" path="/var/lib/kubelet/pods/5f36ee07-3a0b-4db0-af36-46db93217079/volumes" Oct 29 23:02:03.332438 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:03.332415 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94c6a587-bd68-4f2f-8bc0-38261350e01c" path="/var/lib/kubelet/pods/94c6a587-bd68-4f2f-8bc0-38261350e01c/volumes" Oct 29 23:02:03.663156 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:03.663068 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:02:03.864875 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:03.864790 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:02:03.976760 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:03.974889 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:02:04.337633 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:04.337550 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:02:05.401891 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:05.401857 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v2tfm"] Oct 29 23:02:05.435073 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:05.435032 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:02:07.517073 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:07.517022 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:02:08.003629 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:08.003569 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:02:08.128496 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:08.128463 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:02:09.292718 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:09.292682 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:02:09.716248 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:09.716217 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:02:09.960137 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:09.960066 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:02:10.070680 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:10.070586 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:02:10.146286 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:10.146248 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-848fb8-6f97b8d6bb-4cdsr"] Oct 29 23:02:10.147662 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:10.147625 5057 cpu_manager.go:414] "containerMap: removing container" podUID="d88241f0-29ac-41ef-bb25-c68e6e3b55e7" containerName="payload-processing" Oct 29 23:02:10.147662 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:10.147659 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d88241f0-29ac-41ef-bb25-c68e6e3b55e7" containerName="payload-processing" Oct 29 23:02:10.147852 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:10.147693 5057 cpu_manager.go:414] "containerMap: removing container" podUID="f774785c-ad04-4414-80b3-dcbe4d3f75e0" containerName="payload-processing" Oct 29 23:02:10.147852 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:10.147701 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f774785c-ad04-4414-80b3-dcbe4d3f75e0" containerName="payload-processing" Oct 29 23:02:10.147852 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:10.147719 5057 cpu_manager.go:414] "containerMap: removing container" podUID="a921041d-2670-4930-b375-b8bbdef6d573" containerName="maas-api" Oct 29 23:02:10.147852 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:10.147726 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a921041d-2670-4930-b375-b8bbdef6d573" containerName="maas-api" Oct 29 23:02:10.147852 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:10.147748 5057 cpu_manager.go:414] "containerMap: removing container" podUID="767849e9-8c9c-45d1-91e6-0d8fa0ab2945" containerName="payload-pre-processing" Oct 29 23:02:10.147852 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:10.147755 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="767849e9-8c9c-45d1-91e6-0d8fa0ab2945" containerName="payload-pre-processing" Oct 29 23:02:10.147852 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:10.147785 5057 cpu_manager.go:414] "containerMap: removing container" podUID="94c6a587-bd68-4f2f-8bc0-38261350e01c" containerName="payload-processing" Oct 29 23:02:10.147852 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:10.147793 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94c6a587-bd68-4f2f-8bc0-38261350e01c" containerName="payload-processing" Oct 29 23:02:10.147852 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:10.147831 5057 cpu_manager.go:414] "containerMap: removing container" podUID="53fb3a29-fc2d-4454-ac89-8b94c4d065a3" containerName="payload-pre-processing" Oct 29 23:02:10.147852 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:10.147839 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53fb3a29-fc2d-4454-ac89-8b94c4d065a3" containerName="payload-pre-processing" Oct 29 23:02:10.149422 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:10.147865 5057 cpu_manager.go:414] "containerMap: removing container" podUID="5f36ee07-3a0b-4db0-af36-46db93217079" containerName="payload-pre-processing" Oct 29 23:02:10.149422 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:10.147872 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f36ee07-3a0b-4db0-af36-46db93217079" containerName="payload-pre-processing" Oct 29 23:02:10.149422 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:10.148095 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="53fb3a29-fc2d-4454-ac89-8b94c4d065a3" containerName="payload-pre-processing" Oct 29 23:02:10.149422 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:10.148118 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f36ee07-3a0b-4db0-af36-46db93217079" containerName="payload-pre-processing" Oct 29 23:02:10.149422 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:10.148133 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="d88241f0-29ac-41ef-bb25-c68e6e3b55e7" containerName="payload-processing" Oct 29 23:02:10.149422 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:10.148145 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="f774785c-ad04-4414-80b3-dcbe4d3f75e0" containerName="payload-processing" Oct 29 23:02:10.149422 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:10.148155 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="a921041d-2670-4930-b375-b8bbdef6d573" containerName="maas-api" Oct 29 23:02:10.149422 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:10.148169 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="767849e9-8c9c-45d1-91e6-0d8fa0ab2945" containerName="payload-pre-processing" Oct 29 23:02:10.149422 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:10.148183 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="94c6a587-bd68-4f2f-8bc0-38261350e01c" containerName="payload-processing" Oct 29 23:02:10.154797 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:10.154774 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-848fb8-85f6f7cd95-9mhkq"] Oct 29 23:02:10.154891 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:10.154876 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-848fb8-6f97b8d6bb-4cdsr" Oct 29 23:02:10.164666 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:10.161167 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-848fb8-85f6f7cd95-9mhkq" Oct 29 23:02:10.169485 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:10.169462 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-848fb8\"" Oct 29 23:02:10.174142 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:10.174108 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-848fb8-dockercfg-pmvmr\"" Oct 29 23:02:10.174856 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:10.174833 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-848fb8-6f97b8d6bb-4cdsr"] Oct 29 23:02:10.207392 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:10.181811 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-848fb8-85f6f7cd95-9mhkq"] Oct 29 23:02:10.215160 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:10.215111 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:02:10.258402 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:10.253528 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cfebb6e3-0e95-4c69-b2a9-ce6e1e64b443-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-848fb8-6f97b8d6bb-4cdsr\" (UID: \"cfebb6e3-0e95-4c69-b2a9-ce6e1e64b443\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-848fb8-6f97b8d6bb-4cdsr" Oct 29 23:02:10.258402 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:10.253781 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2j27\" (UniqueName: \"kubernetes.io/projected/7cda8b19-10e8-4efc-a2f2-61ca23d7b78a-kube-api-access-b2j27\") pod \"payload-pre-processing-e2e-ipp-b-848fb8-85f6f7cd95-9mhkq\" (UID: \"7cda8b19-10e8-4efc-a2f2-61ca23d7b78a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-848fb8-85f6f7cd95-9mhkq" Oct 29 23:02:10.258402 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:10.253821 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2trx\" (UniqueName: \"kubernetes.io/projected/cfebb6e3-0e95-4c69-b2a9-ce6e1e64b443-kube-api-access-p2trx\") pod \"payload-processing-e2e-ipp-b-848fb8-6f97b8d6bb-4cdsr\" (UID: \"cfebb6e3-0e95-4c69-b2a9-ce6e1e64b443\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-848fb8-6f97b8d6bb-4cdsr" Oct 29 23:02:10.258402 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:10.253867 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7cda8b19-10e8-4efc-a2f2-61ca23d7b78a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-848fb8-85f6f7cd95-9mhkq\" (UID: \"7cda8b19-10e8-4efc-a2f2-61ca23d7b78a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-848fb8-85f6f7cd95-9mhkq" Oct 29 23:02:10.380501 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:10.380419 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2j27\" (UniqueName: \"kubernetes.io/projected/7cda8b19-10e8-4efc-a2f2-61ca23d7b78a-kube-api-access-b2j27\") pod \"payload-pre-processing-e2e-ipp-b-848fb8-85f6f7cd95-9mhkq\" (UID: \"7cda8b19-10e8-4efc-a2f2-61ca23d7b78a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-848fb8-85f6f7cd95-9mhkq" Oct 29 23:02:10.390352 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:10.390303 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2trx\" (UniqueName: \"kubernetes.io/projected/cfebb6e3-0e95-4c69-b2a9-ce6e1e64b443-kube-api-access-p2trx\") pod \"payload-processing-e2e-ipp-b-848fb8-6f97b8d6bb-4cdsr\" (UID: \"cfebb6e3-0e95-4c69-b2a9-ce6e1e64b443\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-848fb8-6f97b8d6bb-4cdsr" Oct 29 23:02:10.390879 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:10.390858 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7cda8b19-10e8-4efc-a2f2-61ca23d7b78a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-848fb8-85f6f7cd95-9mhkq\" (UID: \"7cda8b19-10e8-4efc-a2f2-61ca23d7b78a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-848fb8-85f6f7cd95-9mhkq" Oct 29 23:02:10.392131 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:10.392087 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cfebb6e3-0e95-4c69-b2a9-ce6e1e64b443-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-848fb8-6f97b8d6bb-4cdsr\" (UID: \"cfebb6e3-0e95-4c69-b2a9-ce6e1e64b443\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-848fb8-6f97b8d6bb-4cdsr" Oct 29 23:02:10.395106 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:10.391802 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7cda8b19-10e8-4efc-a2f2-61ca23d7b78a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-848fb8-85f6f7cd95-9mhkq\" (UID: \"7cda8b19-10e8-4efc-a2f2-61ca23d7b78a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-848fb8-85f6f7cd95-9mhkq" Oct 29 23:02:10.395189 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:10.393931 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cfebb6e3-0e95-4c69-b2a9-ce6e1e64b443-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-848fb8-6f97b8d6bb-4cdsr\" (UID: \"cfebb6e3-0e95-4c69-b2a9-ce6e1e64b443\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-848fb8-6f97b8d6bb-4cdsr" Oct 29 23:02:10.414838 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:10.413377 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2trx\" (UniqueName: \"kubernetes.io/projected/cfebb6e3-0e95-4c69-b2a9-ce6e1e64b443-kube-api-access-p2trx\") pod \"payload-processing-e2e-ipp-b-848fb8-6f97b8d6bb-4cdsr\" (UID: \"cfebb6e3-0e95-4c69-b2a9-ce6e1e64b443\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-848fb8-6f97b8d6bb-4cdsr" Oct 29 23:02:10.415686 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:10.415637 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2j27\" (UniqueName: \"kubernetes.io/projected/7cda8b19-10e8-4efc-a2f2-61ca23d7b78a-kube-api-access-b2j27\") pod \"payload-pre-processing-e2e-ipp-b-848fb8-85f6f7cd95-9mhkq\" (UID: \"7cda8b19-10e8-4efc-a2f2-61ca23d7b78a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-848fb8-85f6f7cd95-9mhkq" Oct 29 23:02:10.490446 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:10.490405 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-848fb8-6f97b8d6bb-4cdsr" Oct 29 23:02:10.509470 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:10.508138 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-848fb8-85f6f7cd95-9mhkq" Oct 29 23:02:10.876956 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:10.876927 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-848fb8-85f6f7cd95-9mhkq"] Oct 29 23:02:10.877611 ip-10-0-140-74 kubenswrapper[5057]: W1029 23:02:10.877583 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cda8b19_10e8_4efc_a2f2_61ca23d7b78a.slice/crio-d3f262b54e83cc077606f7def9ae70d2d6aff7318c309e8961df8b5749e835ec WatchSource:0}: Error finding container d3f262b54e83cc077606f7def9ae70d2d6aff7318c309e8961df8b5749e835ec: Status 404 returned error can't find the container with id d3f262b54e83cc077606f7def9ae70d2d6aff7318c309e8961df8b5749e835ec Oct 29 23:02:10.892152 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:10.892124 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-848fb8-6f97b8d6bb-4cdsr"] Oct 29 23:02:10.898293 ip-10-0-140-74 kubenswrapper[5057]: W1029 23:02:10.898263 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfebb6e3_0e95_4c69_b2a9_ce6e1e64b443.slice/crio-7d26720a62916004e9ec14eb374f50bc71dc1dcce081637bd9a9f1024837bbb3 WatchSource:0}: Error finding container 7d26720a62916004e9ec14eb374f50bc71dc1dcce081637bd9a9f1024837bbb3: Status 404 returned error can't find the container with id 7d26720a62916004e9ec14eb374f50bc71dc1dcce081637bd9a9f1024837bbb3 Oct 29 23:02:10.920913 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:10.920880 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-848fb8-6f97b8d6bb-4cdsr" event={"ID":"cfebb6e3-0e95-4c69-b2a9-ce6e1e64b443","Type":"ContainerStarted","Data":"7d26720a62916004e9ec14eb374f50bc71dc1dcce081637bd9a9f1024837bbb3"} Oct 29 23:02:10.922649 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:10.922623 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-848fb8-85f6f7cd95-9mhkq" event={"ID":"7cda8b19-10e8-4efc-a2f2-61ca23d7b78a","Type":"ContainerStarted","Data":"d3f262b54e83cc077606f7def9ae70d2d6aff7318c309e8961df8b5749e835ec"} Oct 29 23:02:11.231689 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:11.231655 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:02:11.945107 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:11.945067 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-848fb8-85f6f7cd95-9mhkq" event={"ID":"7cda8b19-10e8-4efc-a2f2-61ca23d7b78a","Type":"ContainerStarted","Data":"ed6c9f79ce1f67a4e574de4a0dd269e90f96b15970d219d8e4757951c32221b1"} Oct 29 23:02:11.945568 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:11.945166 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-848fb8-85f6f7cd95-9mhkq" Oct 29 23:02:11.946806 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:11.946776 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-848fb8-6f97b8d6bb-4cdsr" event={"ID":"cfebb6e3-0e95-4c69-b2a9-ce6e1e64b443","Type":"ContainerStarted","Data":"770fb9068059bf32055d57f10f411412e0d9a74846a92a6d3a23a6f188ea9bd4"} Oct 29 23:02:11.946915 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:11.946863 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-848fb8-6f97b8d6bb-4cdsr" Oct 29 23:02:11.985106 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:11.985057 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-848fb8-85f6f7cd95-9mhkq" podStartSLOduration=1.985043322 podStartE2EDuration="1.985043322s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:02:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:02:11.98139184 +0000 UTC m=+1712.764605265" watchObservedRunningTime="2026-10-29 23:02:11.985043322 +0000 UTC m=+1712.768256747" Oct 29 23:02:12.022841 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:12.022787 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-848fb8-6f97b8d6bb-4cdsr" podStartSLOduration=2.022774139 podStartE2EDuration="2.022774139s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:02:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:02:12.017995881 +0000 UTC m=+1712.801209307" watchObservedRunningTime="2026-10-29 23:02:12.022774139 +0000 UTC m=+1712.805987565" Oct 29 23:02:13.114545 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:13.114507 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:02:14.922784 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:14.922747 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:02:17.262574 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:17.262533 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:02:17.613070 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:17.612973 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:02:17.723347 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:17.723286 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:02:18.451834 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:18.451786 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:02:18.578284 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:18.578117 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:02:18.803126 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:18.803026 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7d18f7-9fb86cbf9-wshhh"] Oct 29 23:02:18.812560 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:18.812525 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7d18f7-9fb86cbf9-wshhh" Oct 29 23:02:18.826948 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:18.826918 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7d18f7-9fb86cbf9-wshhh"] Oct 29 23:02:18.830597 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:18.830560 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-7d18f7\"" Oct 29 23:02:18.830717 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:18.830601 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-7d18f7-dockercfg-hc76c\"" Oct 29 23:02:18.954351 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:18.954295 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7r6ls\" (UniqueName: \"kubernetes.io/projected/4a6b6875-8ebc-405a-960e-45cdb0073bcb-kube-api-access-7r6ls\") pod \"payload-pre-processing-e2e-ipp-a-7d18f7-9fb86cbf9-wshhh\" (UID: \"4a6b6875-8ebc-405a-960e-45cdb0073bcb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7d18f7-9fb86cbf9-wshhh" Oct 29 23:02:18.954631 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:18.954605 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a6b6875-8ebc-405a-960e-45cdb0073bcb-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-7d18f7-9fb86cbf9-wshhh\" (UID: \"4a6b6875-8ebc-405a-960e-45cdb0073bcb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7d18f7-9fb86cbf9-wshhh" Oct 29 23:02:18.969356 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:18.969308 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7d18f7-595cb9f868-ps49n"] Oct 29 23:02:18.975242 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:18.975218 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-7d18f7-595cb9f868-ps49n" Oct 29 23:02:18.994797 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:18.994769 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7d18f7-595cb9f868-ps49n"] Oct 29 23:02:19.060077 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:02:19.059959 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d2dnn" containerName="extractor" Oct 29 23:02:19.061034 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:19.061008 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7r6ls\" (UniqueName: \"kubernetes.io/projected/4a6b6875-8ebc-405a-960e-45cdb0073bcb-kube-api-access-7r6ls\") pod \"payload-pre-processing-e2e-ipp-a-7d18f7-9fb86cbf9-wshhh\" (UID: \"4a6b6875-8ebc-405a-960e-45cdb0073bcb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7d18f7-9fb86cbf9-wshhh" Oct 29 23:02:19.061230 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:19.061212 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a6b6875-8ebc-405a-960e-45cdb0073bcb-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-7d18f7-9fb86cbf9-wshhh\" (UID: \"4a6b6875-8ebc-405a-960e-45cdb0073bcb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7d18f7-9fb86cbf9-wshhh" Oct 29 23:02:19.061980 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:19.061959 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a6b6875-8ebc-405a-960e-45cdb0073bcb-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-7d18f7-9fb86cbf9-wshhh\" (UID: \"4a6b6875-8ebc-405a-960e-45cdb0073bcb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7d18f7-9fb86cbf9-wshhh" Oct 29 23:02:19.082182 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:19.082161 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7r6ls\" (UniqueName: \"kubernetes.io/projected/4a6b6875-8ebc-405a-960e-45cdb0073bcb-kube-api-access-7r6ls\") pod \"payload-pre-processing-e2e-ipp-a-7d18f7-9fb86cbf9-wshhh\" (UID: \"4a6b6875-8ebc-405a-960e-45cdb0073bcb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7d18f7-9fb86cbf9-wshhh" Oct 29 23:02:19.135206 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:19.135164 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7d18f7-9fb86cbf9-wshhh" Oct 29 23:02:19.162975 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:19.162948 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75272\" (UniqueName: \"kubernetes.io/projected/ac11d963-38a4-41b4-ac9d-0eeadb3a2547-kube-api-access-75272\") pod \"payload-processing-e2e-ipp-a-7d18f7-595cb9f868-ps49n\" (UID: \"ac11d963-38a4-41b4-ac9d-0eeadb3a2547\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7d18f7-595cb9f868-ps49n" Oct 29 23:02:19.163206 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:19.163185 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac11d963-38a4-41b4-ac9d-0eeadb3a2547-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-7d18f7-595cb9f868-ps49n\" (UID: \"ac11d963-38a4-41b4-ac9d-0eeadb3a2547\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7d18f7-595cb9f868-ps49n" Oct 29 23:02:19.320401 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:19.271080 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac11d963-38a4-41b4-ac9d-0eeadb3a2547-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-7d18f7-595cb9f868-ps49n\" (UID: \"ac11d963-38a4-41b4-ac9d-0eeadb3a2547\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7d18f7-595cb9f868-ps49n" Oct 29 23:02:19.320401 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:19.271350 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75272\" (UniqueName: \"kubernetes.io/projected/ac11d963-38a4-41b4-ac9d-0eeadb3a2547-kube-api-access-75272\") pod \"payload-processing-e2e-ipp-a-7d18f7-595cb9f868-ps49n\" (UID: \"ac11d963-38a4-41b4-ac9d-0eeadb3a2547\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7d18f7-595cb9f868-ps49n" Oct 29 23:02:19.320401 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:19.300289 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-75272\" (UniqueName: \"kubernetes.io/projected/ac11d963-38a4-41b4-ac9d-0eeadb3a2547-kube-api-access-75272\") pod \"payload-processing-e2e-ipp-a-7d18f7-595cb9f868-ps49n\" (UID: \"ac11d963-38a4-41b4-ac9d-0eeadb3a2547\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7d18f7-595cb9f868-ps49n" Oct 29 23:02:19.344013 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:19.343975 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac11d963-38a4-41b4-ac9d-0eeadb3a2547-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-7d18f7-595cb9f868-ps49n\" (UID: \"ac11d963-38a4-41b4-ac9d-0eeadb3a2547\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7d18f7-595cb9f868-ps49n" Oct 29 23:02:19.408829 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:19.408600 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:02:19.497570 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:19.497534 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d3e268-9f67b9b8-8d9dg"] Oct 29 23:02:19.498016 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:19.497766 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-d3e268-9f67b9b8-8d9dg" podUID="643465d1-5a26-4f82-ba35-8c9de37291cc" containerName="payload-processing" containerID="cri-o://77f42577484ae3204052ccdb3c6548a79e6c1f61c94179e04145b231e094af97" gracePeriod=30 Oct 29 23:02:19.521536 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:19.521503 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d3e268-6d96fdd5f9-wb2zg"] Oct 29 23:02:19.521970 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:19.521882 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d3e268-6d96fdd5f9-wb2zg" podUID="0bcfdeb7-fb96-4c04-ae29-459ae9d29c65" containerName="payload-pre-processing" containerID="cri-o://46c293dd7a55f47fea2c024a238b8626a29ceebda7ca7e5c0ac55bc4aaf9ad09" gracePeriod=30 Oct 29 23:02:19.579373 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:19.577327 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:02:19.600994 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:19.600957 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-7d18f7-595cb9f868-ps49n" Oct 29 23:02:19.666536 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:19.666511 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7d18f7-9fb86cbf9-wshhh"] Oct 29 23:02:19.690377 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:19.690173 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:02:19.691819 ip-10-0-140-74 kubenswrapper[5057]: W1029 23:02:19.691162 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a6b6875_8ebc_405a_960e_45cdb0073bcb.slice/crio-438f4f758fcb8e107c2f04212e827621e87bc602261ab356ffd89d95576155ff WatchSource:0}: Error finding container 438f4f758fcb8e107c2f04212e827621e87bc602261ab356ffd89d95576155ff: Status 404 returned error can't find the container with id 438f4f758fcb8e107c2f04212e827621e87bc602261ab356ffd89d95576155ff Oct 29 23:02:20.060040 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:20.059989 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:02:20.089793 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:20.089758 5057 generic.go:356] "Generic (PLEG): container finished" podID="0bcfdeb7-fb96-4c04-ae29-459ae9d29c65" containerID="46c293dd7a55f47fea2c024a238b8626a29ceebda7ca7e5c0ac55bc4aaf9ad09" exitCode=0 Oct 29 23:02:20.093320 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:20.091562 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d3e268-6d96fdd5f9-wb2zg" event={"ID":"0bcfdeb7-fb96-4c04-ae29-459ae9d29c65","Type":"ContainerDied","Data":"46c293dd7a55f47fea2c024a238b8626a29ceebda7ca7e5c0ac55bc4aaf9ad09"} Oct 29 23:02:20.100666 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:20.100638 5057 generic.go:356] "Generic (PLEG): container finished" podID="643465d1-5a26-4f82-ba35-8c9de37291cc" containerID="77f42577484ae3204052ccdb3c6548a79e6c1f61c94179e04145b231e094af97" exitCode=0 Oct 29 23:02:20.100909 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:20.100875 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-d3e268-9f67b9b8-8d9dg" event={"ID":"643465d1-5a26-4f82-ba35-8c9de37291cc","Type":"ContainerDied","Data":"77f42577484ae3204052ccdb3c6548a79e6c1f61c94179e04145b231e094af97"} Oct 29 23:02:20.103130 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:20.103107 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7d18f7-9fb86cbf9-wshhh" event={"ID":"4a6b6875-8ebc-405a-960e-45cdb0073bcb","Type":"ContainerStarted","Data":"438f4f758fcb8e107c2f04212e827621e87bc602261ab356ffd89d95576155ff"} Oct 29 23:02:20.177597 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:20.177562 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:02:20.189719 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:20.189688 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d3e268-6d96fdd5f9-wb2zg" Oct 29 23:02:20.200396 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:20.200365 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-d3e268-9f67b9b8-8d9dg" Oct 29 23:02:20.200746 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:20.200719 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7d18f7-595cb9f868-ps49n"] Oct 29 23:02:20.236179 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:20.235197 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/643465d1-5a26-4f82-ba35-8c9de37291cc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/643465d1-5a26-4f82-ba35-8c9de37291cc-plugins-config-volume\") pod \"643465d1-5a26-4f82-ba35-8c9de37291cc\" (UID: \"643465d1-5a26-4f82-ba35-8c9de37291cc\") " Oct 29 23:02:20.236179 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:20.235882 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/643465d1-5a26-4f82-ba35-8c9de37291cc-kube-api-access-vqslp\" (UniqueName: \"kubernetes.io/projected/643465d1-5a26-4f82-ba35-8c9de37291cc-kube-api-access-vqslp\") pod \"643465d1-5a26-4f82-ba35-8c9de37291cc\" (UID: \"643465d1-5a26-4f82-ba35-8c9de37291cc\") " Oct 29 23:02:20.236179 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:20.235999 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0bcfdeb7-fb96-4c04-ae29-459ae9d29c65-kube-api-access-xh2c9\" (UniqueName: \"kubernetes.io/projected/0bcfdeb7-fb96-4c04-ae29-459ae9d29c65-kube-api-access-xh2c9\") pod \"0bcfdeb7-fb96-4c04-ae29-459ae9d29c65\" (UID: \"0bcfdeb7-fb96-4c04-ae29-459ae9d29c65\") " Oct 29 23:02:20.236179 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:20.236251 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0bcfdeb7-fb96-4c04-ae29-459ae9d29c65-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0bcfdeb7-fb96-4c04-ae29-459ae9d29c65-plugins-config-volume\") pod \"0bcfdeb7-fb96-4c04-ae29-459ae9d29c65\" (UID: \"0bcfdeb7-fb96-4c04-ae29-459ae9d29c65\") " Oct 29 23:02:20.238008 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:20.235816 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/643465d1-5a26-4f82-ba35-8c9de37291cc-plugins-config-volume" pod "643465d1-5a26-4f82-ba35-8c9de37291cc" (UID: "643465d1-5a26-4f82-ba35-8c9de37291cc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:02:20.238008 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:20.237980 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0bcfdeb7-fb96-4c04-ae29-459ae9d29c65-plugins-config-volume" pod "0bcfdeb7-fb96-4c04-ae29-459ae9d29c65" (UID: "0bcfdeb7-fb96-4c04-ae29-459ae9d29c65"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:02:20.256413 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:20.240708 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/643465d1-5a26-4f82-ba35-8c9de37291cc-kube-api-access-vqslp" pod "643465d1-5a26-4f82-ba35-8c9de37291cc" (UID: "643465d1-5a26-4f82-ba35-8c9de37291cc"). InnerVolumeSpecName "kube-api-access-vqslp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:02:20.262291 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:20.262255 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0bcfdeb7-fb96-4c04-ae29-459ae9d29c65-kube-api-access-xh2c9" pod "0bcfdeb7-fb96-4c04-ae29-459ae9d29c65" (UID: "0bcfdeb7-fb96-4c04-ae29-459ae9d29c65"). InnerVolumeSpecName "kube-api-access-xh2c9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:02:20.303836 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:20.303793 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:02:20.343807 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:20.342829 5057 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/643465d1-5a26-4f82-ba35-8c9de37291cc-plugins-config-volume\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:02:20.343807 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:20.342857 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vqslp\" (UniqueName: \"kubernetes.io/projected/643465d1-5a26-4f82-ba35-8c9de37291cc-kube-api-access-vqslp\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:02:20.343807 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:20.342874 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xh2c9\" (UniqueName: \"kubernetes.io/projected/0bcfdeb7-fb96-4c04-ae29-459ae9d29c65-kube-api-access-xh2c9\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:02:20.343807 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:20.342887 5057 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0bcfdeb7-fb96-4c04-ae29-459ae9d29c65-plugins-config-volume\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:02:20.754061 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:20.754014 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:02:21.115991 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:21.115916 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-d3e268-9f67b9b8-8d9dg" Oct 29 23:02:21.116272 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:21.115917 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-d3e268-9f67b9b8-8d9dg" event={"ID":"643465d1-5a26-4f82-ba35-8c9de37291cc","Type":"ContainerDied","Data":"0474e5b0446d0a3d8ee2d6d11abadf36f85def398850a224ea28b1213f1cb2fe"} Oct 29 23:02:21.116551 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:21.116531 5057 scope.go:122] "RemoveContainer" containerID="77f42577484ae3204052ccdb3c6548a79e6c1f61c94179e04145b231e094af97" Oct 29 23:02:21.119043 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:21.118983 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7d18f7-9fb86cbf9-wshhh" event={"ID":"4a6b6875-8ebc-405a-960e-45cdb0073bcb","Type":"ContainerStarted","Data":"d95eeab1a2821302ac42571aa281ae35a94774696349a6b298e50eaf6185ec82"} Oct 29 23:02:21.119176 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:21.119146 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7d18f7-9fb86cbf9-wshhh" Oct 29 23:02:21.124526 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:21.124498 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-7d18f7-595cb9f868-ps49n" event={"ID":"ac11d963-38a4-41b4-ac9d-0eeadb3a2547","Type":"ContainerStarted","Data":"bb19401173d4565d9c6d4fa5177cccb4113b56292cf2c58adbe864dd59537725"} Oct 29 23:02:21.124637 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:21.124536 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-7d18f7-595cb9f868-ps49n" event={"ID":"ac11d963-38a4-41b4-ac9d-0eeadb3a2547","Type":"ContainerStarted","Data":"f5c09d9cacb4d960dc288d8611653dc3a47e5260a08d8b16299a515ed06595b0"} Oct 29 23:02:21.124637 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:21.124596 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-7d18f7-595cb9f868-ps49n" Oct 29 23:02:21.128132 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:21.126025 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d3e268-6d96fdd5f9-wb2zg" event={"ID":"0bcfdeb7-fb96-4c04-ae29-459ae9d29c65","Type":"ContainerDied","Data":"bf5c991c1a535d6db4d96bd6b900daadb563f48d748db6d54b23b4b2887b3c3e"} Oct 29 23:02:21.128132 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:21.126050 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d3e268-6d96fdd5f9-wb2zg" Oct 29 23:02:21.132550 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:21.132532 5057 scope.go:122] "RemoveContainer" containerID="46c293dd7a55f47fea2c024a238b8626a29ceebda7ca7e5c0ac55bc4aaf9ad09" Oct 29 23:02:21.173093 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:21.173036 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7d18f7-9fb86cbf9-wshhh" podStartSLOduration=3.173019215 podStartE2EDuration="3.173019215s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:02:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:02:21.167522009 +0000 UTC m=+1721.950735435" watchObservedRunningTime="2026-10-29 23:02:21.173019215 +0000 UTC m=+1721.956232644" Oct 29 23:02:21.205601 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:21.205545 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-7d18f7-595cb9f868-ps49n" podStartSLOduration=3.205528698 podStartE2EDuration="3.205528698s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:02:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:02:21.203976016 +0000 UTC m=+1721.987189442" watchObservedRunningTime="2026-10-29 23:02:21.205528698 +0000 UTC m=+1721.988742126" Oct 29 23:02:21.252819 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:21.252791 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d3e268-9f67b9b8-8d9dg"] Oct 29 23:02:21.258046 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:21.258024 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d3e268-9f67b9b8-8d9dg"] Oct 29 23:02:21.288109 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:21.288080 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d3e268-6d96fdd5f9-wb2zg"] Oct 29 23:02:21.292430 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:21.292405 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d3e268-6d96fdd5f9-wb2zg"] Oct 29 23:02:21.356578 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:21.356542 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0bcfdeb7-fb96-4c04-ae29-459ae9d29c65" path="/var/lib/kubelet/pods/0bcfdeb7-fb96-4c04-ae29-459ae9d29c65/volumes" Oct 29 23:02:21.357888 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:21.357853 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="643465d1-5a26-4f82-ba35-8c9de37291cc" path="/var/lib/kubelet/pods/643465d1-5a26-4f82-ba35-8c9de37291cc/volumes" Oct 29 23:02:22.132724 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:22.132689 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:02:22.510382 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:22.509530 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:02:22.615255 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:22.615222 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-13aa41-85f7584cdb-t2vz5"] Oct 29 23:02:22.615477 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:22.615450 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-13aa41-85f7584cdb-t2vz5" podUID="ceafdaf2-b0d8-4c58-b0cc-e6d5ae54168f" containerName="payload-processing" containerID="cri-o://3e3777a8bcef48c4f40a09c17bc4239c0b14bf4fac12cd958da3563e7e5ba814" gracePeriod=30 Oct 29 23:02:22.650087 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:22.649979 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:02:22.677502 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:22.677465 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-13aa41-6ffbb89c47-dbncv"] Oct 29 23:02:22.677738 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:22.677712 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-13aa41-6ffbb89c47-dbncv" podUID="9ff674d5-04a8-4852-a166-2dbaac9276f3" containerName="payload-pre-processing" containerID="cri-o://68355692154c11ac42a465bc9cd3770aefcc94566415aff948d787ef46ece438" gracePeriod=30 Oct 29 23:02:22.764349 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:22.764252 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:02:22.873959 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:22.873918 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:02:22.961142 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:22.960825 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-848fb8-85f6f7cd95-9mhkq" Oct 29 23:02:22.966402 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:22.964430 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-848fb8-6f97b8d6bb-4cdsr" Oct 29 23:02:23.194621 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:23.180768 5057 generic.go:356] "Generic (PLEG): container finished" podID="ceafdaf2-b0d8-4c58-b0cc-e6d5ae54168f" containerID="3e3777a8bcef48c4f40a09c17bc4239c0b14bf4fac12cd958da3563e7e5ba814" exitCode=0 Oct 29 23:02:23.194621 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:23.180929 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-13aa41-85f7584cdb-t2vz5" event={"ID":"ceafdaf2-b0d8-4c58-b0cc-e6d5ae54168f","Type":"ContainerDied","Data":"3e3777a8bcef48c4f40a09c17bc4239c0b14bf4fac12cd958da3563e7e5ba814"} Oct 29 23:02:23.194621 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:23.180977 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-13aa41-85f7584cdb-t2vz5" event={"ID":"ceafdaf2-b0d8-4c58-b0cc-e6d5ae54168f","Type":"ContainerDied","Data":"a6b334a8a175d8674a5acde2ab4b9d0f1e8a74c6ded00a224d1352472d1a0de9"} Oct 29 23:02:23.194621 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:23.180999 5057 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a6b334a8a175d8674a5acde2ab4b9d0f1e8a74c6ded00a224d1352472d1a0de9" Oct 29 23:02:23.194621 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:23.183432 5057 generic.go:356] "Generic (PLEG): container finished" podID="9ff674d5-04a8-4852-a166-2dbaac9276f3" containerID="68355692154c11ac42a465bc9cd3770aefcc94566415aff948d787ef46ece438" exitCode=0 Oct 29 23:02:23.194621 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:23.183580 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-13aa41-6ffbb89c47-dbncv" event={"ID":"9ff674d5-04a8-4852-a166-2dbaac9276f3","Type":"ContainerDied","Data":"68355692154c11ac42a465bc9cd3770aefcc94566415aff948d787ef46ece438"} Oct 29 23:02:23.246682 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:23.246582 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-13aa41-85f7584cdb-t2vz5" Oct 29 23:02:23.280288 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:23.280263 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-13aa41-6ffbb89c47-dbncv" Oct 29 23:02:23.331393 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:23.330554 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:02:23.343673 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:23.343636 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ceafdaf2-b0d8-4c58-b0cc-e6d5ae54168f-kube-api-access-wqh9r\" (UniqueName: \"kubernetes.io/projected/ceafdaf2-b0d8-4c58-b0cc-e6d5ae54168f-kube-api-access-wqh9r\") pod \"ceafdaf2-b0d8-4c58-b0cc-e6d5ae54168f\" (UID: \"ceafdaf2-b0d8-4c58-b0cc-e6d5ae54168f\") " Oct 29 23:02:23.344039 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:23.344002 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ff674d5-04a8-4852-a166-2dbaac9276f3-kube-api-access-bf2jj\" (UniqueName: \"kubernetes.io/projected/9ff674d5-04a8-4852-a166-2dbaac9276f3-kube-api-access-bf2jj\") pod \"9ff674d5-04a8-4852-a166-2dbaac9276f3\" (UID: \"9ff674d5-04a8-4852-a166-2dbaac9276f3\") " Oct 29 23:02:23.344552 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:23.344523 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ceafdaf2-b0d8-4c58-b0cc-e6d5ae54168f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ceafdaf2-b0d8-4c58-b0cc-e6d5ae54168f-plugins-config-volume\") pod \"ceafdaf2-b0d8-4c58-b0cc-e6d5ae54168f\" (UID: \"ceafdaf2-b0d8-4c58-b0cc-e6d5ae54168f\") " Oct 29 23:02:23.344720 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:23.344676 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ff674d5-04a8-4852-a166-2dbaac9276f3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ff674d5-04a8-4852-a166-2dbaac9276f3-plugins-config-volume\") pod \"9ff674d5-04a8-4852-a166-2dbaac9276f3\" (UID: \"9ff674d5-04a8-4852-a166-2dbaac9276f3\") " Oct 29 23:02:23.345798 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:23.345760 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ff674d5-04a8-4852-a166-2dbaac9276f3-plugins-config-volume" pod "9ff674d5-04a8-4852-a166-2dbaac9276f3" (UID: "9ff674d5-04a8-4852-a166-2dbaac9276f3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:02:23.346665 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:23.346636 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ceafdaf2-b0d8-4c58-b0cc-e6d5ae54168f-plugins-config-volume" pod "ceafdaf2-b0d8-4c58-b0cc-e6d5ae54168f" (UID: "ceafdaf2-b0d8-4c58-b0cc-e6d5ae54168f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:02:23.347641 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:23.347620 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ff674d5-04a8-4852-a166-2dbaac9276f3-kube-api-access-bf2jj" pod "9ff674d5-04a8-4852-a166-2dbaac9276f3" (UID: "9ff674d5-04a8-4852-a166-2dbaac9276f3"). InnerVolumeSpecName "kube-api-access-bf2jj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:02:23.348508 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:23.348484 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ceafdaf2-b0d8-4c58-b0cc-e6d5ae54168f-kube-api-access-wqh9r" pod "ceafdaf2-b0d8-4c58-b0cc-e6d5ae54168f" (UID: "ceafdaf2-b0d8-4c58-b0cc-e6d5ae54168f"). InnerVolumeSpecName "kube-api-access-wqh9r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:02:23.447298 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:23.447271 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bf2jj\" (UniqueName: \"kubernetes.io/projected/9ff674d5-04a8-4852-a166-2dbaac9276f3-kube-api-access-bf2jj\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:02:23.447454 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:23.447306 5057 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ceafdaf2-b0d8-4c58-b0cc-e6d5ae54168f-plugins-config-volume\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:02:23.447454 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:23.447326 5057 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ff674d5-04a8-4852-a166-2dbaac9276f3-plugins-config-volume\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:02:23.447454 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:23.447354 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wqh9r\" (UniqueName: \"kubernetes.io/projected/ceafdaf2-b0d8-4c58-b0cc-e6d5ae54168f-kube-api-access-wqh9r\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:02:23.459154 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:23.459116 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:02:23.597301 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:23.597264 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:02:23.746396 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:23.746299 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:02:24.103209 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:24.103115 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:02:24.195674 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:24.195641 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-13aa41-6ffbb89c47-dbncv" Oct 29 23:02:24.196181 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:24.195678 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-13aa41-6ffbb89c47-dbncv" event={"ID":"9ff674d5-04a8-4852-a166-2dbaac9276f3","Type":"ContainerDied","Data":"a60349b82534165b75145c5772098ad05d3bd8bad1c2ceb83062355a818797ef"} Oct 29 23:02:24.196181 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:24.195750 5057 scope.go:122] "RemoveContainer" containerID="68355692154c11ac42a465bc9cd3770aefcc94566415aff948d787ef46ece438" Oct 29 23:02:24.196181 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:24.195834 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-13aa41-85f7584cdb-t2vz5" Oct 29 23:02:24.257629 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:24.257597 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-13aa41-6ffbb89c47-dbncv"] Oct 29 23:02:24.270278 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:24.270245 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-13aa41-6ffbb89c47-dbncv"] Oct 29 23:02:24.298658 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:24.298630 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-13aa41-85f7584cdb-t2vz5"] Oct 29 23:02:24.305710 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:24.305682 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-13aa41-85f7584cdb-t2vz5"] Oct 29 23:02:25.331548 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:25.331518 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ff674d5-04a8-4852-a166-2dbaac9276f3" path="/var/lib/kubelet/pods/9ff674d5-04a8-4852-a166-2dbaac9276f3/volumes" Oct 29 23:02:25.332035 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:25.332003 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ceafdaf2-b0d8-4c58-b0cc-e6d5ae54168f" path="/var/lib/kubelet/pods/ceafdaf2-b0d8-4c58-b0cc-e6d5ae54168f/volumes" Oct 29 23:02:25.429438 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:25.429404 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:02:26.923930 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:26.923827 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:02:27.039078 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:27.039040 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:02:27.285588 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:27.285527 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:02:27.416492 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:27.416458 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:02:29.759205 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:29.759167 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:02:29.874401 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:29.874362 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:02:29.988276 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:29.988215 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:02:30.110313 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:30.109943 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:02:30.219928 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:30.219895 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:02:30.699549 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:30.699509 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:02:30.858496 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:30.851319 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:02:31.084712 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:31.084625 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:02:31.298291 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:31.298258 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:02:31.410196 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:31.410114 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:02:31.613050 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:31.613012 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-13aa41-openshift-default-59db78b859-tk95q"] Oct 29 23:02:31.613292 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:31.613261 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-13aa41-openshift-default-59db78b859-tk95q" podUID="cc933bed-efee-4aad-9d65-8ee1d04f846b" containerName="istio-proxy" containerID="cri-o://ed28fea70905e1d161ee8c13b156fc5a439646b643dd661c4a7dc6dc56c78747" gracePeriod=30 Oct 29 23:02:31.643567 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:31.643447 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:02:31.904853 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:31.900658 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:02:32.139282 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:32.139252 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-7d18f7-595cb9f868-ps49n" Oct 29 23:02:32.139732 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:32.139713 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7d18f7-9fb86cbf9-wshhh" Oct 29 23:02:32.506600 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:32.506165 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:02:32.721560 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:32.721517 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:02:32.851104 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:32.850971 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:02:33.210919 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:33.210877 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:02:33.827150 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:33.827112 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:02:33.952130 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:33.952090 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:02:34.511387 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:34.511087 5057 patch_prober.go:29] interesting pod/e2e-ipp-b-13aa41-openshift-default-59db78b859-tk95q container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 23:02:34.511387 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:34.511165 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-13aa41-openshift-default-59db78b859-tk95q" podUID="cc933bed-efee-4aad-9d65-8ee1d04f846b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 23:02:34.577510 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:34.577476 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:02:34.702577 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:34.702541 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:02:35.108528 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:35.108493 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:02:35.718099 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:35.718058 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:02:35.838062 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:35.838028 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:02:36.165444 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:36.165300 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:02:36.318298 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:36.318264 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:02:36.524024 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:36.523985 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:02:36.951750 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:36.951723 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-13aa41-openshift-default-59db78b859-tk95q" Oct 29 23:02:37.050058 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:37.050017 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc933bed-efee-4aad-9d65-8ee1d04f846b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc933bed-efee-4aad-9d65-8ee1d04f846b-workload-socket\") pod \"cc933bed-efee-4aad-9d65-8ee1d04f846b\" (UID: \"cc933bed-efee-4aad-9d65-8ee1d04f846b\") " Oct 29 23:02:37.050256 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:37.050185 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc933bed-efee-4aad-9d65-8ee1d04f846b-kube-api-access-b4gs6\" (UniqueName: \"kubernetes.io/projected/cc933bed-efee-4aad-9d65-8ee1d04f846b-kube-api-access-b4gs6\") pod \"cc933bed-efee-4aad-9d65-8ee1d04f846b\" (UID: \"cc933bed-efee-4aad-9d65-8ee1d04f846b\") " Oct 29 23:02:37.050256 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:37.050241 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc933bed-efee-4aad-9d65-8ee1d04f846b-istio-token\" (UniqueName: \"kubernetes.io/projected/cc933bed-efee-4aad-9d65-8ee1d04f846b-istio-token\") pod \"cc933bed-efee-4aad-9d65-8ee1d04f846b\" (UID: \"cc933bed-efee-4aad-9d65-8ee1d04f846b\") " Oct 29 23:02:37.050761 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:37.050417 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc933bed-efee-4aad-9d65-8ee1d04f846b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc933bed-efee-4aad-9d65-8ee1d04f846b-workload-certs\") pod \"cc933bed-efee-4aad-9d65-8ee1d04f846b\" (UID: \"cc933bed-efee-4aad-9d65-8ee1d04f846b\") " Oct 29 23:02:37.050851 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:37.050757 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc933bed-efee-4aad-9d65-8ee1d04f846b-workload-socket" pod "cc933bed-efee-4aad-9d65-8ee1d04f846b" (UID: "cc933bed-efee-4aad-9d65-8ee1d04f846b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:02:37.050851 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:37.050786 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc933bed-efee-4aad-9d65-8ee1d04f846b-workload-certs" pod "cc933bed-efee-4aad-9d65-8ee1d04f846b" (UID: "cc933bed-efee-4aad-9d65-8ee1d04f846b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:02:37.050960 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:37.050897 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc933bed-efee-4aad-9d65-8ee1d04f846b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc933bed-efee-4aad-9d65-8ee1d04f846b-istiod-ca-cert\") pod \"cc933bed-efee-4aad-9d65-8ee1d04f846b\" (UID: \"cc933bed-efee-4aad-9d65-8ee1d04f846b\") " Oct 29 23:02:37.051032 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:37.051012 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc933bed-efee-4aad-9d65-8ee1d04f846b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc933bed-efee-4aad-9d65-8ee1d04f846b-credential-socket\") pod \"cc933bed-efee-4aad-9d65-8ee1d04f846b\" (UID: \"cc933bed-efee-4aad-9d65-8ee1d04f846b\") " Oct 29 23:02:37.051289 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:37.051263 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc933bed-efee-4aad-9d65-8ee1d04f846b-credential-socket" pod "cc933bed-efee-4aad-9d65-8ee1d04f846b" (UID: "cc933bed-efee-4aad-9d65-8ee1d04f846b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:02:37.051404 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:37.051263 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc933bed-efee-4aad-9d65-8ee1d04f846b-istiod-ca-cert" pod "cc933bed-efee-4aad-9d65-8ee1d04f846b" (UID: "cc933bed-efee-4aad-9d65-8ee1d04f846b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:02:37.051513 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:37.051493 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc933bed-efee-4aad-9d65-8ee1d04f846b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc933bed-efee-4aad-9d65-8ee1d04f846b-istio-data\") pod \"cc933bed-efee-4aad-9d65-8ee1d04f846b\" (UID: \"cc933bed-efee-4aad-9d65-8ee1d04f846b\") " Oct 29 23:02:37.051709 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:37.051679 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc933bed-efee-4aad-9d65-8ee1d04f846b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc933bed-efee-4aad-9d65-8ee1d04f846b-istio-envoy\") pod \"cc933bed-efee-4aad-9d65-8ee1d04f846b\" (UID: \"cc933bed-efee-4aad-9d65-8ee1d04f846b\") " Oct 29 23:02:37.051874 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:37.051856 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cc933bed-efee-4aad-9d65-8ee1d04f846b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc933bed-efee-4aad-9d65-8ee1d04f846b-istio-podinfo\") pod \"cc933bed-efee-4aad-9d65-8ee1d04f846b\" (UID: \"cc933bed-efee-4aad-9d65-8ee1d04f846b\") " Oct 29 23:02:37.052789 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:37.051527 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc933bed-efee-4aad-9d65-8ee1d04f846b-istio-data" pod "cc933bed-efee-4aad-9d65-8ee1d04f846b" (UID: "cc933bed-efee-4aad-9d65-8ee1d04f846b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:02:37.052910 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:37.052883 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc933bed-efee-4aad-9d65-8ee1d04f846b-kube-api-access-b4gs6" pod "cc933bed-efee-4aad-9d65-8ee1d04f846b" (UID: "cc933bed-efee-4aad-9d65-8ee1d04f846b"). InnerVolumeSpecName "kube-api-access-b4gs6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:02:37.053435 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:37.053410 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc933bed-efee-4aad-9d65-8ee1d04f846b-istio-token" pod "cc933bed-efee-4aad-9d65-8ee1d04f846b" (UID: "cc933bed-efee-4aad-9d65-8ee1d04f846b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:02:37.053655 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:37.053619 5057 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc933bed-efee-4aad-9d65-8ee1d04f846b-workload-certs\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:02:37.053797 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:37.053780 5057 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc933bed-efee-4aad-9d65-8ee1d04f846b-istiod-ca-cert\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:02:37.053927 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:37.053910 5057 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc933bed-efee-4aad-9d65-8ee1d04f846b-credential-socket\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:02:37.054043 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:37.054017 5057 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc933bed-efee-4aad-9d65-8ee1d04f846b-istio-data\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:02:37.054157 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:37.054144 5057 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc933bed-efee-4aad-9d65-8ee1d04f846b-workload-socket\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:02:37.054264 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:37.054250 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b4gs6\" (UniqueName: \"kubernetes.io/projected/cc933bed-efee-4aad-9d65-8ee1d04f846b-kube-api-access-b4gs6\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:02:37.055060 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:37.055025 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc933bed-efee-4aad-9d65-8ee1d04f846b-istio-envoy" pod "cc933bed-efee-4aad-9d65-8ee1d04f846b" (UID: "cc933bed-efee-4aad-9d65-8ee1d04f846b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:02:37.055398 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:37.055326 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cc933bed-efee-4aad-9d65-8ee1d04f846b-istio-podinfo" pod "cc933bed-efee-4aad-9d65-8ee1d04f846b" (UID: "cc933bed-efee-4aad-9d65-8ee1d04f846b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:02:37.069308 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:37.069234 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:02:37.156131 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:37.156099 5057 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc933bed-efee-4aad-9d65-8ee1d04f846b-istio-podinfo\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:02:37.156131 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:37.156127 5057 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc933bed-efee-4aad-9d65-8ee1d04f846b-istio-token\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:02:37.156131 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:37.156135 5057 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc933bed-efee-4aad-9d65-8ee1d04f846b-istio-envoy\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:02:37.209273 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:37.209235 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:02:37.325602 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:37.325507 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:02:37.581400 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:37.574443 5057 generic.go:356] "Generic (PLEG): container finished" podID="cc933bed-efee-4aad-9d65-8ee1d04f846b" containerID="ed28fea70905e1d161ee8c13b156fc5a439646b643dd661c4a7dc6dc56c78747" exitCode=0 Oct 29 23:02:37.581400 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:37.574571 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-13aa41-openshift-default-59db78b859-tk95q" event={"ID":"cc933bed-efee-4aad-9d65-8ee1d04f846b","Type":"ContainerDied","Data":"ed28fea70905e1d161ee8c13b156fc5a439646b643dd661c4a7dc6dc56c78747"} Oct 29 23:02:37.599356 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:37.585709 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-13aa41-openshift-default-59db78b859-tk95q" event={"ID":"cc933bed-efee-4aad-9d65-8ee1d04f846b","Type":"ContainerDied","Data":"3abc7f2f46c87edaa77f7f9161e64eeb22f752e38d8da4018a3e1d9b1c2b019f"} Oct 29 23:02:37.599356 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:37.585778 5057 scope.go:122] "RemoveContainer" containerID="ed28fea70905e1d161ee8c13b156fc5a439646b643dd661c4a7dc6dc56c78747" Oct 29 23:02:37.599356 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:37.586027 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-13aa41-openshift-default-59db78b859-tk95q" Oct 29 23:02:37.672355 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:37.652129 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-13aa41-openshift-default-59db78b859-tk95q"] Oct 29 23:02:37.672355 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:37.662536 5057 scope.go:122] "RemoveContainer" containerID="ed28fea70905e1d161ee8c13b156fc5a439646b643dd661c4a7dc6dc56c78747" Oct 29 23:02:37.672355 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:02:37.663827 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed28fea70905e1d161ee8c13b156fc5a439646b643dd661c4a7dc6dc56c78747\": container with ID starting with ed28fea70905e1d161ee8c13b156fc5a439646b643dd661c4a7dc6dc56c78747 not found: ID does not exist" containerID="ed28fea70905e1d161ee8c13b156fc5a439646b643dd661c4a7dc6dc56c78747" Oct 29 23:02:37.672355 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:37.663872 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed28fea70905e1d161ee8c13b156fc5a439646b643dd661c4a7dc6dc56c78747"} err="failed to get container status \"ed28fea70905e1d161ee8c13b156fc5a439646b643dd661c4a7dc6dc56c78747\": rpc error: code = NotFound desc = could not find container \"ed28fea70905e1d161ee8c13b156fc5a439646b643dd661c4a7dc6dc56c78747\": container with ID starting with ed28fea70905e1d161ee8c13b156fc5a439646b643dd661c4a7dc6dc56c78747 not found: ID does not exist" Oct 29 23:02:37.681351 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:37.679016 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-13aa41-openshift-default-59db78b859-tk95q"] Oct 29 23:02:38.195945 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:38.195913 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:02:38.329448 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:38.329410 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:02:39.265786 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:39.265753 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:02:39.332450 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:39.332419 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc933bed-efee-4aad-9d65-8ee1d04f846b" path="/var/lib/kubelet/pods/cc933bed-efee-4aad-9d65-8ee1d04f846b/volumes" Oct 29 23:02:39.991037 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:39.990994 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:02:40.116195 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:40.116156 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:02:40.632034 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:40.631995 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:02:40.747675 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:40.747602 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:02:41.012757 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:41.012665 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:02:41.135871 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:41.135832 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:02:41.191830 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:41.191799 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-d67deb-d675bc8c9-v9tg7"] Oct 29 23:02:41.192696 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:41.192669 5057 cpu_manager.go:414] "containerMap: removing container" podUID="9ff674d5-04a8-4852-a166-2dbaac9276f3" containerName="payload-pre-processing" Oct 29 23:02:41.192696 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:41.192687 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ff674d5-04a8-4852-a166-2dbaac9276f3" containerName="payload-pre-processing" Oct 29 23:02:41.192836 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:41.192713 5057 cpu_manager.go:414] "containerMap: removing container" podUID="0bcfdeb7-fb96-4c04-ae29-459ae9d29c65" containerName="payload-pre-processing" Oct 29 23:02:41.192836 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:41.192719 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0bcfdeb7-fb96-4c04-ae29-459ae9d29c65" containerName="payload-pre-processing" Oct 29 23:02:41.192836 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:41.192737 5057 cpu_manager.go:414] "containerMap: removing container" podUID="cc933bed-efee-4aad-9d65-8ee1d04f846b" containerName="istio-proxy" Oct 29 23:02:41.192836 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:41.192742 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc933bed-efee-4aad-9d65-8ee1d04f846b" containerName="istio-proxy" Oct 29 23:02:41.192836 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:41.192775 5057 cpu_manager.go:414] "containerMap: removing container" podUID="ceafdaf2-b0d8-4c58-b0cc-e6d5ae54168f" containerName="payload-processing" Oct 29 23:02:41.192836 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:41.192781 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ceafdaf2-b0d8-4c58-b0cc-e6d5ae54168f" containerName="payload-processing" Oct 29 23:02:41.192836 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:41.192810 5057 cpu_manager.go:414] "containerMap: removing container" podUID="643465d1-5a26-4f82-ba35-8c9de37291cc" containerName="payload-processing" Oct 29 23:02:41.192836 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:41.192818 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="643465d1-5a26-4f82-ba35-8c9de37291cc" containerName="payload-processing" Oct 29 23:02:41.193075 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:41.192985 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="643465d1-5a26-4f82-ba35-8c9de37291cc" containerName="payload-processing" Oct 29 23:02:41.193075 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:41.192997 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ff674d5-04a8-4852-a166-2dbaac9276f3" containerName="payload-pre-processing" Oct 29 23:02:41.193075 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:41.193007 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="0bcfdeb7-fb96-4c04-ae29-459ae9d29c65" containerName="payload-pre-processing" Oct 29 23:02:41.193075 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:41.193017 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc933bed-efee-4aad-9d65-8ee1d04f846b" containerName="istio-proxy" Oct 29 23:02:41.193075 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:41.193030 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="ceafdaf2-b0d8-4c58-b0cc-e6d5ae54168f" containerName="payload-processing" Oct 29 23:02:41.198492 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:41.198473 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d67deb-d675bc8c9-v9tg7" Oct 29 23:02:41.215943 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:41.215917 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-d67deb-d675bc8c9-v9tg7"] Oct 29 23:02:41.219539 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:41.219519 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-d67deb\"" Oct 29 23:02:41.219674 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:41.219562 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-d67deb-dockercfg-827m2\"" Oct 29 23:02:41.371411 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:41.353994 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-d67deb-8598cdf4b8-jr6n7"] Oct 29 23:02:41.403527 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:41.389300 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9gbg\" (UniqueName: \"kubernetes.io/projected/cd71b31d-7725-420c-94ab-f94eb006b35f-kube-api-access-m9gbg\") pod \"payload-pre-processing-e2e-praxis-a-d67deb-d675bc8c9-v9tg7\" (UID: \"cd71b31d-7725-420c-94ab-f94eb006b35f\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d67deb-d675bc8c9-v9tg7" Oct 29 23:02:41.403527 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:41.389630 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd71b31d-7725-420c-94ab-f94eb006b35f-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-d67deb-d675bc8c9-v9tg7\" (UID: \"cd71b31d-7725-420c-94ab-f94eb006b35f\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d67deb-d675bc8c9-v9tg7" Oct 29 23:02:41.445434 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:41.445403 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-d67deb-8598cdf4b8-jr6n7" Oct 29 23:02:41.499886 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:41.494205 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9gbg\" (UniqueName: \"kubernetes.io/projected/cd71b31d-7725-420c-94ab-f94eb006b35f-kube-api-access-m9gbg\") pod \"payload-pre-processing-e2e-praxis-a-d67deb-d675bc8c9-v9tg7\" (UID: \"cd71b31d-7725-420c-94ab-f94eb006b35f\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d67deb-d675bc8c9-v9tg7" Oct 29 23:02:41.499886 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:41.494417 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnvgk\" (UniqueName: \"kubernetes.io/projected/b4535855-9770-43ca-abb3-3ee2a56d03d3-kube-api-access-nnvgk\") pod \"payload-processing-e2e-praxis-a-d67deb-8598cdf4b8-jr6n7\" (UID: \"b4535855-9770-43ca-abb3-3ee2a56d03d3\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-d67deb-8598cdf4b8-jr6n7" Oct 29 23:02:41.499886 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:41.494493 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4535855-9770-43ca-abb3-3ee2a56d03d3-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-d67deb-8598cdf4b8-jr6n7\" (UID: \"b4535855-9770-43ca-abb3-3ee2a56d03d3\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-d67deb-8598cdf4b8-jr6n7" Oct 29 23:02:41.499886 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:41.494632 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd71b31d-7725-420c-94ab-f94eb006b35f-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-d67deb-d675bc8c9-v9tg7\" (UID: \"cd71b31d-7725-420c-94ab-f94eb006b35f\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d67deb-d675bc8c9-v9tg7" Oct 29 23:02:41.499886 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:41.495528 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd71b31d-7725-420c-94ab-f94eb006b35f-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-d67deb-d675bc8c9-v9tg7\" (UID: \"cd71b31d-7725-420c-94ab-f94eb006b35f\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d67deb-d675bc8c9-v9tg7" Oct 29 23:02:41.559150 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:41.559105 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9gbg\" (UniqueName: \"kubernetes.io/projected/cd71b31d-7725-420c-94ab-f94eb006b35f-kube-api-access-m9gbg\") pod \"payload-pre-processing-e2e-praxis-a-d67deb-d675bc8c9-v9tg7\" (UID: \"cd71b31d-7725-420c-94ab-f94eb006b35f\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d67deb-d675bc8c9-v9tg7" Oct 29 23:02:41.599615 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:41.599588 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-mvnxc"] Oct 29 23:02:41.601819 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:41.601787 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnvgk\" (UniqueName: \"kubernetes.io/projected/b4535855-9770-43ca-abb3-3ee2a56d03d3-kube-api-access-nnvgk\") pod \"payload-processing-e2e-praxis-a-d67deb-8598cdf4b8-jr6n7\" (UID: \"b4535855-9770-43ca-abb3-3ee2a56d03d3\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-d67deb-8598cdf4b8-jr6n7" Oct 29 23:02:41.601975 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:41.601886 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4535855-9770-43ca-abb3-3ee2a56d03d3-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-d67deb-8598cdf4b8-jr6n7\" (UID: \"b4535855-9770-43ca-abb3-3ee2a56d03d3\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-d67deb-8598cdf4b8-jr6n7" Oct 29 23:02:41.603326 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:41.603298 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4535855-9770-43ca-abb3-3ee2a56d03d3-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-d67deb-8598cdf4b8-jr6n7\" (UID: \"b4535855-9770-43ca-abb3-3ee2a56d03d3\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-d67deb-8598cdf4b8-jr6n7" Oct 29 23:02:41.609029 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:41.609002 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-d67deb-8598cdf4b8-jr6n7"] Oct 29 23:02:41.609254 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:41.609228 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-mvnxc"] Oct 29 23:02:41.609513 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:41.609492 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-mvnxc" Oct 29 23:02:41.664734 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:41.656632 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-0266dd-openshift-default-dockercfg-gr5jd\"" Oct 29 23:02:41.664734 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:41.657997 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnvgk\" (UniqueName: \"kubernetes.io/projected/b4535855-9770-43ca-abb3-3ee2a56d03d3-kube-api-access-nnvgk\") pod \"payload-processing-e2e-praxis-a-d67deb-8598cdf4b8-jr6n7\" (UID: \"b4535855-9770-43ca-abb3-3ee2a56d03d3\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-d67deb-8598cdf4b8-jr6n7" Oct 29 23:02:41.664734 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:41.664131 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:02:41.719930 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:41.719238 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70201148-712a-4488-9b2c-2d2fdcd7f5a4-workload-certs\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-mvnxc\" (UID: \"70201148-712a-4488-9b2c-2d2fdcd7f5a4\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-mvnxc" Oct 29 23:02:41.719930 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:41.719313 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70201148-712a-4488-9b2c-2d2fdcd7f5a4-istio-token\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-mvnxc\" (UID: \"70201148-712a-4488-9b2c-2d2fdcd7f5a4\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-mvnxc" Oct 29 23:02:41.719930 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:41.719432 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70201148-712a-4488-9b2c-2d2fdcd7f5a4-istio-envoy\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-mvnxc\" (UID: \"70201148-712a-4488-9b2c-2d2fdcd7f5a4\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-mvnxc" Oct 29 23:02:41.730246 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:41.724823 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70201148-712a-4488-9b2c-2d2fdcd7f5a4-credential-socket\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-mvnxc\" (UID: \"70201148-712a-4488-9b2c-2d2fdcd7f5a4\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-mvnxc" Oct 29 23:02:41.730246 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:41.724878 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v44zw\" (UniqueName: \"kubernetes.io/projected/70201148-712a-4488-9b2c-2d2fdcd7f5a4-kube-api-access-v44zw\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-mvnxc\" (UID: \"70201148-712a-4488-9b2c-2d2fdcd7f5a4\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-mvnxc" Oct 29 23:02:41.730246 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:41.725011 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70201148-712a-4488-9b2c-2d2fdcd7f5a4-istio-data\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-mvnxc\" (UID: \"70201148-712a-4488-9b2c-2d2fdcd7f5a4\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-mvnxc" Oct 29 23:02:41.730246 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:41.725057 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70201148-712a-4488-9b2c-2d2fdcd7f5a4-istiod-ca-cert\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-mvnxc\" (UID: \"70201148-712a-4488-9b2c-2d2fdcd7f5a4\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-mvnxc" Oct 29 23:02:41.730246 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:41.725125 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70201148-712a-4488-9b2c-2d2fdcd7f5a4-workload-socket\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-mvnxc\" (UID: \"70201148-712a-4488-9b2c-2d2fdcd7f5a4\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-mvnxc" Oct 29 23:02:41.730246 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:41.725189 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70201148-712a-4488-9b2c-2d2fdcd7f5a4-istio-podinfo\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-mvnxc\" (UID: \"70201148-712a-4488-9b2c-2d2fdcd7f5a4\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-mvnxc" Oct 29 23:02:41.795856 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:41.795812 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-d67deb-8598cdf4b8-jr6n7" Oct 29 23:02:41.796067 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:41.796021 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:02:41.832183 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:41.826860 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70201148-712a-4488-9b2c-2d2fdcd7f5a4-workload-certs\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-mvnxc\" (UID: \"70201148-712a-4488-9b2c-2d2fdcd7f5a4\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-mvnxc" Oct 29 23:02:41.832183 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:41.826928 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70201148-712a-4488-9b2c-2d2fdcd7f5a4-istio-token\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-mvnxc\" (UID: \"70201148-712a-4488-9b2c-2d2fdcd7f5a4\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-mvnxc" Oct 29 23:02:41.832183 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:41.827009 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70201148-712a-4488-9b2c-2d2fdcd7f5a4-istio-envoy\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-mvnxc\" (UID: \"70201148-712a-4488-9b2c-2d2fdcd7f5a4\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-mvnxc" Oct 29 23:02:41.832183 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:41.827066 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70201148-712a-4488-9b2c-2d2fdcd7f5a4-credential-socket\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-mvnxc\" (UID: \"70201148-712a-4488-9b2c-2d2fdcd7f5a4\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-mvnxc" Oct 29 23:02:41.832183 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:41.827098 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v44zw\" (UniqueName: \"kubernetes.io/projected/70201148-712a-4488-9b2c-2d2fdcd7f5a4-kube-api-access-v44zw\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-mvnxc\" (UID: \"70201148-712a-4488-9b2c-2d2fdcd7f5a4\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-mvnxc" Oct 29 23:02:41.832183 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:41.827200 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70201148-712a-4488-9b2c-2d2fdcd7f5a4-istio-data\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-mvnxc\" (UID: \"70201148-712a-4488-9b2c-2d2fdcd7f5a4\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-mvnxc" Oct 29 23:02:41.832183 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:41.827237 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70201148-712a-4488-9b2c-2d2fdcd7f5a4-istiod-ca-cert\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-mvnxc\" (UID: \"70201148-712a-4488-9b2c-2d2fdcd7f5a4\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-mvnxc" Oct 29 23:02:41.832183 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:41.827282 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70201148-712a-4488-9b2c-2d2fdcd7f5a4-workload-socket\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-mvnxc\" (UID: \"70201148-712a-4488-9b2c-2d2fdcd7f5a4\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-mvnxc" Oct 29 23:02:41.832183 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:41.827361 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70201148-712a-4488-9b2c-2d2fdcd7f5a4-istio-podinfo\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-mvnxc\" (UID: \"70201148-712a-4488-9b2c-2d2fdcd7f5a4\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-mvnxc" Oct 29 23:02:41.832183 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:41.828254 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70201148-712a-4488-9b2c-2d2fdcd7f5a4-credential-socket\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-mvnxc\" (UID: \"70201148-712a-4488-9b2c-2d2fdcd7f5a4\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-mvnxc" Oct 29 23:02:41.832183 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:41.828512 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70201148-712a-4488-9b2c-2d2fdcd7f5a4-workload-certs\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-mvnxc\" (UID: \"70201148-712a-4488-9b2c-2d2fdcd7f5a4\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-mvnxc" Oct 29 23:02:41.832183 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:41.829352 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70201148-712a-4488-9b2c-2d2fdcd7f5a4-istio-data\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-mvnxc\" (UID: \"70201148-712a-4488-9b2c-2d2fdcd7f5a4\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-mvnxc" Oct 29 23:02:41.832183 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:41.830223 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70201148-712a-4488-9b2c-2d2fdcd7f5a4-istiod-ca-cert\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-mvnxc\" (UID: \"70201148-712a-4488-9b2c-2d2fdcd7f5a4\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-mvnxc" Oct 29 23:02:41.832183 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:41.830521 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70201148-712a-4488-9b2c-2d2fdcd7f5a4-workload-socket\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-mvnxc\" (UID: \"70201148-712a-4488-9b2c-2d2fdcd7f5a4\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-mvnxc" Oct 29 23:02:41.832183 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:41.831685 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d67deb-d675bc8c9-v9tg7" Oct 29 23:02:41.834719 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:41.834693 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70201148-712a-4488-9b2c-2d2fdcd7f5a4-istio-podinfo\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-mvnxc\" (UID: \"70201148-712a-4488-9b2c-2d2fdcd7f5a4\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-mvnxc" Oct 29 23:02:41.836165 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:41.835788 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70201148-712a-4488-9b2c-2d2fdcd7f5a4-istio-envoy\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-mvnxc\" (UID: \"70201148-712a-4488-9b2c-2d2fdcd7f5a4\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-mvnxc" Oct 29 23:02:41.859105 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:41.859010 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v44zw\" (UniqueName: \"kubernetes.io/projected/70201148-712a-4488-9b2c-2d2fdcd7f5a4-kube-api-access-v44zw\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-mvnxc\" (UID: \"70201148-712a-4488-9b2c-2d2fdcd7f5a4\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-mvnxc" Oct 29 23:02:41.859262 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:41.859227 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70201148-712a-4488-9b2c-2d2fdcd7f5a4-istio-token\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-mvnxc\" (UID: \"70201148-712a-4488-9b2c-2d2fdcd7f5a4\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-mvnxc" Oct 29 23:02:41.942364 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:41.942315 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-mvnxc" Oct 29 23:02:42.198463 ip-10-0-140-74 kubenswrapper[5057]: W1029 23:02:42.198433 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4535855_9770_43ca_abb3_3ee2a56d03d3.slice/crio-211ca9c67377b85cdb5ac47fa801ff23c75b83230170592576b0443d848eceeb WatchSource:0}: Error finding container 211ca9c67377b85cdb5ac47fa801ff23c75b83230170592576b0443d848eceeb: Status 404 returned error can't find the container with id 211ca9c67377b85cdb5ac47fa801ff23c75b83230170592576b0443d848eceeb Oct 29 23:02:42.201737 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:42.201712 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-d67deb-8598cdf4b8-jr6n7"] Oct 29 23:02:42.211754 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:42.211726 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-d67deb-d675bc8c9-v9tg7"] Oct 29 23:02:42.223808 ip-10-0-140-74 kubenswrapper[5057]: W1029 23:02:42.223778 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd71b31d_7725_420c_94ab_f94eb006b35f.slice/crio-6ee0a34a09fa045ee25d5f711f1de0a2e38f463ee3e7689c26f441069bb6b084 WatchSource:0}: Error finding container 6ee0a34a09fa045ee25d5f711f1de0a2e38f463ee3e7689c26f441069bb6b084: Status 404 returned error can't find the container with id 6ee0a34a09fa045ee25d5f711f1de0a2e38f463ee3e7689c26f441069bb6b084 Oct 29 23:02:42.333641 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:42.333594 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-mvnxc"] Oct 29 23:02:42.345066 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:42.344645 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 23:02:42.345066 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:42.344737 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 23:02:42.345066 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:42.344777 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 23:02:42.683560 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:42.683522 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-d67deb-8598cdf4b8-jr6n7" event={"ID":"b4535855-9770-43ca-abb3-3ee2a56d03d3","Type":"ContainerStarted","Data":"adbc57b07e9826106e6717704ae2c272e710f7fa49dede434bc3693b332f16c2"} Oct 29 23:02:42.683997 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:42.683589 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-d67deb-8598cdf4b8-jr6n7" event={"ID":"b4535855-9770-43ca-abb3-3ee2a56d03d3","Type":"ContainerStarted","Data":"211ca9c67377b85cdb5ac47fa801ff23c75b83230170592576b0443d848eceeb"} Oct 29 23:02:42.683997 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:42.683617 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-d67deb-8598cdf4b8-jr6n7" Oct 29 23:02:42.685713 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:42.685681 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d67deb-d675bc8c9-v9tg7" event={"ID":"cd71b31d-7725-420c-94ab-f94eb006b35f","Type":"ContainerStarted","Data":"63e632a5435d8c6d99e09bb85d22e31951cec29bf78cfc7895fb4617c581ba52"} Oct 29 23:02:42.685951 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:42.685732 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d67deb-d675bc8c9-v9tg7" event={"ID":"cd71b31d-7725-420c-94ab-f94eb006b35f","Type":"ContainerStarted","Data":"6ee0a34a09fa045ee25d5f711f1de0a2e38f463ee3e7689c26f441069bb6b084"} Oct 29 23:02:42.685951 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:42.685764 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d67deb-d675bc8c9-v9tg7" Oct 29 23:02:42.688740 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:42.688713 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-mvnxc" event={"ID":"70201148-712a-4488-9b2c-2d2fdcd7f5a4","Type":"ContainerStarted","Data":"3c485a202bd9e063c1dcd7ec8e5ea396107d2888ea4c3ba8d13e7cf6ae646cd8"} Oct 29 23:02:42.688878 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:42.688759 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-mvnxc" event={"ID":"70201148-712a-4488-9b2c-2d2fdcd7f5a4","Type":"ContainerStarted","Data":"f67ab1076fff9e6ec9f86d285fe773c1411aca03e263c348994fd71378d8422a"} Oct 29 23:02:42.724938 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:42.724835 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-d67deb-8598cdf4b8-jr6n7" podStartSLOduration=1.724817217 podStartE2EDuration="1.724817217s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:02:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:02:42.722631348 +0000 UTC m=+1743.505844815" watchObservedRunningTime="2026-10-29 23:02:42.724817217 +0000 UTC m=+1743.508030647" Oct 29 23:02:42.763464 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:42.763030 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d67deb-d675bc8c9-v9tg7" podStartSLOduration=1.763011777 podStartE2EDuration="1.763011777s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:02:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:02:42.758599703 +0000 UTC m=+1743.541813130" watchObservedRunningTime="2026-10-29 23:02:42.763011777 +0000 UTC m=+1743.546225203" Oct 29 23:02:42.801197 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:42.801148 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-mvnxc" podStartSLOduration=1.8011349129999998 podStartE2EDuration="1.801134913s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:02:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:02:42.798280701 +0000 UTC m=+1743.581494127" watchObservedRunningTime="2026-10-29 23:02:42.801134913 +0000 UTC m=+1743.584348339" Oct 29 23:02:42.951762 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:42.951726 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-mvnxc" Oct 29 23:02:43.804928 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:43.804889 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:02:43.968930 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:43.968893 5057 patch_prober.go:29] interesting pod/e2e-praxis-a-0266dd-openshift-default-c89468fc-mvnxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body= Oct 29 23:02:43.969104 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:43.968975 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-mvnxc" podUID="70201148-712a-4488-9b2c-2d2fdcd7f5a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" Oct 29 23:02:44.732907 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:44.732868 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:02:44.858368 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:44.858289 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:02:44.952755 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:44.952720 5057 patch_prober.go:29] interesting pod/e2e-praxis-a-0266dd-openshift-default-c89468fc-mvnxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body= Oct 29 23:02:44.952917 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:44.952785 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-mvnxc" podUID="70201148-712a-4488-9b2c-2d2fdcd7f5a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" Oct 29 23:02:45.952024 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:45.951991 5057 patch_prober.go:29] interesting pod/e2e-praxis-a-0266dd-openshift-default-c89468fc-mvnxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body= Oct 29 23:02:45.952423 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:45.952044 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-mvnxc" podUID="70201148-712a-4488-9b2c-2d2fdcd7f5a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" Oct 29 23:02:46.206899 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:46.206762 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:02:46.951721 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:46.951680 5057 patch_prober.go:29] interesting pod/e2e-praxis-a-0266dd-openshift-default-c89468fc-mvnxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body= Oct 29 23:02:46.951949 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:46.951745 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-mvnxc" podUID="70201148-712a-4488-9b2c-2d2fdcd7f5a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" Oct 29 23:02:47.952567 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:47.952528 5057 patch_prober.go:29] interesting pod/e2e-praxis-a-0266dd-openshift-default-c89468fc-mvnxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body= Oct 29 23:02:47.952977 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:47.952583 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-mvnxc" podUID="70201148-712a-4488-9b2c-2d2fdcd7f5a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" Oct 29 23:02:48.225200 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:48.225114 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:02:48.956532 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:48.956503 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-mvnxc" Oct 29 23:02:48.956996 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:48.956974 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-mvnxc" Oct 29 23:02:48.957662 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:48.957644 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-mvnxc" Oct 29 23:02:49.029715 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:02:49.029677 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d2dnn" containerName="extractor" Oct 29 23:02:50.736090 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:50.735992 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:02:52.829320 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:52.829278 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:02:53.715753 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:53.715724 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-d67deb-8598cdf4b8-jr6n7" Oct 29 23:02:53.716061 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:53.715797 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d67deb-d675bc8c9-v9tg7" Oct 29 23:02:54.489816 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:54.489773 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:02:56.043790 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:56.043750 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:02:57.568525 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:57.568486 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:02:59.105939 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:59.105901 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:02:59.229502 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:59.229464 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:02:59.354512 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:02:59.354445 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:03:00.920695 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:00.920655 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:03:01.531842 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:01.531761 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v2tfm"] Oct 29 23:03:02.986279 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:02.986240 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:03:03.096106 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:03.096067 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:03:03.333125 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:03.333042 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:03:05.020660 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:05.020614 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:03:07.086839 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:07.086753 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:03:07.205394 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:07.205317 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:03:07.348138 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:07.348044 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:03:09.273461 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:09.273400 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:03:09.379198 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:09.379165 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:03:09.546171 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:09.546081 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:03:10.645739 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:10.645698 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-848fb8-6f97b8d6bb-4cdsr"] Oct 29 23:03:10.679384 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:10.679343 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-848fb8-85f6f7cd95-9mhkq"] Oct 29 23:03:10.797826 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:10.797790 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:03:10.913563 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:10.913468 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:03:11.196804 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:11.196730 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:03:11.523038 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:11.522989 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-9dnxn"] Oct 29 23:03:11.530326 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:11.530293 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-9dnxn" Oct 29 23:03:11.586635 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:11.586246 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e1b34c3-955d-4d56-b7c6-64779cf3eae8-istio-envoy\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-9dnxn\" (UID: \"5e1b34c3-955d-4d56-b7c6-64779cf3eae8\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-9dnxn" Oct 29 23:03:11.586635 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:11.586289 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e1b34c3-955d-4d56-b7c6-64779cf3eae8-istio-token\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-9dnxn\" (UID: \"5e1b34c3-955d-4d56-b7c6-64779cf3eae8\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-9dnxn" Oct 29 23:03:11.586635 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:11.586426 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e1b34c3-955d-4d56-b7c6-64779cf3eae8-istio-podinfo\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-9dnxn\" (UID: \"5e1b34c3-955d-4d56-b7c6-64779cf3eae8\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-9dnxn" Oct 29 23:03:11.586635 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:11.586454 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zf96\" (UniqueName: \"kubernetes.io/projected/5e1b34c3-955d-4d56-b7c6-64779cf3eae8-kube-api-access-9zf96\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-9dnxn\" (UID: \"5e1b34c3-955d-4d56-b7c6-64779cf3eae8\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-9dnxn" Oct 29 23:03:11.586635 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:11.586510 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e1b34c3-955d-4d56-b7c6-64779cf3eae8-credential-socket\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-9dnxn\" (UID: \"5e1b34c3-955d-4d56-b7c6-64779cf3eae8\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-9dnxn" Oct 29 23:03:11.586635 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:11.586538 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e1b34c3-955d-4d56-b7c6-64779cf3eae8-workload-certs\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-9dnxn\" (UID: \"5e1b34c3-955d-4d56-b7c6-64779cf3eae8\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-9dnxn" Oct 29 23:03:11.587105 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:11.586842 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e1b34c3-955d-4d56-b7c6-64779cf3eae8-istiod-ca-cert\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-9dnxn\" (UID: \"5e1b34c3-955d-4d56-b7c6-64779cf3eae8\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-9dnxn" Oct 29 23:03:11.587105 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:11.586883 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e1b34c3-955d-4d56-b7c6-64779cf3eae8-istio-data\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-9dnxn\" (UID: \"5e1b34c3-955d-4d56-b7c6-64779cf3eae8\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-9dnxn" Oct 29 23:03:11.587105 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:11.587006 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e1b34c3-955d-4d56-b7c6-64779cf3eae8-workload-socket\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-9dnxn\" (UID: \"5e1b34c3-955d-4d56-b7c6-64779cf3eae8\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-9dnxn" Oct 29 23:03:11.638614 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:11.638584 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-9dnxn"] Oct 29 23:03:11.689468 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:11.689434 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e1b34c3-955d-4d56-b7c6-64779cf3eae8-istiod-ca-cert\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-9dnxn\" (UID: \"5e1b34c3-955d-4d56-b7c6-64779cf3eae8\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-9dnxn" Oct 29 23:03:11.689965 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:11.689941 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e1b34c3-955d-4d56-b7c6-64779cf3eae8-istio-data\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-9dnxn\" (UID: \"5e1b34c3-955d-4d56-b7c6-64779cf3eae8\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-9dnxn" Oct 29 23:03:11.690087 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:11.690061 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e1b34c3-955d-4d56-b7c6-64779cf3eae8-workload-socket\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-9dnxn\" (UID: \"5e1b34c3-955d-4d56-b7c6-64779cf3eae8\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-9dnxn" Oct 29 23:03:11.690187 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:11.690170 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e1b34c3-955d-4d56-b7c6-64779cf3eae8-istio-envoy\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-9dnxn\" (UID: \"5e1b34c3-955d-4d56-b7c6-64779cf3eae8\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-9dnxn" Oct 29 23:03:11.690309 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:11.690292 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e1b34c3-955d-4d56-b7c6-64779cf3eae8-istio-token\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-9dnxn\" (UID: \"5e1b34c3-955d-4d56-b7c6-64779cf3eae8\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-9dnxn" Oct 29 23:03:11.690534 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:11.690509 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e1b34c3-955d-4d56-b7c6-64779cf3eae8-workload-socket\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-9dnxn\" (UID: \"5e1b34c3-955d-4d56-b7c6-64779cf3eae8\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-9dnxn" Oct 29 23:03:11.690662 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:11.690391 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e1b34c3-955d-4d56-b7c6-64779cf3eae8-istiod-ca-cert\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-9dnxn\" (UID: \"5e1b34c3-955d-4d56-b7c6-64779cf3eae8\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-9dnxn" Oct 29 23:03:11.690662 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:11.690516 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e1b34c3-955d-4d56-b7c6-64779cf3eae8-istio-podinfo\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-9dnxn\" (UID: \"5e1b34c3-955d-4d56-b7c6-64779cf3eae8\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-9dnxn" Oct 29 23:03:11.690662 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:11.690562 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e1b34c3-955d-4d56-b7c6-64779cf3eae8-istio-data\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-9dnxn\" (UID: \"5e1b34c3-955d-4d56-b7c6-64779cf3eae8\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-9dnxn" Oct 29 23:03:11.690662 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:11.690588 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zf96\" (UniqueName: \"kubernetes.io/projected/5e1b34c3-955d-4d56-b7c6-64779cf3eae8-kube-api-access-9zf96\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-9dnxn\" (UID: \"5e1b34c3-955d-4d56-b7c6-64779cf3eae8\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-9dnxn" Oct 29 23:03:11.690662 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:11.690637 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e1b34c3-955d-4d56-b7c6-64779cf3eae8-credential-socket\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-9dnxn\" (UID: \"5e1b34c3-955d-4d56-b7c6-64779cf3eae8\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-9dnxn" Oct 29 23:03:11.690662 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:11.690660 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e1b34c3-955d-4d56-b7c6-64779cf3eae8-workload-certs\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-9dnxn\" (UID: \"5e1b34c3-955d-4d56-b7c6-64779cf3eae8\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-9dnxn" Oct 29 23:03:11.691058 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:11.691030 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e1b34c3-955d-4d56-b7c6-64779cf3eae8-credential-socket\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-9dnxn\" (UID: \"5e1b34c3-955d-4d56-b7c6-64779cf3eae8\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-9dnxn" Oct 29 23:03:11.691244 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:11.691211 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e1b34c3-955d-4d56-b7c6-64779cf3eae8-workload-certs\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-9dnxn\" (UID: \"5e1b34c3-955d-4d56-b7c6-64779cf3eae8\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-9dnxn" Oct 29 23:03:11.693702 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:11.693678 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e1b34c3-955d-4d56-b7c6-64779cf3eae8-istio-envoy\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-9dnxn\" (UID: \"5e1b34c3-955d-4d56-b7c6-64779cf3eae8\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-9dnxn" Oct 29 23:03:11.696093 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:11.696050 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e1b34c3-955d-4d56-b7c6-64779cf3eae8-istio-podinfo\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-9dnxn\" (UID: \"5e1b34c3-955d-4d56-b7c6-64779cf3eae8\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-9dnxn" Oct 29 23:03:11.712148 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:11.712119 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zf96\" (UniqueName: \"kubernetes.io/projected/5e1b34c3-955d-4d56-b7c6-64779cf3eae8-kube-api-access-9zf96\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-9dnxn\" (UID: \"5e1b34c3-955d-4d56-b7c6-64779cf3eae8\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-9dnxn" Oct 29 23:03:11.712828 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:11.712804 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e1b34c3-955d-4d56-b7c6-64779cf3eae8-istio-token\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-9dnxn\" (UID: \"5e1b34c3-955d-4d56-b7c6-64779cf3eae8\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-9dnxn" Oct 29 23:03:11.925387 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:11.925284 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-9dnxn" Oct 29 23:03:12.266546 ip-10-0-140-74 kubenswrapper[5057]: W1029 23:03:12.266510 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e1b34c3_955d_4d56_b7c6_64779cf3eae8.slice/crio-a3f26a5e7028f36badf17fefbc78d866918e19020e57154f0671e370343d0821 WatchSource:0}: Error finding container a3f26a5e7028f36badf17fefbc78d866918e19020e57154f0671e370343d0821: Status 404 returned error can't find the container with id a3f26a5e7028f36badf17fefbc78d866918e19020e57154f0671e370343d0821 Oct 29 23:03:12.266889 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:12.266866 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-9dnxn"] Oct 29 23:03:12.269534 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:12.269499 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 23:03:12.269650 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:12.269607 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 23:03:12.269722 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:12.269655 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 23:03:13.017864 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:13.017828 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-9dnxn" event={"ID":"5e1b34c3-955d-4d56-b7c6-64779cf3eae8","Type":"ContainerStarted","Data":"9a282b4eace1a756619b6b733a75d2512324f0a80f5c75f06ff02a438135fbdd"} Oct 29 23:03:13.018288 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:13.017887 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-9dnxn" event={"ID":"5e1b34c3-955d-4d56-b7c6-64779cf3eae8","Type":"ContainerStarted","Data":"a3f26a5e7028f36badf17fefbc78d866918e19020e57154f0671e370343d0821"} Oct 29 23:03:13.024696 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:13.024661 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:03:13.060655 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:13.060487 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-9dnxn" podStartSLOduration=2.06047011 podStartE2EDuration="2.06047011s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:03:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:03:13.057744385 +0000 UTC m=+1773.840957811" watchObservedRunningTime="2026-10-29 23:03:13.06047011 +0000 UTC m=+1773.843683537" Oct 29 23:03:13.926396 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:13.926363 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-9dnxn" Oct 29 23:03:13.927919 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:13.927894 5057 patch_prober.go:29] interesting pod/e2e-praxis-a-0266dd-openshift-default-c89468fc-9dnxn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body= Oct 29 23:03:13.928028 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:13.927951 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-9dnxn" podUID="5e1b34c3-955d-4d56-b7c6-64779cf3eae8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" Oct 29 23:03:14.707676 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:14.692781 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:03:14.944552 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:14.928890 5057 patch_prober.go:29] interesting pod/e2e-praxis-a-0266dd-openshift-default-c89468fc-9dnxn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body= Oct 29 23:03:14.944751 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:14.944599 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-9dnxn" podUID="5e1b34c3-955d-4d56-b7c6-64779cf3eae8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" Oct 29 23:03:15.646508 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:15.646474 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-848fb8-6f97b8d6bb-4cdsr" podUID="cfebb6e3-0e95-4c69-b2a9-ce6e1e64b443" containerName="payload-processing" containerID="cri-o://770fb9068059bf32055d57f10f411412e0d9a74846a92a6d3a23a6f188ea9bd4" gracePeriod=25 Oct 29 23:03:15.702926 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:15.702658 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-848fb8-85f6f7cd95-9mhkq" podUID="7cda8b19-10e8-4efc-a2f2-61ca23d7b78a" containerName="payload-pre-processing" containerID="cri-o://ed6c9f79ce1f67a4e574de4a0dd269e90f96b15970d219d8e4757951c32221b1" gracePeriod=25 Oct 29 23:03:15.926098 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:15.926065 5057 patch_prober.go:29] interesting pod/e2e-praxis-a-0266dd-openshift-default-c89468fc-9dnxn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body= Oct 29 23:03:15.926606 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:15.926125 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-9dnxn" podUID="5e1b34c3-955d-4d56-b7c6-64779cf3eae8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" Oct 29 23:03:16.056798 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:16.056758 5057 generic.go:356] "Generic (PLEG): container finished" podID="7cda8b19-10e8-4efc-a2f2-61ca23d7b78a" containerID="ed6c9f79ce1f67a4e574de4a0dd269e90f96b15970d219d8e4757951c32221b1" exitCode=0 Oct 29 23:03:16.056980 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:16.056918 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-848fb8-85f6f7cd95-9mhkq" event={"ID":"7cda8b19-10e8-4efc-a2f2-61ca23d7b78a","Type":"ContainerDied","Data":"ed6c9f79ce1f67a4e574de4a0dd269e90f96b15970d219d8e4757951c32221b1"} Oct 29 23:03:16.058685 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:16.058658 5057 generic.go:356] "Generic (PLEG): container finished" podID="cfebb6e3-0e95-4c69-b2a9-ce6e1e64b443" containerID="770fb9068059bf32055d57f10f411412e0d9a74846a92a6d3a23a6f188ea9bd4" exitCode=0 Oct 29 23:03:16.058816 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:16.058718 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-848fb8-6f97b8d6bb-4cdsr" event={"ID":"cfebb6e3-0e95-4c69-b2a9-ce6e1e64b443","Type":"ContainerDied","Data":"770fb9068059bf32055d57f10f411412e0d9a74846a92a6d3a23a6f188ea9bd4"} Oct 29 23:03:16.120193 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:16.120162 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-848fb8-6f97b8d6bb-4cdsr" Oct 29 23:03:16.214438 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:16.214406 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cfebb6e3-0e95-4c69-b2a9-ce6e1e64b443-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cfebb6e3-0e95-4c69-b2a9-ce6e1e64b443-plugins-config-volume\") pod \"cfebb6e3-0e95-4c69-b2a9-ce6e1e64b443\" (UID: \"cfebb6e3-0e95-4c69-b2a9-ce6e1e64b443\") " Oct 29 23:03:16.214586 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:16.214524 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cfebb6e3-0e95-4c69-b2a9-ce6e1e64b443-kube-api-access-p2trx\" (UniqueName: \"kubernetes.io/projected/cfebb6e3-0e95-4c69-b2a9-ce6e1e64b443-kube-api-access-p2trx\") pod \"cfebb6e3-0e95-4c69-b2a9-ce6e1e64b443\" (UID: \"cfebb6e3-0e95-4c69-b2a9-ce6e1e64b443\") " Oct 29 23:03:16.216019 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:16.215991 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cfebb6e3-0e95-4c69-b2a9-ce6e1e64b443-plugins-config-volume" pod "cfebb6e3-0e95-4c69-b2a9-ce6e1e64b443" (UID: "cfebb6e3-0e95-4c69-b2a9-ce6e1e64b443"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:03:16.227836 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:16.225480 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfebb6e3-0e95-4c69-b2a9-ce6e1e64b443-kube-api-access-p2trx" pod "cfebb6e3-0e95-4c69-b2a9-ce6e1e64b443" (UID: "cfebb6e3-0e95-4c69-b2a9-ce6e1e64b443"). InnerVolumeSpecName "kube-api-access-p2trx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:03:16.227836 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:16.227755 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-848fb8-85f6f7cd95-9mhkq" Oct 29 23:03:16.316034 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:16.315991 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7cda8b19-10e8-4efc-a2f2-61ca23d7b78a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7cda8b19-10e8-4efc-a2f2-61ca23d7b78a-plugins-config-volume\") pod \"7cda8b19-10e8-4efc-a2f2-61ca23d7b78a\" (UID: \"7cda8b19-10e8-4efc-a2f2-61ca23d7b78a\") " Oct 29 23:03:16.316321 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:16.316294 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7cda8b19-10e8-4efc-a2f2-61ca23d7b78a-plugins-config-volume" pod "7cda8b19-10e8-4efc-a2f2-61ca23d7b78a" (UID: "7cda8b19-10e8-4efc-a2f2-61ca23d7b78a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:03:16.316437 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:16.316416 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7cda8b19-10e8-4efc-a2f2-61ca23d7b78a-kube-api-access-b2j27\" (UniqueName: \"kubernetes.io/projected/7cda8b19-10e8-4efc-a2f2-61ca23d7b78a-kube-api-access-b2j27\") pod \"7cda8b19-10e8-4efc-a2f2-61ca23d7b78a\" (UID: \"7cda8b19-10e8-4efc-a2f2-61ca23d7b78a\") " Oct 29 23:03:16.317412 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:16.317385 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p2trx\" (UniqueName: \"kubernetes.io/projected/cfebb6e3-0e95-4c69-b2a9-ce6e1e64b443-kube-api-access-p2trx\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:03:16.317539 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:16.317436 5057 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7cda8b19-10e8-4efc-a2f2-61ca23d7b78a-plugins-config-volume\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:03:16.317539 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:16.317454 5057 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cfebb6e3-0e95-4c69-b2a9-ce6e1e64b443-plugins-config-volume\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:03:16.319078 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:16.319051 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7cda8b19-10e8-4efc-a2f2-61ca23d7b78a-kube-api-access-b2j27" pod "7cda8b19-10e8-4efc-a2f2-61ca23d7b78a" (UID: "7cda8b19-10e8-4efc-a2f2-61ca23d7b78a"). InnerVolumeSpecName "kube-api-access-b2j27". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:03:16.419495 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:16.419454 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b2j27\" (UniqueName: \"kubernetes.io/projected/7cda8b19-10e8-4efc-a2f2-61ca23d7b78a-kube-api-access-b2j27\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:03:16.697348 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:16.692852 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:03:16.933649 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:16.933509 5057 patch_prober.go:29] interesting pod/e2e-praxis-a-0266dd-openshift-default-c89468fc-9dnxn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body= Oct 29 23:03:16.934376 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:16.934292 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-9dnxn" podUID="5e1b34c3-955d-4d56-b7c6-64779cf3eae8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" Oct 29 23:03:17.071787 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:17.071748 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-848fb8-85f6f7cd95-9mhkq" event={"ID":"7cda8b19-10e8-4efc-a2f2-61ca23d7b78a","Type":"ContainerDied","Data":"d3f262b54e83cc077606f7def9ae70d2d6aff7318c309e8961df8b5749e835ec"} Oct 29 23:03:17.071787 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:17.071778 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-848fb8-85f6f7cd95-9mhkq" Oct 29 23:03:17.072044 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:17.071849 5057 scope.go:122] "RemoveContainer" containerID="ed6c9f79ce1f67a4e574de4a0dd269e90f96b15970d219d8e4757951c32221b1" Oct 29 23:03:17.074099 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:17.074080 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-848fb8-6f97b8d6bb-4cdsr" event={"ID":"cfebb6e3-0e95-4c69-b2a9-ce6e1e64b443","Type":"ContainerDied","Data":"7d26720a62916004e9ec14eb374f50bc71dc1dcce081637bd9a9f1024837bbb3"} Oct 29 23:03:17.074301 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:17.074289 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-848fb8-6f97b8d6bb-4cdsr" Oct 29 23:03:17.124833 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:17.124812 5057 scope.go:122] "RemoveContainer" containerID="770fb9068059bf32055d57f10f411412e0d9a74846a92a6d3a23a6f188ea9bd4" Oct 29 23:03:17.156748 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:17.156701 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-848fb8-85f6f7cd95-9mhkq"] Oct 29 23:03:17.165240 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:17.165213 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-848fb8-85f6f7cd95-9mhkq"] Oct 29 23:03:17.211883 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:17.211849 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-848fb8-6f97b8d6bb-4cdsr"] Oct 29 23:03:17.216979 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:17.216948 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-848fb8-6f97b8d6bb-4cdsr"] Oct 29 23:03:17.335666 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:17.335623 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7cda8b19-10e8-4efc-a2f2-61ca23d7b78a" path="/var/lib/kubelet/pods/7cda8b19-10e8-4efc-a2f2-61ca23d7b78a/volumes" Oct 29 23:03:17.336172 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:17.336151 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cfebb6e3-0e95-4c69-b2a9-ce6e1e64b443" path="/var/lib/kubelet/pods/cfebb6e3-0e95-4c69-b2a9-ce6e1e64b443/volumes" Oct 29 23:03:17.925971 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:17.925933 5057 patch_prober.go:29] interesting pod/e2e-praxis-a-0266dd-openshift-default-c89468fc-9dnxn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body= Oct 29 23:03:17.925971 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:17.926013 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-9dnxn" podUID="5e1b34c3-955d-4d56-b7c6-64779cf3eae8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" Oct 29 23:03:18.931393 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:18.931315 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-9dnxn" Oct 29 23:03:18.931818 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:18.931779 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-9dnxn" Oct 29 23:03:18.932537 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:18.932516 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-9dnxn" Oct 29 23:03:18.946828 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:18.946802 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:03:19.032152 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:03:19.032118 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d2dnn" containerName="extractor" Oct 29 23:03:21.487099 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:21.487056 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:03:23.585770 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:23.585731 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:03:23.758765 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:23.758705 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:03:24.240130 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:24.240094 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:03:24.425063 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:24.425020 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:03:25.455863 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:25.455829 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:03:26.442500 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:26.442452 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-w86t8"] Oct 29 23:03:26.443418 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:26.443385 5057 cpu_manager.go:414] "containerMap: removing container" podUID="7cda8b19-10e8-4efc-a2f2-61ca23d7b78a" containerName="payload-pre-processing" Oct 29 23:03:26.443418 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:26.443409 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7cda8b19-10e8-4efc-a2f2-61ca23d7b78a" containerName="payload-pre-processing" Oct 29 23:03:26.443620 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:26.443439 5057 cpu_manager.go:414] "containerMap: removing container" podUID="cfebb6e3-0e95-4c69-b2a9-ce6e1e64b443" containerName="payload-processing" Oct 29 23:03:26.443620 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:26.443446 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cfebb6e3-0e95-4c69-b2a9-ce6e1e64b443" containerName="payload-processing" Oct 29 23:03:26.443737 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:26.443621 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="7cda8b19-10e8-4efc-a2f2-61ca23d7b78a" containerName="payload-pre-processing" Oct 29 23:03:26.443737 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:26.443633 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="cfebb6e3-0e95-4c69-b2a9-ce6e1e64b443" containerName="payload-processing" Oct 29 23:03:26.449629 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:26.449588 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-w86t8" Oct 29 23:03:26.455301 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:26.455274 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-w86t8"] Oct 29 23:03:26.542585 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:26.542553 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81d1fd17-d835-463f-bd7d-280f05f87d8c-istio-envoy\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-w86t8\" (UID: \"81d1fd17-d835-463f-bd7d-280f05f87d8c\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-w86t8" Oct 29 23:03:26.543037 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:26.542706 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vctjb\" (UniqueName: \"kubernetes.io/projected/81d1fd17-d835-463f-bd7d-280f05f87d8c-kube-api-access-vctjb\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-w86t8\" (UID: \"81d1fd17-d835-463f-bd7d-280f05f87d8c\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-w86t8" Oct 29 23:03:26.543037 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:26.542806 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81d1fd17-d835-463f-bd7d-280f05f87d8c-istio-podinfo\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-w86t8\" (UID: \"81d1fd17-d835-463f-bd7d-280f05f87d8c\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-w86t8" Oct 29 23:03:26.543037 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:26.542908 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81d1fd17-d835-463f-bd7d-280f05f87d8c-istiod-ca-cert\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-w86t8\" (UID: \"81d1fd17-d835-463f-bd7d-280f05f87d8c\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-w86t8" Oct 29 23:03:26.543037 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:26.543003 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81d1fd17-d835-463f-bd7d-280f05f87d8c-istio-data\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-w86t8\" (UID: \"81d1fd17-d835-463f-bd7d-280f05f87d8c\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-w86t8" Oct 29 23:03:26.543037 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:26.543035 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81d1fd17-d835-463f-bd7d-280f05f87d8c-workload-certs\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-w86t8\" (UID: \"81d1fd17-d835-463f-bd7d-280f05f87d8c\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-w86t8" Oct 29 23:03:26.543319 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:26.543068 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81d1fd17-d835-463f-bd7d-280f05f87d8c-istio-token\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-w86t8\" (UID: \"81d1fd17-d835-463f-bd7d-280f05f87d8c\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-w86t8" Oct 29 23:03:26.543319 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:26.543139 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81d1fd17-d835-463f-bd7d-280f05f87d8c-workload-socket\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-w86t8\" (UID: \"81d1fd17-d835-463f-bd7d-280f05f87d8c\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-w86t8" Oct 29 23:03:26.543319 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:26.543165 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81d1fd17-d835-463f-bd7d-280f05f87d8c-credential-socket\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-w86t8\" (UID: \"81d1fd17-d835-463f-bd7d-280f05f87d8c\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-w86t8" Oct 29 23:03:26.644253 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:26.644220 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vctjb\" (UniqueName: \"kubernetes.io/projected/81d1fd17-d835-463f-bd7d-280f05f87d8c-kube-api-access-vctjb\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-w86t8\" (UID: \"81d1fd17-d835-463f-bd7d-280f05f87d8c\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-w86t8" Oct 29 23:03:26.644438 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:26.644281 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81d1fd17-d835-463f-bd7d-280f05f87d8c-istio-podinfo\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-w86t8\" (UID: \"81d1fd17-d835-463f-bd7d-280f05f87d8c\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-w86t8" Oct 29 23:03:26.644438 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:26.644400 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81d1fd17-d835-463f-bd7d-280f05f87d8c-istiod-ca-cert\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-w86t8\" (UID: \"81d1fd17-d835-463f-bd7d-280f05f87d8c\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-w86t8" Oct 29 23:03:26.644561 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:26.644443 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81d1fd17-d835-463f-bd7d-280f05f87d8c-istio-data\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-w86t8\" (UID: \"81d1fd17-d835-463f-bd7d-280f05f87d8c\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-w86t8" Oct 29 23:03:26.644561 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:26.644460 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81d1fd17-d835-463f-bd7d-280f05f87d8c-workload-certs\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-w86t8\" (UID: \"81d1fd17-d835-463f-bd7d-280f05f87d8c\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-w86t8" Oct 29 23:03:26.644652 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:26.644557 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81d1fd17-d835-463f-bd7d-280f05f87d8c-istio-token\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-w86t8\" (UID: \"81d1fd17-d835-463f-bd7d-280f05f87d8c\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-w86t8" Oct 29 23:03:26.644652 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:26.644645 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81d1fd17-d835-463f-bd7d-280f05f87d8c-workload-socket\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-w86t8\" (UID: \"81d1fd17-d835-463f-bd7d-280f05f87d8c\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-w86t8" Oct 29 23:03:26.644758 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:26.644670 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81d1fd17-d835-463f-bd7d-280f05f87d8c-credential-socket\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-w86t8\" (UID: \"81d1fd17-d835-463f-bd7d-280f05f87d8c\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-w86t8" Oct 29 23:03:26.644819 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:26.644770 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81d1fd17-d835-463f-bd7d-280f05f87d8c-istio-envoy\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-w86t8\" (UID: \"81d1fd17-d835-463f-bd7d-280f05f87d8c\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-w86t8" Oct 29 23:03:26.644908 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:26.644878 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81d1fd17-d835-463f-bd7d-280f05f87d8c-istio-data\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-w86t8\" (UID: \"81d1fd17-d835-463f-bd7d-280f05f87d8c\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-w86t8" Oct 29 23:03:26.644972 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:26.644929 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81d1fd17-d835-463f-bd7d-280f05f87d8c-workload-certs\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-w86t8\" (UID: \"81d1fd17-d835-463f-bd7d-280f05f87d8c\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-w86t8" Oct 29 23:03:26.645100 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:26.645075 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81d1fd17-d835-463f-bd7d-280f05f87d8c-workload-socket\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-w86t8\" (UID: \"81d1fd17-d835-463f-bd7d-280f05f87d8c\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-w86t8" Oct 29 23:03:26.645183 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:26.645155 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81d1fd17-d835-463f-bd7d-280f05f87d8c-credential-socket\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-w86t8\" (UID: \"81d1fd17-d835-463f-bd7d-280f05f87d8c\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-w86t8" Oct 29 23:03:26.645513 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:26.645497 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81d1fd17-d835-463f-bd7d-280f05f87d8c-istiod-ca-cert\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-w86t8\" (UID: \"81d1fd17-d835-463f-bd7d-280f05f87d8c\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-w86t8" Oct 29 23:03:26.646910 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:26.646891 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81d1fd17-d835-463f-bd7d-280f05f87d8c-istio-envoy\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-w86t8\" (UID: \"81d1fd17-d835-463f-bd7d-280f05f87d8c\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-w86t8" Oct 29 23:03:26.647483 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:26.647462 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81d1fd17-d835-463f-bd7d-280f05f87d8c-istio-podinfo\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-w86t8\" (UID: \"81d1fd17-d835-463f-bd7d-280f05f87d8c\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-w86t8" Oct 29 23:03:26.666215 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:26.666188 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81d1fd17-d835-463f-bd7d-280f05f87d8c-istio-token\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-w86t8\" (UID: \"81d1fd17-d835-463f-bd7d-280f05f87d8c\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-w86t8" Oct 29 23:03:26.666501 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:26.666483 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vctjb\" (UniqueName: \"kubernetes.io/projected/81d1fd17-d835-463f-bd7d-280f05f87d8c-kube-api-access-vctjb\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-w86t8\" (UID: \"81d1fd17-d835-463f-bd7d-280f05f87d8c\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-w86t8" Oct 29 23:03:26.775760 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:26.775729 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-w86t8" Oct 29 23:03:27.186067 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:27.182641 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-w86t8"] Oct 29 23:03:27.192667 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:27.192630 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 23:03:27.192790 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:27.192713 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 23:03:27.192790 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:27.192760 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 23:03:28.069393 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:28.069306 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:03:28.230172 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:28.230030 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-w86t8" event={"ID":"81d1fd17-d835-463f-bd7d-280f05f87d8c","Type":"ContainerStarted","Data":"1a1e1921c3ef5aa4f4a4cd3be6b860dd8779339fc6513c9acee47c6e7e04a41d"} Oct 29 23:03:28.230172 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:28.230092 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-w86t8" event={"ID":"81d1fd17-d835-463f-bd7d-280f05f87d8c","Type":"ContainerStarted","Data":"e817864ebaf2926c28e298a77ff4c930b5437219c558965c580ba0b15cc103e8"} Oct 29 23:03:28.312355 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:28.309972 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-w86t8" podStartSLOduration=2.309949125 podStartE2EDuration="2.309949125s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:03:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:03:28.296865732 +0000 UTC m=+1789.080079198" watchObservedRunningTime="2026-10-29 23:03:28.309949125 +0000 UTC m=+1789.093162551" Oct 29 23:03:28.440286 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:28.440196 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:03:28.634947 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:28.634908 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:03:28.776208 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:28.776177 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-w86t8" Oct 29 23:03:28.782507 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:28.782474 5057 patch_prober.go:29] interesting pod/e2e-praxis-a-0266dd-openshift-default-c89468fc-w86t8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body= Oct 29 23:03:28.787486 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:28.787446 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-w86t8" podUID="81d1fd17-d835-463f-bd7d-280f05f87d8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" Oct 29 23:03:29.780677 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:29.780639 5057 patch_prober.go:29] interesting pod/e2e-praxis-a-0266dd-openshift-default-c89468fc-w86t8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body= Oct 29 23:03:29.781254 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:29.781219 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-w86t8" podUID="81d1fd17-d835-463f-bd7d-280f05f87d8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" Oct 29 23:03:30.095910 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:30.095818 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:03:30.251948 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:30.251914 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:03:30.374913 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:30.374788 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:03:30.800721 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:30.800655 5057 patch_prober.go:29] interesting pod/e2e-praxis-a-0266dd-openshift-default-c89468fc-w86t8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body= Oct 29 23:03:30.804670 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:30.801295 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-w86t8" podUID="81d1fd17-d835-463f-bd7d-280f05f87d8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" Oct 29 23:03:31.780476 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:31.780437 5057 patch_prober.go:29] interesting pod/e2e-praxis-a-0266dd-openshift-default-c89468fc-w86t8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body= Oct 29 23:03:31.800677 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:31.780669 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-w86t8" podUID="81d1fd17-d835-463f-bd7d-280f05f87d8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" Oct 29 23:03:31.874543 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:31.874510 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:03:32.762014 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:32.761973 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:03:32.776029 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:32.775998 5057 patch_prober.go:29] interesting pod/e2e-praxis-a-0266dd-openshift-default-c89468fc-w86t8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body= Oct 29 23:03:32.776179 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:32.776087 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-w86t8" podUID="81d1fd17-d835-463f-bd7d-280f05f87d8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" Oct 29 23:03:32.866413 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:32.866375 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:03:33.023258 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:33.023137 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:03:33.426561 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:33.426463 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:03:33.853261 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:33.853160 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-w86t8" Oct 29 23:03:33.853760 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:33.853739 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-w86t8" Oct 29 23:03:33.855620 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:33.855173 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-w86t8" Oct 29 23:03:35.010051 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:35.010000 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:03:35.582290 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:35.582251 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:03:35.708879 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:35.708840 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:03:35.815234 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:35.815196 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:03:36.855598 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:36.855558 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:03:36.964210 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:36.963906 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:03:37.460048 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:37.460010 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:03:37.584955 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:37.584912 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:03:39.332884 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:39.332854 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:03:39.610543 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:39.610451 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-vj6md_aa6a190e-3283-458a-9dfd-2de7cb193fc5/console-operator/2.log" Oct 29 23:03:39.623496 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:39.623472 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-vj6md_aa6a190e-3283-458a-9dfd-2de7cb193fc5/console-operator/2.log" Oct 29 23:03:39.935239 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:39.935151 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:03:40.060724 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:40.060678 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:03:40.346152 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:40.346107 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:03:40.877261 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:40.877207 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:03:41.027065 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:41.027025 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:03:41.315918 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:41.315881 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:03:41.526285 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:41.511998 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-848fb8-6b99cc4c76-bqmmk"] Oct 29 23:03:41.526285 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:41.525812 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-848fb8-6b99cc4c76-bqmmk"] Oct 29 23:03:41.526285 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:41.525930 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-848fb8-6b99cc4c76-bqmmk" Oct 29 23:03:41.539919 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:41.539871 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-848fb8-dockercfg-xdz72\"" Oct 29 23:03:41.540569 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:41.540545 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-848fb8\"" Oct 29 23:03:41.588630 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:41.588423 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:03:41.676876 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:41.665999 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-848fb8-854cfcf4bc-tbrng"] Oct 29 23:03:41.676876 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:41.675169 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-848fb8-854cfcf4bc-tbrng"] Oct 29 23:03:41.676876 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:41.675294 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-848fb8-854cfcf4bc-tbrng" Oct 29 23:03:41.707075 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:41.707046 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ffb2f6b-50ec-4e72-b634-27cba87e3439-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-848fb8-6b99cc4c76-bqmmk\" (UID: \"3ffb2f6b-50ec-4e72-b634-27cba87e3439\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-848fb8-6b99cc4c76-bqmmk" Oct 29 23:03:41.707230 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:41.707154 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxg4h\" (UniqueName: \"kubernetes.io/projected/3ffb2f6b-50ec-4e72-b634-27cba87e3439-kube-api-access-mxg4h\") pod \"payload-processing-e2e-ipp-b-848fb8-6b99cc4c76-bqmmk\" (UID: \"3ffb2f6b-50ec-4e72-b634-27cba87e3439\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-848fb8-6b99cc4c76-bqmmk" Oct 29 23:03:41.713360 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:41.713112 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:03:41.809377 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:41.809322 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ffb2f6b-50ec-4e72-b634-27cba87e3439-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-848fb8-6b99cc4c76-bqmmk\" (UID: \"3ffb2f6b-50ec-4e72-b634-27cba87e3439\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-848fb8-6b99cc4c76-bqmmk" Oct 29 23:03:41.819351 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:41.810441 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ffb2f6b-50ec-4e72-b634-27cba87e3439-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-848fb8-6b99cc4c76-bqmmk\" (UID: \"3ffb2f6b-50ec-4e72-b634-27cba87e3439\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-848fb8-6b99cc4c76-bqmmk" Oct 29 23:03:41.819351 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:41.816822 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmmsf\" (UniqueName: \"kubernetes.io/projected/9bdef886-0a1a-491e-8ae8-0c5fefd94ee4-kube-api-access-hmmsf\") pod \"payload-pre-processing-e2e-ipp-b-848fb8-854cfcf4bc-tbrng\" (UID: \"9bdef886-0a1a-491e-8ae8-0c5fefd94ee4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-848fb8-854cfcf4bc-tbrng" Oct 29 23:03:41.819351 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:41.816870 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxg4h\" (UniqueName: \"kubernetes.io/projected/3ffb2f6b-50ec-4e72-b634-27cba87e3439-kube-api-access-mxg4h\") pod \"payload-processing-e2e-ipp-b-848fb8-6b99cc4c76-bqmmk\" (UID: \"3ffb2f6b-50ec-4e72-b634-27cba87e3439\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-848fb8-6b99cc4c76-bqmmk" Oct 29 23:03:41.819351 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:41.816913 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9bdef886-0a1a-491e-8ae8-0c5fefd94ee4-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-848fb8-854cfcf4bc-tbrng\" (UID: \"9bdef886-0a1a-491e-8ae8-0c5fefd94ee4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-848fb8-854cfcf4bc-tbrng" Oct 29 23:03:41.845482 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:41.845402 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:03:41.846503 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:41.846475 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxg4h\" (UniqueName: \"kubernetes.io/projected/3ffb2f6b-50ec-4e72-b634-27cba87e3439-kube-api-access-mxg4h\") pod \"payload-processing-e2e-ipp-b-848fb8-6b99cc4c76-bqmmk\" (UID: \"3ffb2f6b-50ec-4e72-b634-27cba87e3439\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-848fb8-6b99cc4c76-bqmmk" Oct 29 23:03:41.850241 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:41.850207 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-848fb8-6b99cc4c76-bqmmk" Oct 29 23:03:41.927818 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:41.924547 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmmsf\" (UniqueName: \"kubernetes.io/projected/9bdef886-0a1a-491e-8ae8-0c5fefd94ee4-kube-api-access-hmmsf\") pod \"payload-pre-processing-e2e-ipp-b-848fb8-854cfcf4bc-tbrng\" (UID: \"9bdef886-0a1a-491e-8ae8-0c5fefd94ee4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-848fb8-854cfcf4bc-tbrng" Oct 29 23:03:41.927818 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:41.924617 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9bdef886-0a1a-491e-8ae8-0c5fefd94ee4-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-848fb8-854cfcf4bc-tbrng\" (UID: \"9bdef886-0a1a-491e-8ae8-0c5fefd94ee4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-848fb8-854cfcf4bc-tbrng" Oct 29 23:03:41.927818 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:41.925652 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9bdef886-0a1a-491e-8ae8-0c5fefd94ee4-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-848fb8-854cfcf4bc-tbrng\" (UID: \"9bdef886-0a1a-491e-8ae8-0c5fefd94ee4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-848fb8-854cfcf4bc-tbrng" Oct 29 23:03:41.949812 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:41.949787 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmmsf\" (UniqueName: \"kubernetes.io/projected/9bdef886-0a1a-491e-8ae8-0c5fefd94ee4-kube-api-access-hmmsf\") pod \"payload-pre-processing-e2e-ipp-b-848fb8-854cfcf4bc-tbrng\" (UID: \"9bdef886-0a1a-491e-8ae8-0c5fefd94ee4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-848fb8-854cfcf4bc-tbrng" Oct 29 23:03:42.039402 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:42.035958 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-848fb8-854cfcf4bc-tbrng" Oct 29 23:03:42.229785 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:42.229753 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-848fb8-6b99cc4c76-bqmmk"] Oct 29 23:03:42.533375 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:42.532756 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-848fb8-6b99cc4c76-bqmmk" event={"ID":"3ffb2f6b-50ec-4e72-b634-27cba87e3439","Type":"ContainerStarted","Data":"0e46f00d35dfab0db2d234566fedeb024d63e172e8ffc2c992dce616b423eb20"} Oct 29 23:03:42.533375 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:42.532811 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-848fb8-6b99cc4c76-bqmmk" event={"ID":"3ffb2f6b-50ec-4e72-b634-27cba87e3439","Type":"ContainerStarted","Data":"6564aff3dadf9e4a01c5c4c7a47b95412054ed6adecc114bb9eca27f68820b80"} Oct 29 23:03:42.533375 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:42.532834 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-848fb8-6b99cc4c76-bqmmk" Oct 29 23:03:42.576074 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:42.576008 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-848fb8-6b99cc4c76-bqmmk" podStartSLOduration=1.5759903629999998 podStartE2EDuration="1.575990363s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:03:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:03:42.569089832 +0000 UTC m=+1803.352303259" watchObservedRunningTime="2026-10-29 23:03:42.575990363 +0000 UTC m=+1803.359203791" Oct 29 23:03:42.623223 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:42.618624 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:03:42.640289 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:42.640096 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-848fb8-854cfcf4bc-tbrng"] Oct 29 23:03:42.647585 ip-10-0-140-74 kubenswrapper[5057]: W1029 23:03:42.647562 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bdef886_0a1a_491e_8ae8_0c5fefd94ee4.slice/crio-fc280e678ec684f15bcca71a476f7070173b06ce5205ce3e9a1d152711743ac3 WatchSource:0}: Error finding container fc280e678ec684f15bcca71a476f7070173b06ce5205ce3e9a1d152711743ac3: Status 404 returned error can't find the container with id fc280e678ec684f15bcca71a476f7070173b06ce5205ce3e9a1d152711743ac3 Oct 29 23:03:42.738991 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:42.738951 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:03:43.078844 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:43.078808 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:03:43.186664 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:43.186604 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:03:43.543853 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:43.543810 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-848fb8-854cfcf4bc-tbrng" event={"ID":"9bdef886-0a1a-491e-8ae8-0c5fefd94ee4","Type":"ContainerStarted","Data":"505b71caa23c6a171a6bdd47d08db678667a5531b1670a1f80a4b521528f7283"} Oct 29 23:03:43.544350 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:43.543889 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-848fb8-854cfcf4bc-tbrng" event={"ID":"9bdef886-0a1a-491e-8ae8-0c5fefd94ee4","Type":"ContainerStarted","Data":"fc280e678ec684f15bcca71a476f7070173b06ce5205ce3e9a1d152711743ac3"} Oct 29 23:03:43.544350 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:43.544068 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-848fb8-854cfcf4bc-tbrng" Oct 29 23:03:43.573465 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:43.573433 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:03:43.591782 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:43.591730 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-848fb8-854cfcf4bc-tbrng" podStartSLOduration=2.591716468 podStartE2EDuration="2.591716468s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:03:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:03:43.581228792 +0000 UTC m=+1804.364442223" watchObservedRunningTime="2026-10-29 23:03:43.591716468 +0000 UTC m=+1804.374929894" Oct 29 23:03:43.688383 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:43.688321 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:03:45.596741 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:45.596706 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:03:46.781804 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:46.781774 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:03:46.992136 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:46.992102 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:03:47.810128 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:47.810086 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:03:49.030840 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:03:49.030800 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d2dnn" containerName="extractor" Oct 29 23:03:49.036136 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:49.036102 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:03:49.538029 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:49.537999 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:03:49.649588 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:49.649550 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:03:49.757523 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:49.757486 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:03:50.323580 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:50.323540 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:03:50.936405 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:50.923029 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:03:51.313703 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:51.313643 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:03:51.524873 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:51.524841 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:03:51.895277 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:51.895241 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:03:52.020397 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:52.016654 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:03:52.245294 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:52.245248 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:03:53.416539 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:53.416486 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:03:53.546469 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:53.546422 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-848fb8-6b99cc4c76-bqmmk" Oct 29 23:03:54.555245 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:54.555214 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-848fb8-854cfcf4bc-tbrng" Oct 29 23:03:54.662024 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:54.661984 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:03:54.773719 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:54.773685 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:03:54.969367 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:54.969316 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:03:55.079077 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:55.079026 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f173b1-7b9d5d9867-52n67"] Oct 29 23:03:55.079665 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:55.079281 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f173b1-7b9d5d9867-52n67" podUID="9e3693ec-7e28-479f-a285-9720616a7b53" containerName="payload-pre-processing" containerID="cri-o://1bc9fe8ec5dc9fc9589c7a1411837caf5b92288e8a4f765272c759ad961f253e" gracePeriod=30 Oct 29 23:03:55.150414 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:55.150352 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:03:55.275546 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:55.275258 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:03:55.575425 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:55.574933 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f173b1-7b9d5d9867-52n67" Oct 29 23:03:55.647370 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:55.647172 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:03:55.699358 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:55.688418 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e3693ec-7e28-479f-a285-9720616a7b53-kube-api-access-xzvl7\" (UniqueName: \"kubernetes.io/projected/9e3693ec-7e28-479f-a285-9720616a7b53-kube-api-access-xzvl7\") pod \"9e3693ec-7e28-479f-a285-9720616a7b53\" (UID: \"9e3693ec-7e28-479f-a285-9720616a7b53\") " Oct 29 23:03:55.699358 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:55.688489 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e3693ec-7e28-479f-a285-9720616a7b53-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e3693ec-7e28-479f-a285-9720616a7b53-plugins-config-volume\") pod \"9e3693ec-7e28-479f-a285-9720616a7b53\" (UID: \"9e3693ec-7e28-479f-a285-9720616a7b53\") " Oct 29 23:03:55.699358 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:55.690585 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e3693ec-7e28-479f-a285-9720616a7b53-plugins-config-volume" pod "9e3693ec-7e28-479f-a285-9720616a7b53" (UID: "9e3693ec-7e28-479f-a285-9720616a7b53"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:03:55.741144 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:55.741086 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e3693ec-7e28-479f-a285-9720616a7b53-kube-api-access-xzvl7" pod "9e3693ec-7e28-479f-a285-9720616a7b53" (UID: "9e3693ec-7e28-479f-a285-9720616a7b53"). InnerVolumeSpecName "kube-api-access-xzvl7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:03:55.759179 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:55.755243 5057 generic.go:356] "Generic (PLEG): container finished" podID="9e3693ec-7e28-479f-a285-9720616a7b53" containerID="1bc9fe8ec5dc9fc9589c7a1411837caf5b92288e8a4f765272c759ad961f253e" exitCode=0 Oct 29 23:03:55.759179 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:55.755312 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f173b1-7b9d5d9867-52n67" event={"ID":"9e3693ec-7e28-479f-a285-9720616a7b53","Type":"ContainerDied","Data":"1bc9fe8ec5dc9fc9589c7a1411837caf5b92288e8a4f765272c759ad961f253e"} Oct 29 23:03:55.759179 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:55.755442 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f173b1-7b9d5d9867-52n67" event={"ID":"9e3693ec-7e28-479f-a285-9720616a7b53","Type":"ContainerDied","Data":"494d845b09f1714b4f9d56a19f3acb9bdcecc2887fe74f1acfeffca8410bedd1"} Oct 29 23:03:55.759179 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:55.755481 5057 scope.go:122] "RemoveContainer" containerID="1bc9fe8ec5dc9fc9589c7a1411837caf5b92288e8a4f765272c759ad961f253e" Oct 29 23:03:55.759179 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:55.755647 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f173b1-7b9d5d9867-52n67" Oct 29 23:03:55.770923 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:55.770888 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:03:55.777845 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:55.777826 5057 scope.go:122] "RemoveContainer" containerID="1bc9fe8ec5dc9fc9589c7a1411837caf5b92288e8a4f765272c759ad961f253e" Oct 29 23:03:55.778271 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:03:55.778249 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1bc9fe8ec5dc9fc9589c7a1411837caf5b92288e8a4f765272c759ad961f253e\": container with ID starting with 1bc9fe8ec5dc9fc9589c7a1411837caf5b92288e8a4f765272c759ad961f253e not found: ID does not exist" containerID="1bc9fe8ec5dc9fc9589c7a1411837caf5b92288e8a4f765272c759ad961f253e" Oct 29 23:03:55.778494 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:55.778281 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1bc9fe8ec5dc9fc9589c7a1411837caf5b92288e8a4f765272c759ad961f253e"} err="failed to get container status \"1bc9fe8ec5dc9fc9589c7a1411837caf5b92288e8a4f765272c759ad961f253e\": rpc error: code = NotFound desc = could not find container \"1bc9fe8ec5dc9fc9589c7a1411837caf5b92288e8a4f765272c759ad961f253e\": container with ID starting with 1bc9fe8ec5dc9fc9589c7a1411837caf5b92288e8a4f765272c759ad961f253e not found: ID does not exist" Oct 29 23:03:55.792231 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:55.792199 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xzvl7\" (UniqueName: \"kubernetes.io/projected/9e3693ec-7e28-479f-a285-9720616a7b53-kube-api-access-xzvl7\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:03:55.792231 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:55.792232 5057 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e3693ec-7e28-479f-a285-9720616a7b53-plugins-config-volume\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:03:55.811451 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:55.811385 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f173b1-7b9d5d9867-52n67"] Oct 29 23:03:55.818760 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:55.818681 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f173b1-7b9d5d9867-52n67"] Oct 29 23:03:55.882116 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:55.882033 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:03:55.992565 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:55.992526 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:03:56.316997 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:56.316959 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:03:57.204299 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:57.204241 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:03:57.313867 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:57.313826 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:03:57.333324 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:57.333294 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e3693ec-7e28-479f-a285-9720616a7b53" path="/var/lib/kubelet/pods/9e3693ec-7e28-479f-a285-9720616a7b53/volumes" Oct 29 23:03:58.599011 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:03:58.598974 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:04:00.284216 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:00.284175 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:04:01.813732 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:01.813699 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:04:01.930180 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:01.930145 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:04:02.135404 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:02.135285 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:04:02.690813 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:02.690784 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:04:02.814801 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:02.813522 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:04:03.099468 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:03.099431 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:04:03.523540 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:03.523506 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:04:04.539131 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:04.539098 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:04:04.683796 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:04.683761 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:04:04.821671 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:04.821567 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:04:05.224180 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:05.224143 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:04:06.494826 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:06.494754 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:04:08.134074 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:08.134034 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:04:08.520356 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:08.517917 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v2tfm"] Oct 29 23:04:10.166878 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:10.166843 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:04:11.751452 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:11.751414 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:04:11.865663 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:11.865628 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:04:12.103487 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:12.103409 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-w86t8"] Oct 29 23:04:12.103684 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:12.103660 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-w86t8" podUID="81d1fd17-d835-463f-bd7d-280f05f87d8c" containerName="istio-proxy" containerID="cri-o://1a1e1921c3ef5aa4f4a4cd3be6b860dd8779339fc6513c9acee47c6e7e04a41d" gracePeriod=30 Oct 29 23:04:12.112135 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:12.112105 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-mvnxc"] Oct 29 23:04:12.112394 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:12.112320 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-mvnxc" podUID="70201148-712a-4488-9b2c-2d2fdcd7f5a4" containerName="istio-proxy" containerID="cri-o://3c485a202bd9e063c1dcd7ec8e5ea396107d2888ea4c3ba8d13e7cf6ae646cd8" gracePeriod=30 Oct 29 23:04:12.118935 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:12.118554 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-9dnxn"] Oct 29 23:04:12.118935 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:12.118767 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-9dnxn" podUID="5e1b34c3-955d-4d56-b7c6-64779cf3eae8" containerName="istio-proxy" containerID="cri-o://9a282b4eace1a756619b6b733a75d2512324f0a80f5c75f06ff02a438135fbdd" gracePeriod=30 Oct 29 23:04:12.159559 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:12.159524 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:04:14.067985 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:14.067944 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:04:14.190187 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:14.190148 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:04:14.303455 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:14.300674 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:04:16.008235 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:16.008181 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:04:17.532401 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:17.528958 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-w86t8" Oct 29 23:04:17.586090 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:17.586056 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-mvnxc" Oct 29 23:04:17.660793 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:17.660755 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70201148-712a-4488-9b2c-2d2fdcd7f5a4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70201148-712a-4488-9b2c-2d2fdcd7f5a4-istio-data\") pod \"70201148-712a-4488-9b2c-2d2fdcd7f5a4\" (UID: \"70201148-712a-4488-9b2c-2d2fdcd7f5a4\") " Oct 29 23:04:17.660967 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:17.660807 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/70201148-712a-4488-9b2c-2d2fdcd7f5a4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70201148-712a-4488-9b2c-2d2fdcd7f5a4-istiod-ca-cert\") pod \"70201148-712a-4488-9b2c-2d2fdcd7f5a4\" (UID: \"70201148-712a-4488-9b2c-2d2fdcd7f5a4\") " Oct 29 23:04:17.660967 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:17.660835 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81d1fd17-d835-463f-bd7d-280f05f87d8c-istio-token\" (UniqueName: \"kubernetes.io/projected/81d1fd17-d835-463f-bd7d-280f05f87d8c-istio-token\") pod \"81d1fd17-d835-463f-bd7d-280f05f87d8c\" (UID: \"81d1fd17-d835-463f-bd7d-280f05f87d8c\") " Oct 29 23:04:17.660967 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:17.660887 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81d1fd17-d835-463f-bd7d-280f05f87d8c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81d1fd17-d835-463f-bd7d-280f05f87d8c-istio-envoy\") pod \"81d1fd17-d835-463f-bd7d-280f05f87d8c\" (UID: \"81d1fd17-d835-463f-bd7d-280f05f87d8c\") " Oct 29 23:04:17.660967 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:17.660952 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70201148-712a-4488-9b2c-2d2fdcd7f5a4-kube-api-access-v44zw\" (UniqueName: \"kubernetes.io/projected/70201148-712a-4488-9b2c-2d2fdcd7f5a4-kube-api-access-v44zw\") pod \"70201148-712a-4488-9b2c-2d2fdcd7f5a4\" (UID: \"70201148-712a-4488-9b2c-2d2fdcd7f5a4\") " Oct 29 23:04:17.661197 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:17.660986 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/70201148-712a-4488-9b2c-2d2fdcd7f5a4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70201148-712a-4488-9b2c-2d2fdcd7f5a4-istio-podinfo\") pod \"70201148-712a-4488-9b2c-2d2fdcd7f5a4\" (UID: \"70201148-712a-4488-9b2c-2d2fdcd7f5a4\") " Oct 29 23:04:17.661197 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:17.661047 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70201148-712a-4488-9b2c-2d2fdcd7f5a4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70201148-712a-4488-9b2c-2d2fdcd7f5a4-credential-socket\") pod \"70201148-712a-4488-9b2c-2d2fdcd7f5a4\" (UID: \"70201148-712a-4488-9b2c-2d2fdcd7f5a4\") " Oct 29 23:04:17.661197 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:17.661135 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70201148-712a-4488-9b2c-2d2fdcd7f5a4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70201148-712a-4488-9b2c-2d2fdcd7f5a4-workload-socket\") pod \"70201148-712a-4488-9b2c-2d2fdcd7f5a4\" (UID: \"70201148-712a-4488-9b2c-2d2fdcd7f5a4\") " Oct 29 23:04:17.661376 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:17.661200 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81d1fd17-d835-463f-bd7d-280f05f87d8c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81d1fd17-d835-463f-bd7d-280f05f87d8c-workload-certs\") pod \"81d1fd17-d835-463f-bd7d-280f05f87d8c\" (UID: \"81d1fd17-d835-463f-bd7d-280f05f87d8c\") " Oct 29 23:04:17.661376 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:17.661270 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81d1fd17-d835-463f-bd7d-280f05f87d8c-kube-api-access-vctjb\" (UniqueName: \"kubernetes.io/projected/81d1fd17-d835-463f-bd7d-280f05f87d8c-kube-api-access-vctjb\") pod \"81d1fd17-d835-463f-bd7d-280f05f87d8c\" (UID: \"81d1fd17-d835-463f-bd7d-280f05f87d8c\") " Oct 29 23:04:17.661376 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:17.661346 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/81d1fd17-d835-463f-bd7d-280f05f87d8c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81d1fd17-d835-463f-bd7d-280f05f87d8c-istio-podinfo\") pod \"81d1fd17-d835-463f-bd7d-280f05f87d8c\" (UID: \"81d1fd17-d835-463f-bd7d-280f05f87d8c\") " Oct 29 23:04:17.661548 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:17.661382 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70201148-712a-4488-9b2c-2d2fdcd7f5a4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70201148-712a-4488-9b2c-2d2fdcd7f5a4-workload-certs\") pod \"70201148-712a-4488-9b2c-2d2fdcd7f5a4\" (UID: \"70201148-712a-4488-9b2c-2d2fdcd7f5a4\") " Oct 29 23:04:17.661548 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:17.661415 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81d1fd17-d835-463f-bd7d-280f05f87d8c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81d1fd17-d835-463f-bd7d-280f05f87d8c-workload-socket\") pod \"81d1fd17-d835-463f-bd7d-280f05f87d8c\" (UID: \"81d1fd17-d835-463f-bd7d-280f05f87d8c\") " Oct 29 23:04:17.661548 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:17.661465 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70201148-712a-4488-9b2c-2d2fdcd7f5a4-istio-token\" (UniqueName: \"kubernetes.io/projected/70201148-712a-4488-9b2c-2d2fdcd7f5a4-istio-token\") pod \"70201148-712a-4488-9b2c-2d2fdcd7f5a4\" (UID: \"70201148-712a-4488-9b2c-2d2fdcd7f5a4\") " Oct 29 23:04:17.661548 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:17.661541 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70201148-712a-4488-9b2c-2d2fdcd7f5a4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70201148-712a-4488-9b2c-2d2fdcd7f5a4-istio-envoy\") pod \"70201148-712a-4488-9b2c-2d2fdcd7f5a4\" (UID: \"70201148-712a-4488-9b2c-2d2fdcd7f5a4\") " Oct 29 23:04:17.661757 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:17.661600 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/81d1fd17-d835-463f-bd7d-280f05f87d8c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81d1fd17-d835-463f-bd7d-280f05f87d8c-istiod-ca-cert\") pod \"81d1fd17-d835-463f-bd7d-280f05f87d8c\" (UID: \"81d1fd17-d835-463f-bd7d-280f05f87d8c\") " Oct 29 23:04:17.661757 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:17.661698 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81d1fd17-d835-463f-bd7d-280f05f87d8c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81d1fd17-d835-463f-bd7d-280f05f87d8c-istio-data\") pod \"81d1fd17-d835-463f-bd7d-280f05f87d8c\" (UID: \"81d1fd17-d835-463f-bd7d-280f05f87d8c\") " Oct 29 23:04:17.661757 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:17.661729 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81d1fd17-d835-463f-bd7d-280f05f87d8c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81d1fd17-d835-463f-bd7d-280f05f87d8c-credential-socket\") pod \"81d1fd17-d835-463f-bd7d-280f05f87d8c\" (UID: \"81d1fd17-d835-463f-bd7d-280f05f87d8c\") " Oct 29 23:04:17.662782 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:17.662740 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81d1fd17-d835-463f-bd7d-280f05f87d8c-workload-certs" pod "81d1fd17-d835-463f-bd7d-280f05f87d8c" (UID: "81d1fd17-d835-463f-bd7d-280f05f87d8c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:04:17.662962 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:17.662940 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81d1fd17-d835-463f-bd7d-280f05f87d8c-credential-socket" pod "81d1fd17-d835-463f-bd7d-280f05f87d8c" (UID: "81d1fd17-d835-463f-bd7d-280f05f87d8c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:04:17.663037 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:17.662958 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70201148-712a-4488-9b2c-2d2fdcd7f5a4-istio-data" pod "70201148-712a-4488-9b2c-2d2fdcd7f5a4" (UID: "70201148-712a-4488-9b2c-2d2fdcd7f5a4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:04:17.663311 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:17.663288 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70201148-712a-4488-9b2c-2d2fdcd7f5a4-istiod-ca-cert" pod "70201148-712a-4488-9b2c-2d2fdcd7f5a4" (UID: "70201148-712a-4488-9b2c-2d2fdcd7f5a4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:04:17.665843 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:17.665817 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70201148-712a-4488-9b2c-2d2fdcd7f5a4-credential-socket" pod "70201148-712a-4488-9b2c-2d2fdcd7f5a4" (UID: "70201148-712a-4488-9b2c-2d2fdcd7f5a4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:04:17.666036 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:17.666012 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70201148-712a-4488-9b2c-2d2fdcd7f5a4-workload-socket" pod "70201148-712a-4488-9b2c-2d2fdcd7f5a4" (UID: "70201148-712a-4488-9b2c-2d2fdcd7f5a4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:04:17.667998 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:17.667968 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70201148-712a-4488-9b2c-2d2fdcd7f5a4-istio-token" pod "70201148-712a-4488-9b2c-2d2fdcd7f5a4" (UID: "70201148-712a-4488-9b2c-2d2fdcd7f5a4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:04:17.669068 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:17.669042 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81d1fd17-d835-463f-bd7d-280f05f87d8c-istiod-ca-cert" pod "81d1fd17-d835-463f-bd7d-280f05f87d8c" (UID: "81d1fd17-d835-463f-bd7d-280f05f87d8c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:04:17.671163 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:17.670028 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81d1fd17-d835-463f-bd7d-280f05f87d8c-istio-data" pod "81d1fd17-d835-463f-bd7d-280f05f87d8c" (UID: "81d1fd17-d835-463f-bd7d-280f05f87d8c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:04:17.671163 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:17.670361 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81d1fd17-d835-463f-bd7d-280f05f87d8c-workload-socket" pod "81d1fd17-d835-463f-bd7d-280f05f87d8c" (UID: "81d1fd17-d835-463f-bd7d-280f05f87d8c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:04:17.671163 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:17.671047 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70201148-712a-4488-9b2c-2d2fdcd7f5a4-workload-certs" pod "70201148-712a-4488-9b2c-2d2fdcd7f5a4" (UID: "70201148-712a-4488-9b2c-2d2fdcd7f5a4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:04:17.674911 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:17.674602 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-9dnxn" Oct 29 23:04:17.682957 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:17.682936 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81d1fd17-d835-463f-bd7d-280f05f87d8c-istio-token" pod "81d1fd17-d835-463f-bd7d-280f05f87d8c" (UID: "81d1fd17-d835-463f-bd7d-280f05f87d8c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:04:17.683061 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:17.682993 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81d1fd17-d835-463f-bd7d-280f05f87d8c-istio-envoy" pod "81d1fd17-d835-463f-bd7d-280f05f87d8c" (UID: "81d1fd17-d835-463f-bd7d-280f05f87d8c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:04:17.683125 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:17.683065 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70201148-712a-4488-9b2c-2d2fdcd7f5a4-istio-envoy" pod "70201148-712a-4488-9b2c-2d2fdcd7f5a4" (UID: "70201148-712a-4488-9b2c-2d2fdcd7f5a4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:04:17.683313 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:17.683282 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70201148-712a-4488-9b2c-2d2fdcd7f5a4-kube-api-access-v44zw" pod "70201148-712a-4488-9b2c-2d2fdcd7f5a4" (UID: "70201148-712a-4488-9b2c-2d2fdcd7f5a4"). InnerVolumeSpecName "kube-api-access-v44zw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:04:17.683419 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:17.683389 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/70201148-712a-4488-9b2c-2d2fdcd7f5a4-istio-podinfo" pod "70201148-712a-4488-9b2c-2d2fdcd7f5a4" (UID: "70201148-712a-4488-9b2c-2d2fdcd7f5a4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:04:17.683732 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:17.683713 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81d1fd17-d835-463f-bd7d-280f05f87d8c-kube-api-access-vctjb" pod "81d1fd17-d835-463f-bd7d-280f05f87d8c" (UID: "81d1fd17-d835-463f-bd7d-280f05f87d8c"). InnerVolumeSpecName "kube-api-access-vctjb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:04:17.684291 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:17.684273 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/81d1fd17-d835-463f-bd7d-280f05f87d8c-istio-podinfo" pod "81d1fd17-d835-463f-bd7d-280f05f87d8c" (UID: "81d1fd17-d835-463f-bd7d-280f05f87d8c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:04:17.764297 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:17.764201 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e1b34c3-955d-4d56-b7c6-64779cf3eae8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e1b34c3-955d-4d56-b7c6-64779cf3eae8-istio-data\") pod \"5e1b34c3-955d-4d56-b7c6-64779cf3eae8\" (UID: \"5e1b34c3-955d-4d56-b7c6-64779cf3eae8\") " Oct 29 23:04:17.764521 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:17.764308 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e1b34c3-955d-4d56-b7c6-64779cf3eae8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e1b34c3-955d-4d56-b7c6-64779cf3eae8-istio-envoy\") pod \"5e1b34c3-955d-4d56-b7c6-64779cf3eae8\" (UID: \"5e1b34c3-955d-4d56-b7c6-64779cf3eae8\") " Oct 29 23:04:17.764521 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:17.764416 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e1b34c3-955d-4d56-b7c6-64779cf3eae8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e1b34c3-955d-4d56-b7c6-64779cf3eae8-workload-socket\") pod \"5e1b34c3-955d-4d56-b7c6-64779cf3eae8\" (UID: \"5e1b34c3-955d-4d56-b7c6-64779cf3eae8\") " Oct 29 23:04:17.764521 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:17.764448 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e1b34c3-955d-4d56-b7c6-64779cf3eae8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e1b34c3-955d-4d56-b7c6-64779cf3eae8-istiod-ca-cert\") pod \"5e1b34c3-955d-4d56-b7c6-64779cf3eae8\" (UID: \"5e1b34c3-955d-4d56-b7c6-64779cf3eae8\") " Oct 29 23:04:17.764521 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:17.764490 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e1b34c3-955d-4d56-b7c6-64779cf3eae8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e1b34c3-955d-4d56-b7c6-64779cf3eae8-credential-socket\") pod \"5e1b34c3-955d-4d56-b7c6-64779cf3eae8\" (UID: \"5e1b34c3-955d-4d56-b7c6-64779cf3eae8\") " Oct 29 23:04:17.764521 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:17.764500 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e1b34c3-955d-4d56-b7c6-64779cf3eae8-istio-data" pod "5e1b34c3-955d-4d56-b7c6-64779cf3eae8" (UID: "5e1b34c3-955d-4d56-b7c6-64779cf3eae8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:04:17.764521 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:17.764524 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e1b34c3-955d-4d56-b7c6-64779cf3eae8-istio-token\" (UniqueName: \"kubernetes.io/projected/5e1b34c3-955d-4d56-b7c6-64779cf3eae8-istio-token\") pod \"5e1b34c3-955d-4d56-b7c6-64779cf3eae8\" (UID: \"5e1b34c3-955d-4d56-b7c6-64779cf3eae8\") " Oct 29 23:04:17.764872 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:17.764618 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5e1b34c3-955d-4d56-b7c6-64779cf3eae8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e1b34c3-955d-4d56-b7c6-64779cf3eae8-istio-podinfo\") pod \"5e1b34c3-955d-4d56-b7c6-64779cf3eae8\" (UID: \"5e1b34c3-955d-4d56-b7c6-64779cf3eae8\") " Oct 29 23:04:17.764872 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:17.764733 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e1b34c3-955d-4d56-b7c6-64779cf3eae8-workload-socket" pod "5e1b34c3-955d-4d56-b7c6-64779cf3eae8" (UID: "5e1b34c3-955d-4d56-b7c6-64779cf3eae8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:04:17.764872 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:17.764848 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e1b34c3-955d-4d56-b7c6-64779cf3eae8-kube-api-access-9zf96\" (UniqueName: \"kubernetes.io/projected/5e1b34c3-955d-4d56-b7c6-64779cf3eae8-kube-api-access-9zf96\") pod \"5e1b34c3-955d-4d56-b7c6-64779cf3eae8\" (UID: \"5e1b34c3-955d-4d56-b7c6-64779cf3eae8\") " Oct 29 23:04:17.765040 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:17.764892 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e1b34c3-955d-4d56-b7c6-64779cf3eae8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e1b34c3-955d-4d56-b7c6-64779cf3eae8-workload-certs\") pod \"5e1b34c3-955d-4d56-b7c6-64779cf3eae8\" (UID: \"5e1b34c3-955d-4d56-b7c6-64779cf3eae8\") " Oct 29 23:04:17.765040 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:17.764925 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e1b34c3-955d-4d56-b7c6-64779cf3eae8-istiod-ca-cert" pod "5e1b34c3-955d-4d56-b7c6-64779cf3eae8" (UID: "5e1b34c3-955d-4d56-b7c6-64779cf3eae8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:04:17.765040 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:17.764928 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e1b34c3-955d-4d56-b7c6-64779cf3eae8-credential-socket" pod "5e1b34c3-955d-4d56-b7c6-64779cf3eae8" (UID: "5e1b34c3-955d-4d56-b7c6-64779cf3eae8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:04:17.765242 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:17.765195 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e1b34c3-955d-4d56-b7c6-64779cf3eae8-workload-certs" pod "5e1b34c3-955d-4d56-b7c6-64779cf3eae8" (UID: "5e1b34c3-955d-4d56-b7c6-64779cf3eae8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:04:17.765857 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:17.765827 5057 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70201148-712a-4488-9b2c-2d2fdcd7f5a4-istio-envoy\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:04:17.765978 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:17.765860 5057 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e1b34c3-955d-4d56-b7c6-64779cf3eae8-istio-data\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:04:17.765978 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:17.765886 5057 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81d1fd17-d835-463f-bd7d-280f05f87d8c-istiod-ca-cert\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:04:17.765978 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:17.765899 5057 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81d1fd17-d835-463f-bd7d-280f05f87d8c-istio-data\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:04:17.765978 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:17.765912 5057 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81d1fd17-d835-463f-bd7d-280f05f87d8c-credential-socket\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:04:17.765978 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:17.765933 5057 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e1b34c3-955d-4d56-b7c6-64779cf3eae8-workload-socket\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:04:17.765978 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:17.765947 5057 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e1b34c3-955d-4d56-b7c6-64779cf3eae8-istiod-ca-cert\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:04:17.765978 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:17.765959 5057 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70201148-712a-4488-9b2c-2d2fdcd7f5a4-istio-data\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:04:17.765978 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:17.765971 5057 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70201148-712a-4488-9b2c-2d2fdcd7f5a4-istiod-ca-cert\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:04:17.766360 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:17.765983 5057 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81d1fd17-d835-463f-bd7d-280f05f87d8c-istio-token\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:04:17.766360 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:17.765995 5057 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e1b34c3-955d-4d56-b7c6-64779cf3eae8-credential-socket\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:04:17.766360 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:17.766008 5057 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81d1fd17-d835-463f-bd7d-280f05f87d8c-istio-envoy\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:04:17.766360 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:17.766023 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v44zw\" (UniqueName: \"kubernetes.io/projected/70201148-712a-4488-9b2c-2d2fdcd7f5a4-kube-api-access-v44zw\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:04:17.766360 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:17.766037 5057 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70201148-712a-4488-9b2c-2d2fdcd7f5a4-istio-podinfo\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:04:17.766360 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:17.766051 5057 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70201148-712a-4488-9b2c-2d2fdcd7f5a4-credential-socket\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:04:17.766360 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:17.766070 5057 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70201148-712a-4488-9b2c-2d2fdcd7f5a4-workload-socket\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:04:17.766360 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:17.766085 5057 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81d1fd17-d835-463f-bd7d-280f05f87d8c-workload-certs\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:04:17.766360 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:17.766099 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vctjb\" (UniqueName: \"kubernetes.io/projected/81d1fd17-d835-463f-bd7d-280f05f87d8c-kube-api-access-vctjb\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:04:17.766360 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:17.766113 5057 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81d1fd17-d835-463f-bd7d-280f05f87d8c-istio-podinfo\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:04:17.766360 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:17.766128 5057 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70201148-712a-4488-9b2c-2d2fdcd7f5a4-workload-certs\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:04:17.766360 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:17.766140 5057 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e1b34c3-955d-4d56-b7c6-64779cf3eae8-workload-certs\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:04:17.766360 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:17.766153 5057 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81d1fd17-d835-463f-bd7d-280f05f87d8c-workload-socket\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:04:17.766360 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:17.766165 5057 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70201148-712a-4488-9b2c-2d2fdcd7f5a4-istio-token\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:04:17.767088 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:17.767052 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e1b34c3-955d-4d56-b7c6-64779cf3eae8-istio-envoy" pod "5e1b34c3-955d-4d56-b7c6-64779cf3eae8" (UID: "5e1b34c3-955d-4d56-b7c6-64779cf3eae8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:04:17.767218 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:17.767192 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e1b34c3-955d-4d56-b7c6-64779cf3eae8-kube-api-access-9zf96" pod "5e1b34c3-955d-4d56-b7c6-64779cf3eae8" (UID: "5e1b34c3-955d-4d56-b7c6-64779cf3eae8"). InnerVolumeSpecName "kube-api-access-9zf96". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:04:17.767362 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:17.767271 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e1b34c3-955d-4d56-b7c6-64779cf3eae8-istio-token" pod "5e1b34c3-955d-4d56-b7c6-64779cf3eae8" (UID: "5e1b34c3-955d-4d56-b7c6-64779cf3eae8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:04:17.767362 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:17.767269 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5e1b34c3-955d-4d56-b7c6-64779cf3eae8-istio-podinfo" pod "5e1b34c3-955d-4d56-b7c6-64779cf3eae8" (UID: "5e1b34c3-955d-4d56-b7c6-64779cf3eae8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:04:17.867820 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:17.867791 5057 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e1b34c3-955d-4d56-b7c6-64779cf3eae8-istio-envoy\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:04:17.867820 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:17.867817 5057 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e1b34c3-955d-4d56-b7c6-64779cf3eae8-istio-token\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:04:17.867820 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:17.867826 5057 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e1b34c3-955d-4d56-b7c6-64779cf3eae8-istio-podinfo\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:04:17.868084 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:17.867836 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9zf96\" (UniqueName: \"kubernetes.io/projected/5e1b34c3-955d-4d56-b7c6-64779cf3eae8-kube-api-access-9zf96\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:04:18.005327 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:18.005286 5057 generic.go:356] "Generic (PLEG): container finished" podID="70201148-712a-4488-9b2c-2d2fdcd7f5a4" containerID="3c485a202bd9e063c1dcd7ec8e5ea396107d2888ea4c3ba8d13e7cf6ae646cd8" exitCode=0 Oct 29 23:04:18.005536 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:18.005370 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-mvnxc" event={"ID":"70201148-712a-4488-9b2c-2d2fdcd7f5a4","Type":"ContainerDied","Data":"3c485a202bd9e063c1dcd7ec8e5ea396107d2888ea4c3ba8d13e7cf6ae646cd8"} Oct 29 23:04:18.005536 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:18.005392 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-mvnxc" Oct 29 23:04:18.005536 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:18.005435 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-mvnxc" event={"ID":"70201148-712a-4488-9b2c-2d2fdcd7f5a4","Type":"ContainerDied","Data":"f67ab1076fff9e6ec9f86d285fe773c1411aca03e263c348994fd71378d8422a"} Oct 29 23:04:18.005536 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:18.005464 5057 scope.go:122] "RemoveContainer" containerID="3c485a202bd9e063c1dcd7ec8e5ea396107d2888ea4c3ba8d13e7cf6ae646cd8" Oct 29 23:04:18.007459 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:18.007435 5057 generic.go:356] "Generic (PLEG): container finished" podID="81d1fd17-d835-463f-bd7d-280f05f87d8c" containerID="1a1e1921c3ef5aa4f4a4cd3be6b860dd8779339fc6513c9acee47c6e7e04a41d" exitCode=0 Oct 29 23:04:18.007754 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:18.007494 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-w86t8" Oct 29 23:04:18.007754 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:18.007529 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-w86t8" event={"ID":"81d1fd17-d835-463f-bd7d-280f05f87d8c","Type":"ContainerDied","Data":"1a1e1921c3ef5aa4f4a4cd3be6b860dd8779339fc6513c9acee47c6e7e04a41d"} Oct 29 23:04:18.007754 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:18.007592 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-w86t8" event={"ID":"81d1fd17-d835-463f-bd7d-280f05f87d8c","Type":"ContainerDied","Data":"e817864ebaf2926c28e298a77ff4c930b5437219c558965c580ba0b15cc103e8"} Oct 29 23:04:18.009410 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:18.009385 5057 generic.go:356] "Generic (PLEG): container finished" podID="5e1b34c3-955d-4d56-b7c6-64779cf3eae8" containerID="9a282b4eace1a756619b6b733a75d2512324f0a80f5c75f06ff02a438135fbdd" exitCode=0 Oct 29 23:04:18.009630 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:18.009456 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-9dnxn" Oct 29 23:04:18.009630 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:18.009468 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-9dnxn" event={"ID":"5e1b34c3-955d-4d56-b7c6-64779cf3eae8","Type":"ContainerDied","Data":"9a282b4eace1a756619b6b733a75d2512324f0a80f5c75f06ff02a438135fbdd"} Oct 29 23:04:18.009630 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:18.009518 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-9dnxn" event={"ID":"5e1b34c3-955d-4d56-b7c6-64779cf3eae8","Type":"ContainerDied","Data":"a3f26a5e7028f36badf17fefbc78d866918e19020e57154f0671e370343d0821"} Oct 29 23:04:18.022465 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:18.022447 5057 scope.go:122] "RemoveContainer" containerID="3c485a202bd9e063c1dcd7ec8e5ea396107d2888ea4c3ba8d13e7cf6ae646cd8" Oct 29 23:04:18.022768 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:04:18.022742 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c485a202bd9e063c1dcd7ec8e5ea396107d2888ea4c3ba8d13e7cf6ae646cd8\": container with ID starting with 3c485a202bd9e063c1dcd7ec8e5ea396107d2888ea4c3ba8d13e7cf6ae646cd8 not found: ID does not exist" containerID="3c485a202bd9e063c1dcd7ec8e5ea396107d2888ea4c3ba8d13e7cf6ae646cd8" Oct 29 23:04:18.022909 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:18.022780 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c485a202bd9e063c1dcd7ec8e5ea396107d2888ea4c3ba8d13e7cf6ae646cd8"} err="failed to get container status \"3c485a202bd9e063c1dcd7ec8e5ea396107d2888ea4c3ba8d13e7cf6ae646cd8\": rpc error: code = NotFound desc = could not find container \"3c485a202bd9e063c1dcd7ec8e5ea396107d2888ea4c3ba8d13e7cf6ae646cd8\": container with ID starting with 3c485a202bd9e063c1dcd7ec8e5ea396107d2888ea4c3ba8d13e7cf6ae646cd8 not found: ID does not exist" Oct 29 23:04:18.022909 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:18.022802 5057 scope.go:122] "RemoveContainer" containerID="1a1e1921c3ef5aa4f4a4cd3be6b860dd8779339fc6513c9acee47c6e7e04a41d" Oct 29 23:04:18.042164 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:18.042138 5057 scope.go:122] "RemoveContainer" containerID="1a1e1921c3ef5aa4f4a4cd3be6b860dd8779339fc6513c9acee47c6e7e04a41d" Oct 29 23:04:18.042739 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:04:18.042705 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a1e1921c3ef5aa4f4a4cd3be6b860dd8779339fc6513c9acee47c6e7e04a41d\": container with ID starting with 1a1e1921c3ef5aa4f4a4cd3be6b860dd8779339fc6513c9acee47c6e7e04a41d not found: ID does not exist" containerID="1a1e1921c3ef5aa4f4a4cd3be6b860dd8779339fc6513c9acee47c6e7e04a41d" Oct 29 23:04:18.042920 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:18.042750 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a1e1921c3ef5aa4f4a4cd3be6b860dd8779339fc6513c9acee47c6e7e04a41d"} err="failed to get container status \"1a1e1921c3ef5aa4f4a4cd3be6b860dd8779339fc6513c9acee47c6e7e04a41d\": rpc error: code = NotFound desc = could not find container \"1a1e1921c3ef5aa4f4a4cd3be6b860dd8779339fc6513c9acee47c6e7e04a41d\": container with ID starting with 1a1e1921c3ef5aa4f4a4cd3be6b860dd8779339fc6513c9acee47c6e7e04a41d not found: ID does not exist" Oct 29 23:04:18.042920 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:18.042773 5057 scope.go:122] "RemoveContainer" containerID="9a282b4eace1a756619b6b733a75d2512324f0a80f5c75f06ff02a438135fbdd" Oct 29 23:04:18.053294 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:18.053273 5057 scope.go:122] "RemoveContainer" containerID="9a282b4eace1a756619b6b733a75d2512324f0a80f5c75f06ff02a438135fbdd" Oct 29 23:04:18.053627 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:04:18.053607 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a282b4eace1a756619b6b733a75d2512324f0a80f5c75f06ff02a438135fbdd\": container with ID starting with 9a282b4eace1a756619b6b733a75d2512324f0a80f5c75f06ff02a438135fbdd not found: ID does not exist" containerID="9a282b4eace1a756619b6b733a75d2512324f0a80f5c75f06ff02a438135fbdd" Oct 29 23:04:18.053702 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:18.053635 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a282b4eace1a756619b6b733a75d2512324f0a80f5c75f06ff02a438135fbdd"} err="failed to get container status \"9a282b4eace1a756619b6b733a75d2512324f0a80f5c75f06ff02a438135fbdd\": rpc error: code = NotFound desc = could not find container \"9a282b4eace1a756619b6b733a75d2512324f0a80f5c75f06ff02a438135fbdd\": container with ID starting with 9a282b4eace1a756619b6b733a75d2512324f0a80f5c75f06ff02a438135fbdd not found: ID does not exist" Oct 29 23:04:18.079646 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:18.079614 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-mvnxc"] Oct 29 23:04:18.087233 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:18.087190 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:04:18.090562 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:18.090535 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-mvnxc"] Oct 29 23:04:18.113864 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:18.113835 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-w86t8"] Oct 29 23:04:18.125232 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:18.125201 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-w86t8"] Oct 29 23:04:18.157963 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:18.157901 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-9dnxn"] Oct 29 23:04:18.164534 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:18.164508 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-9dnxn"] Oct 29 23:04:19.031535 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:04:19.031498 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d2dnn" containerName="extractor" Oct 29 23:04:19.328774 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:19.328689 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:04:19.331153 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:19.331125 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e1b34c3-955d-4d56-b7c6-64779cf3eae8" path="/var/lib/kubelet/pods/5e1b34c3-955d-4d56-b7c6-64779cf3eae8/volumes" Oct 29 23:04:19.331700 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:19.331682 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="70201148-712a-4488-9b2c-2d2fdcd7f5a4" path="/var/lib/kubelet/pods/70201148-712a-4488-9b2c-2d2fdcd7f5a4/volumes" Oct 29 23:04:19.332142 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:19.332129 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="81d1fd17-d835-463f-bd7d-280f05f87d8c" path="/var/lib/kubelet/pods/81d1fd17-d835-463f-bd7d-280f05f87d8c/volumes" Oct 29 23:04:19.522787 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:19.521195 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:04:19.723053 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:19.723012 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c88955-5c45cdbd7f-sdtmr"] Oct 29 23:04:19.723892 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:19.723874 5057 cpu_manager.go:414] "containerMap: removing container" podUID="81d1fd17-d835-463f-bd7d-280f05f87d8c" containerName="istio-proxy" Oct 29 23:04:19.723892 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:19.723892 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81d1fd17-d835-463f-bd7d-280f05f87d8c" containerName="istio-proxy" Oct 29 23:04:19.723980 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:19.723915 5057 cpu_manager.go:414] "containerMap: removing container" podUID="70201148-712a-4488-9b2c-2d2fdcd7f5a4" containerName="istio-proxy" Oct 29 23:04:19.723980 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:19.723921 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70201148-712a-4488-9b2c-2d2fdcd7f5a4" containerName="istio-proxy" Oct 29 23:04:19.723980 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:19.723938 5057 cpu_manager.go:414] "containerMap: removing container" podUID="9e3693ec-7e28-479f-a285-9720616a7b53" containerName="payload-pre-processing" Oct 29 23:04:19.723980 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:19.723943 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e3693ec-7e28-479f-a285-9720616a7b53" containerName="payload-pre-processing" Oct 29 23:04:19.723980 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:19.723979 5057 cpu_manager.go:414] "containerMap: removing container" podUID="5e1b34c3-955d-4d56-b7c6-64779cf3eae8" containerName="istio-proxy" Oct 29 23:04:19.724137 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:19.723985 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e1b34c3-955d-4d56-b7c6-64779cf3eae8" containerName="istio-proxy" Oct 29 23:04:19.724137 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:19.724106 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e3693ec-7e28-479f-a285-9720616a7b53" containerName="payload-pre-processing" Oct 29 23:04:19.724137 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:19.724122 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e1b34c3-955d-4d56-b7c6-64779cf3eae8" containerName="istio-proxy" Oct 29 23:04:19.724137 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:19.724131 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="81d1fd17-d835-463f-bd7d-280f05f87d8c" containerName="istio-proxy" Oct 29 23:04:19.724266 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:19.724141 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="70201148-712a-4488-9b2c-2d2fdcd7f5a4" containerName="istio-proxy" Oct 29 23:04:19.729688 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:19.729669 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c88955-5c45cdbd7f-sdtmr" Oct 29 23:04:19.747427 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:19.747401 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c88955-5c45cdbd7f-sdtmr"] Oct 29 23:04:19.752452 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:19.752423 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-c88955\"" Oct 29 23:04:19.752550 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:19.752456 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-c88955-dockercfg-lw6zp\"" Oct 29 23:04:19.829709 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:19.829678 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c9bed15-70b8-401f-820b-b793e47becd6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-c88955-5c45cdbd7f-sdtmr\" (UID: \"8c9bed15-70b8-401f-820b-b793e47becd6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c88955-5c45cdbd7f-sdtmr" Oct 29 23:04:19.829883 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:19.829855 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srzqh\" (UniqueName: \"kubernetes.io/projected/8c9bed15-70b8-401f-820b-b793e47becd6-kube-api-access-srzqh\") pod \"payload-pre-processing-e2e-ipp-b-c88955-5c45cdbd7f-sdtmr\" (UID: \"8c9bed15-70b8-401f-820b-b793e47becd6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c88955-5c45cdbd7f-sdtmr" Oct 29 23:04:19.945351 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:19.938048 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c9bed15-70b8-401f-820b-b793e47becd6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-c88955-5c45cdbd7f-sdtmr\" (UID: \"8c9bed15-70b8-401f-820b-b793e47becd6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c88955-5c45cdbd7f-sdtmr" Oct 29 23:04:19.945351 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:19.938185 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srzqh\" (UniqueName: \"kubernetes.io/projected/8c9bed15-70b8-401f-820b-b793e47becd6-kube-api-access-srzqh\") pod \"payload-pre-processing-e2e-ipp-b-c88955-5c45cdbd7f-sdtmr\" (UID: \"8c9bed15-70b8-401f-820b-b793e47becd6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c88955-5c45cdbd7f-sdtmr" Oct 29 23:04:19.945351 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:19.943839 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c9bed15-70b8-401f-820b-b793e47becd6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-c88955-5c45cdbd7f-sdtmr\" (UID: \"8c9bed15-70b8-401f-820b-b793e47becd6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c88955-5c45cdbd7f-sdtmr" Oct 29 23:04:20.043994 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:20.043913 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-srzqh\" (UniqueName: \"kubernetes.io/projected/8c9bed15-70b8-401f-820b-b793e47becd6-kube-api-access-srzqh\") pod \"payload-pre-processing-e2e-ipp-b-c88955-5c45cdbd7f-sdtmr\" (UID: \"8c9bed15-70b8-401f-820b-b793e47becd6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c88955-5c45cdbd7f-sdtmr" Oct 29 23:04:20.048029 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:20.046505 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c88955-5c45cdbd7f-sdtmr" Oct 29 23:04:20.442563 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:20.442478 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:04:20.460453 ip-10-0-140-74 kubenswrapper[5057]: W1029 23:04:20.460421 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c9bed15_70b8_401f_820b_b793e47becd6.slice/crio-95517aed5e508ad1a9649c872e8b0c01ae4e13687fece244d72268d345141d5f WatchSource:0}: Error finding container 95517aed5e508ad1a9649c872e8b0c01ae4e13687fece244d72268d345141d5f: Status 404 returned error can't find the container with id 95517aed5e508ad1a9649c872e8b0c01ae4e13687fece244d72268d345141d5f Oct 29 23:04:20.470785 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:20.470758 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c88955-5c45cdbd7f-sdtmr"] Oct 29 23:04:20.551207 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:20.551174 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:04:20.690898 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:20.690810 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:04:21.073527 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:21.073483 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c88955-5c45cdbd7f-sdtmr" event={"ID":"8c9bed15-70b8-401f-820b-b793e47becd6","Type":"ContainerStarted","Data":"f5de29f403607277e8f853ff5db1666d72e872f617b33eaeef650c38407ece88"} Oct 29 23:04:21.074004 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:21.073551 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c88955-5c45cdbd7f-sdtmr" Oct 29 23:04:21.074004 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:21.073574 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c88955-5c45cdbd7f-sdtmr" event={"ID":"8c9bed15-70b8-401f-820b-b793e47becd6","Type":"ContainerStarted","Data":"95517aed5e508ad1a9649c872e8b0c01ae4e13687fece244d72268d345141d5f"} Oct 29 23:04:21.114604 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:21.114543 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c88955-5c45cdbd7f-sdtmr" podStartSLOduration=2.114524311 podStartE2EDuration="2.114524311s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:04:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:04:21.110251654 +0000 UTC m=+1841.893465081" watchObservedRunningTime="2026-10-29 23:04:21.114524311 +0000 UTC m=+1841.897737738" Oct 29 23:04:22.304077 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:22.304041 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:04:23.308566 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:23.308525 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:04:23.417789 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:23.417751 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:04:24.249737 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:24.249704 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:04:25.885923 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:25.885887 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:04:27.465233 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:27.465194 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:04:27.590634 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:27.590596 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:04:29.444822 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:29.444777 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:04:31.100971 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:31.100940 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:04:32.086370 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:32.086324 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c88955-5c45cdbd7f-sdtmr" Oct 29 23:04:32.889764 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:32.889726 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:04:33.210433 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:33.210394 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:04:34.817959 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:34.817922 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:04:35.411170 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:35.411133 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:04:36.396140 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:36.396101 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:04:38.080646 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:38.080610 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:04:39.823409 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:39.823373 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:04:39.936482 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:39.936444 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:04:41.047006 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:41.046971 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:04:41.624979 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:41.624931 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:04:41.737757 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:41.737716 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:04:41.872156 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:41.872123 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:04:42.989490 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:42.989455 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:04:43.383815 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:43.383644 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:04:43.505806 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:43.505775 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-848fb8-6b99cc4c76-bqmmk"] Oct 29 23:04:43.506079 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:43.506051 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-848fb8-6b99cc4c76-bqmmk" podUID="3ffb2f6b-50ec-4e72-b634-27cba87e3439" containerName="payload-processing" containerID="cri-o://0e46f00d35dfab0db2d234566fedeb024d63e172e8ffc2c992dce616b423eb20" gracePeriod=30 Oct 29 23:04:43.527738 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:43.526486 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-848fb8-854cfcf4bc-tbrng"] Oct 29 23:04:43.527738 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:43.526743 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-848fb8-854cfcf4bc-tbrng" podUID="9bdef886-0a1a-491e-8ae8-0c5fefd94ee4" containerName="payload-pre-processing" containerID="cri-o://505b71caa23c6a171a6bdd47d08db678667a5531b1670a1f80a4b521528f7283" gracePeriod=30 Oct 29 23:04:43.561937 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:43.561892 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-ipp-b-848fb8-6b99cc4c76-bqmmk" podUID="3ffb2f6b-50ec-4e72-b634-27cba87e3439" containerName="payload-processing" probeResult="failure" output="dial tcp 10.132.0.144:9004: connect: connection refused" Oct 29 23:04:43.563021 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:43.562963 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:04:43.689678 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:43.689640 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:04:44.023492 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:44.023129 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-848fb8-6b99cc4c76-bqmmk" Oct 29 23:04:44.103573 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:44.103541 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-848fb8-854cfcf4bc-tbrng" Oct 29 23:04:44.122881 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:44.122836 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ffb2f6b-50ec-4e72-b634-27cba87e3439-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ffb2f6b-50ec-4e72-b634-27cba87e3439-plugins-config-volume\") pod \"3ffb2f6b-50ec-4e72-b634-27cba87e3439\" (UID: \"3ffb2f6b-50ec-4e72-b634-27cba87e3439\") " Oct 29 23:04:44.123402 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:44.123185 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ffb2f6b-50ec-4e72-b634-27cba87e3439-kube-api-access-mxg4h\" (UniqueName: \"kubernetes.io/projected/3ffb2f6b-50ec-4e72-b634-27cba87e3439-kube-api-access-mxg4h\") pod \"3ffb2f6b-50ec-4e72-b634-27cba87e3439\" (UID: \"3ffb2f6b-50ec-4e72-b634-27cba87e3439\") " Oct 29 23:04:44.123402 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:44.123237 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ffb2f6b-50ec-4e72-b634-27cba87e3439-plugins-config-volume" pod "3ffb2f6b-50ec-4e72-b634-27cba87e3439" (UID: "3ffb2f6b-50ec-4e72-b634-27cba87e3439"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:04:44.124097 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:44.124070 5057 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ffb2f6b-50ec-4e72-b634-27cba87e3439-plugins-config-volume\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:04:44.125851 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:44.125828 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ffb2f6b-50ec-4e72-b634-27cba87e3439-kube-api-access-mxg4h" pod "3ffb2f6b-50ec-4e72-b634-27cba87e3439" (UID: "3ffb2f6b-50ec-4e72-b634-27cba87e3439"). InnerVolumeSpecName "kube-api-access-mxg4h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:04:44.178976 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:44.178793 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:04:44.225341 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:44.225216 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9bdef886-0a1a-491e-8ae8-0c5fefd94ee4-kube-api-access-hmmsf\" (UniqueName: \"kubernetes.io/projected/9bdef886-0a1a-491e-8ae8-0c5fefd94ee4-kube-api-access-hmmsf\") pod \"9bdef886-0a1a-491e-8ae8-0c5fefd94ee4\" (UID: \"9bdef886-0a1a-491e-8ae8-0c5fefd94ee4\") " Oct 29 23:04:44.225529 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:44.225326 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9bdef886-0a1a-491e-8ae8-0c5fefd94ee4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9bdef886-0a1a-491e-8ae8-0c5fefd94ee4-plugins-config-volume\") pod \"9bdef886-0a1a-491e-8ae8-0c5fefd94ee4\" (UID: \"9bdef886-0a1a-491e-8ae8-0c5fefd94ee4\") " Oct 29 23:04:44.226110 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:44.226081 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mxg4h\" (UniqueName: \"kubernetes.io/projected/3ffb2f6b-50ec-4e72-b634-27cba87e3439-kube-api-access-mxg4h\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:04:44.226110 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:44.226081 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9bdef886-0a1a-491e-8ae8-0c5fefd94ee4-plugins-config-volume" pod "9bdef886-0a1a-491e-8ae8-0c5fefd94ee4" (UID: "9bdef886-0a1a-491e-8ae8-0c5fefd94ee4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:04:44.228958 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:44.228933 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bdef886-0a1a-491e-8ae8-0c5fefd94ee4-kube-api-access-hmmsf" pod "9bdef886-0a1a-491e-8ae8-0c5fefd94ee4" (UID: "9bdef886-0a1a-491e-8ae8-0c5fefd94ee4"). InnerVolumeSpecName "kube-api-access-hmmsf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:04:44.297460 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:44.297421 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:04:44.297973 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:44.297946 5057 generic.go:356] "Generic (PLEG): container finished" podID="3ffb2f6b-50ec-4e72-b634-27cba87e3439" containerID="0e46f00d35dfab0db2d234566fedeb024d63e172e8ffc2c992dce616b423eb20" exitCode=0 Oct 29 23:04:44.298062 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:44.297990 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-848fb8-6b99cc4c76-bqmmk" Oct 29 23:04:44.298062 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:44.298029 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-848fb8-6b99cc4c76-bqmmk" event={"ID":"3ffb2f6b-50ec-4e72-b634-27cba87e3439","Type":"ContainerDied","Data":"0e46f00d35dfab0db2d234566fedeb024d63e172e8ffc2c992dce616b423eb20"} Oct 29 23:04:44.298181 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:44.298081 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-848fb8-6b99cc4c76-bqmmk" event={"ID":"3ffb2f6b-50ec-4e72-b634-27cba87e3439","Type":"ContainerDied","Data":"6564aff3dadf9e4a01c5c4c7a47b95412054ed6adecc114bb9eca27f68820b80"} Oct 29 23:04:44.298181 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:44.298106 5057 scope.go:122] "RemoveContainer" containerID="0e46f00d35dfab0db2d234566fedeb024d63e172e8ffc2c992dce616b423eb20" Oct 29 23:04:44.299884 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:44.299858 5057 generic.go:356] "Generic (PLEG): container finished" podID="9bdef886-0a1a-491e-8ae8-0c5fefd94ee4" containerID="505b71caa23c6a171a6bdd47d08db678667a5531b1670a1f80a4b521528f7283" exitCode=0 Oct 29 23:04:44.300013 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:44.299895 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-848fb8-854cfcf4bc-tbrng" event={"ID":"9bdef886-0a1a-491e-8ae8-0c5fefd94ee4","Type":"ContainerDied","Data":"505b71caa23c6a171a6bdd47d08db678667a5531b1670a1f80a4b521528f7283"} Oct 29 23:04:44.300013 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:44.299903 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-848fb8-854cfcf4bc-tbrng" Oct 29 23:04:44.300013 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:44.299935 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-848fb8-854cfcf4bc-tbrng" event={"ID":"9bdef886-0a1a-491e-8ae8-0c5fefd94ee4","Type":"ContainerDied","Data":"fc280e678ec684f15bcca71a476f7070173b06ce5205ce3e9a1d152711743ac3"} Oct 29 23:04:44.311173 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:44.311149 5057 scope.go:122] "RemoveContainer" containerID="0e46f00d35dfab0db2d234566fedeb024d63e172e8ffc2c992dce616b423eb20" Oct 29 23:04:44.311603 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:04:44.311576 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e46f00d35dfab0db2d234566fedeb024d63e172e8ffc2c992dce616b423eb20\": container with ID starting with 0e46f00d35dfab0db2d234566fedeb024d63e172e8ffc2c992dce616b423eb20 not found: ID does not exist" containerID="0e46f00d35dfab0db2d234566fedeb024d63e172e8ffc2c992dce616b423eb20" Oct 29 23:04:44.311696 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:44.311612 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e46f00d35dfab0db2d234566fedeb024d63e172e8ffc2c992dce616b423eb20"} err="failed to get container status \"0e46f00d35dfab0db2d234566fedeb024d63e172e8ffc2c992dce616b423eb20\": rpc error: code = NotFound desc = could not find container \"0e46f00d35dfab0db2d234566fedeb024d63e172e8ffc2c992dce616b423eb20\": container with ID starting with 0e46f00d35dfab0db2d234566fedeb024d63e172e8ffc2c992dce616b423eb20 not found: ID does not exist" Oct 29 23:04:44.311696 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:44.311629 5057 scope.go:122] "RemoveContainer" containerID="505b71caa23c6a171a6bdd47d08db678667a5531b1670a1f80a4b521528f7283" Oct 29 23:04:44.324482 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:44.324461 5057 scope.go:122] "RemoveContainer" containerID="505b71caa23c6a171a6bdd47d08db678667a5531b1670a1f80a4b521528f7283" Oct 29 23:04:44.324832 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:04:44.324794 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"505b71caa23c6a171a6bdd47d08db678667a5531b1670a1f80a4b521528f7283\": container with ID starting with 505b71caa23c6a171a6bdd47d08db678667a5531b1670a1f80a4b521528f7283 not found: ID does not exist" containerID="505b71caa23c6a171a6bdd47d08db678667a5531b1670a1f80a4b521528f7283" Oct 29 23:04:44.324936 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:44.324843 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"505b71caa23c6a171a6bdd47d08db678667a5531b1670a1f80a4b521528f7283"} err="failed to get container status \"505b71caa23c6a171a6bdd47d08db678667a5531b1670a1f80a4b521528f7283\": rpc error: code = NotFound desc = could not find container \"505b71caa23c6a171a6bdd47d08db678667a5531b1670a1f80a4b521528f7283\": container with ID starting with 505b71caa23c6a171a6bdd47d08db678667a5531b1670a1f80a4b521528f7283 not found: ID does not exist" Oct 29 23:04:44.328850 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:44.328823 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hmmsf\" (UniqueName: \"kubernetes.io/projected/9bdef886-0a1a-491e-8ae8-0c5fefd94ee4-kube-api-access-hmmsf\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:04:44.328944 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:44.328880 5057 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9bdef886-0a1a-491e-8ae8-0c5fefd94ee4-plugins-config-volume\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:04:44.345997 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:44.345973 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-848fb8-6b99cc4c76-bqmmk"] Oct 29 23:04:44.353199 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:44.353177 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-848fb8-6b99cc4c76-bqmmk"] Oct 29 23:04:44.400349 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:44.399157 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-848fb8-854cfcf4bc-tbrng"] Oct 29 23:04:44.405686 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:44.405655 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-848fb8-854cfcf4bc-tbrng"] Oct 29 23:04:44.425815 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:44.425776 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:04:44.550384 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:44.547789 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:04:44.880861 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:44.880768 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:04:45.331171 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:45.331137 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ffb2f6b-50ec-4e72-b634-27cba87e3439" path="/var/lib/kubelet/pods/3ffb2f6b-50ec-4e72-b634-27cba87e3439/volumes" Oct 29 23:04:45.331760 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:45.331736 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9bdef886-0a1a-491e-8ae8-0c5fefd94ee4" path="/var/lib/kubelet/pods/9bdef886-0a1a-491e-8ae8-0c5fefd94ee4/volumes" Oct 29 23:04:45.953041 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:45.953002 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:04:46.074088 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:46.074056 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:04:46.271126 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:46.271040 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:04:47.898900 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:47.898860 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:04:48.336128 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:48.336095 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:04:48.456410 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:48.456369 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:04:48.507044 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:48.507012 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-4f2e3a-697864b755-jcjzh"] Oct 29 23:04:48.507920 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:48.507902 5057 cpu_manager.go:414] "containerMap: removing container" podUID="3ffb2f6b-50ec-4e72-b634-27cba87e3439" containerName="payload-processing" Oct 29 23:04:48.507982 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:48.507921 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ffb2f6b-50ec-4e72-b634-27cba87e3439" containerName="payload-processing" Oct 29 23:04:48.507982 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:48.507945 5057 cpu_manager.go:414] "containerMap: removing container" podUID="9bdef886-0a1a-491e-8ae8-0c5fefd94ee4" containerName="payload-pre-processing" Oct 29 23:04:48.507982 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:48.507950 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9bdef886-0a1a-491e-8ae8-0c5fefd94ee4" containerName="payload-pre-processing" Oct 29 23:04:48.508079 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:48.508061 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ffb2f6b-50ec-4e72-b634-27cba87e3439" containerName="payload-processing" Oct 29 23:04:48.508079 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:48.508073 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="9bdef886-0a1a-491e-8ae8-0c5fefd94ee4" containerName="payload-pre-processing" Oct 29 23:04:48.513899 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:48.513878 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-4f2e3a-697864b755-jcjzh" Oct 29 23:04:48.531724 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:48.531643 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-4f2e3a-697864b755-jcjzh"] Oct 29 23:04:48.544347 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:48.543526 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-4f2e3a-dockercfg-gfrvh\"" Oct 29 23:04:48.544347 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:48.543586 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-4f2e3a\"" Oct 29 23:04:48.598058 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:48.597503 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c11bc924-7135-4185-a52b-26baa3ae934e-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-4f2e3a-697864b755-jcjzh\" (UID: \"c11bc924-7135-4185-a52b-26baa3ae934e\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-4f2e3a-697864b755-jcjzh" Oct 29 23:04:48.598058 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:48.597782 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9djkf\" (UniqueName: \"kubernetes.io/projected/c11bc924-7135-4185-a52b-26baa3ae934e-kube-api-access-9djkf\") pod \"payload-pre-processing-e2e-praxis-b-4f2e3a-697864b755-jcjzh\" (UID: \"c11bc924-7135-4185-a52b-26baa3ae934e\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-4f2e3a-697864b755-jcjzh" Oct 29 23:04:48.806541 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:48.806494 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9djkf\" (UniqueName: \"kubernetes.io/projected/c11bc924-7135-4185-a52b-26baa3ae934e-kube-api-access-9djkf\") pod \"payload-pre-processing-e2e-praxis-b-4f2e3a-697864b755-jcjzh\" (UID: \"c11bc924-7135-4185-a52b-26baa3ae934e\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-4f2e3a-697864b755-jcjzh" Oct 29 23:04:48.812492 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:48.812447 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c11bc924-7135-4185-a52b-26baa3ae934e-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-4f2e3a-697864b755-jcjzh\" (UID: \"c11bc924-7135-4185-a52b-26baa3ae934e\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-4f2e3a-697864b755-jcjzh" Oct 29 23:04:48.815521 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:48.815117 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c11bc924-7135-4185-a52b-26baa3ae934e-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-4f2e3a-697864b755-jcjzh\" (UID: \"c11bc924-7135-4185-a52b-26baa3ae934e\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-4f2e3a-697864b755-jcjzh" Oct 29 23:04:48.845358 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:48.830204 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9djkf\" (UniqueName: \"kubernetes.io/projected/c11bc924-7135-4185-a52b-26baa3ae934e-kube-api-access-9djkf\") pod \"payload-pre-processing-e2e-praxis-b-4f2e3a-697864b755-jcjzh\" (UID: \"c11bc924-7135-4185-a52b-26baa3ae934e\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-4f2e3a-697864b755-jcjzh" Oct 29 23:04:48.852185 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:48.850713 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-4f2e3a-697864b755-jcjzh" Oct 29 23:04:49.035077 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:04:49.035041 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d2dnn" containerName="extractor" Oct 29 23:04:49.177390 ip-10-0-140-74 kubenswrapper[5057]: W1029 23:04:49.177318 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc11bc924_7135_4185_a52b_26baa3ae934e.slice/crio-114f32cf83e54c66d3c401b80ee5bcf1a70b494a0b46ae7881fa59f271dee53d WatchSource:0}: Error finding container 114f32cf83e54c66d3c401b80ee5bcf1a70b494a0b46ae7881fa59f271dee53d: Status 404 returned error can't find the container with id 114f32cf83e54c66d3c401b80ee5bcf1a70b494a0b46ae7881fa59f271dee53d Oct 29 23:04:49.177516 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:49.177444 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-4f2e3a-697864b755-jcjzh"] Oct 29 23:04:49.349267 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:49.349220 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:04:49.484698 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:49.484660 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-4f2e3a-697864b755-jcjzh" event={"ID":"c11bc924-7135-4185-a52b-26baa3ae934e","Type":"ContainerStarted","Data":"f8769c73066f18fd051e4f2b948ea960d3767c54b24f174f5fc6fd04ac85ce06"} Oct 29 23:04:49.485938 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:49.485906 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-4f2e3a-697864b755-jcjzh" event={"ID":"c11bc924-7135-4185-a52b-26baa3ae934e","Type":"ContainerStarted","Data":"114f32cf83e54c66d3c401b80ee5bcf1a70b494a0b46ae7881fa59f271dee53d"} Oct 29 23:04:49.486101 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:49.485953 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-4f2e3a-697864b755-jcjzh" Oct 29 23:04:49.556981 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:49.556927 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-4f2e3a-697864b755-jcjzh" podStartSLOduration=1.556912905 podStartE2EDuration="1.556912905s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:04:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:04:49.551495498 +0000 UTC m=+1870.334708924" watchObservedRunningTime="2026-10-29 23:04:49.556912905 +0000 UTC m=+1870.340126331" Oct 29 23:04:50.720150 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:50.720107 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:04:50.830204 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:50.830170 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:04:51.073375 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:51.073278 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:04:51.482052 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:51.482017 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:04:51.604851 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:51.604816 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:04:51.827686 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:51.827587 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:04:52.072364 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:52.065150 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:04:53.385471 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:53.385421 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:04:54.132349 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:54.131885 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:04:54.461140 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:54.461101 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:04:54.953383 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:54.953350 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:04:56.787930 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:56.787822 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:04:57.850973 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:57.850942 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-z2jqh"] Oct 29 23:04:57.857671 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:57.857647 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-z2jqh" Oct 29 23:04:57.878299 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:57.878275 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-z2jqh"] Oct 29 23:04:57.880631 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:57.880610 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-1b7950-openshift-default-dockercfg-pqvgz\"" Oct 29 23:04:58.025120 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:58.025056 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:04:58.041344 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:58.041300 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e2e4597-f643-42fb-a968-1dd7f39407fe-istio-envoy\") pod \"e2e-ipp-a-1b7950-openshift-default-6cbd77447f-z2jqh\" (UID: \"7e2e4597-f643-42fb-a968-1dd7f39407fe\") " pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-z2jqh" Oct 29 23:04:58.041514 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:58.041347 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jq6vk\" (UniqueName: \"kubernetes.io/projected/7e2e4597-f643-42fb-a968-1dd7f39407fe-kube-api-access-jq6vk\") pod \"e2e-ipp-a-1b7950-openshift-default-6cbd77447f-z2jqh\" (UID: \"7e2e4597-f643-42fb-a968-1dd7f39407fe\") " pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-z2jqh" Oct 29 23:04:58.041580 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:58.041501 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e2e4597-f643-42fb-a968-1dd7f39407fe-credential-socket\") pod \"e2e-ipp-a-1b7950-openshift-default-6cbd77447f-z2jqh\" (UID: \"7e2e4597-f643-42fb-a968-1dd7f39407fe\") " pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-z2jqh" Oct 29 23:04:58.041626 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:58.041599 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e2e4597-f643-42fb-a968-1dd7f39407fe-workload-socket\") pod \"e2e-ipp-a-1b7950-openshift-default-6cbd77447f-z2jqh\" (UID: \"7e2e4597-f643-42fb-a968-1dd7f39407fe\") " pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-z2jqh" Oct 29 23:04:58.041663 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:58.041625 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e2e4597-f643-42fb-a968-1dd7f39407fe-istio-token\") pod \"e2e-ipp-a-1b7950-openshift-default-6cbd77447f-z2jqh\" (UID: \"7e2e4597-f643-42fb-a968-1dd7f39407fe\") " pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-z2jqh" Oct 29 23:04:58.041663 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:58.041648 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e2e4597-f643-42fb-a968-1dd7f39407fe-workload-certs\") pod \"e2e-ipp-a-1b7950-openshift-default-6cbd77447f-z2jqh\" (UID: \"7e2e4597-f643-42fb-a968-1dd7f39407fe\") " pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-z2jqh" Oct 29 23:04:58.041756 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:58.041677 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e2e4597-f643-42fb-a968-1dd7f39407fe-istiod-ca-cert\") pod \"e2e-ipp-a-1b7950-openshift-default-6cbd77447f-z2jqh\" (UID: \"7e2e4597-f643-42fb-a968-1dd7f39407fe\") " pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-z2jqh" Oct 29 23:04:58.041841 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:58.041825 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e2e4597-f643-42fb-a968-1dd7f39407fe-istio-podinfo\") pod \"e2e-ipp-a-1b7950-openshift-default-6cbd77447f-z2jqh\" (UID: \"7e2e4597-f643-42fb-a968-1dd7f39407fe\") " pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-z2jqh" Oct 29 23:04:58.042099 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:58.042009 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e2e4597-f643-42fb-a968-1dd7f39407fe-istio-data\") pod \"e2e-ipp-a-1b7950-openshift-default-6cbd77447f-z2jqh\" (UID: \"7e2e4597-f643-42fb-a968-1dd7f39407fe\") " pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-z2jqh" Oct 29 23:04:58.143545 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:58.143466 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e2e4597-f643-42fb-a968-1dd7f39407fe-istio-envoy\") pod \"e2e-ipp-a-1b7950-openshift-default-6cbd77447f-z2jqh\" (UID: \"7e2e4597-f643-42fb-a968-1dd7f39407fe\") " pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-z2jqh" Oct 29 23:04:58.143545 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:58.143500 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jq6vk\" (UniqueName: \"kubernetes.io/projected/7e2e4597-f643-42fb-a968-1dd7f39407fe-kube-api-access-jq6vk\") pod \"e2e-ipp-a-1b7950-openshift-default-6cbd77447f-z2jqh\" (UID: \"7e2e4597-f643-42fb-a968-1dd7f39407fe\") " pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-z2jqh" Oct 29 23:04:58.143793 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:58.143547 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e2e4597-f643-42fb-a968-1dd7f39407fe-credential-socket\") pod \"e2e-ipp-a-1b7950-openshift-default-6cbd77447f-z2jqh\" (UID: \"7e2e4597-f643-42fb-a968-1dd7f39407fe\") " pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-z2jqh" Oct 29 23:04:58.143793 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:58.143607 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e2e4597-f643-42fb-a968-1dd7f39407fe-workload-socket\") pod \"e2e-ipp-a-1b7950-openshift-default-6cbd77447f-z2jqh\" (UID: \"7e2e4597-f643-42fb-a968-1dd7f39407fe\") " pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-z2jqh" Oct 29 23:04:58.143793 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:58.143644 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e2e4597-f643-42fb-a968-1dd7f39407fe-istio-token\") pod \"e2e-ipp-a-1b7950-openshift-default-6cbd77447f-z2jqh\" (UID: \"7e2e4597-f643-42fb-a968-1dd7f39407fe\") " pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-z2jqh" Oct 29 23:04:58.143793 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:58.143671 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e2e4597-f643-42fb-a968-1dd7f39407fe-workload-certs\") pod \"e2e-ipp-a-1b7950-openshift-default-6cbd77447f-z2jqh\" (UID: \"7e2e4597-f643-42fb-a968-1dd7f39407fe\") " pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-z2jqh" Oct 29 23:04:58.143793 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:58.143707 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e2e4597-f643-42fb-a968-1dd7f39407fe-istiod-ca-cert\") pod \"e2e-ipp-a-1b7950-openshift-default-6cbd77447f-z2jqh\" (UID: \"7e2e4597-f643-42fb-a968-1dd7f39407fe\") " pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-z2jqh" Oct 29 23:04:58.144152 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:58.143805 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e2e4597-f643-42fb-a968-1dd7f39407fe-istio-podinfo\") pod \"e2e-ipp-a-1b7950-openshift-default-6cbd77447f-z2jqh\" (UID: \"7e2e4597-f643-42fb-a968-1dd7f39407fe\") " pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-z2jqh" Oct 29 23:04:58.144152 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:58.144000 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e2e4597-f643-42fb-a968-1dd7f39407fe-credential-socket\") pod \"e2e-ipp-a-1b7950-openshift-default-6cbd77447f-z2jqh\" (UID: \"7e2e4597-f643-42fb-a968-1dd7f39407fe\") " pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-z2jqh" Oct 29 23:04:58.144152 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:58.144043 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e2e4597-f643-42fb-a968-1dd7f39407fe-workload-socket\") pod \"e2e-ipp-a-1b7950-openshift-default-6cbd77447f-z2jqh\" (UID: \"7e2e4597-f643-42fb-a968-1dd7f39407fe\") " pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-z2jqh" Oct 29 23:04:58.144152 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:58.144073 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e2e4597-f643-42fb-a968-1dd7f39407fe-workload-certs\") pod \"e2e-ipp-a-1b7950-openshift-default-6cbd77447f-z2jqh\" (UID: \"7e2e4597-f643-42fb-a968-1dd7f39407fe\") " pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-z2jqh" Oct 29 23:04:58.144152 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:58.144084 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e2e4597-f643-42fb-a968-1dd7f39407fe-istio-data\") pod \"e2e-ipp-a-1b7950-openshift-default-6cbd77447f-z2jqh\" (UID: \"7e2e4597-f643-42fb-a968-1dd7f39407fe\") " pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-z2jqh" Oct 29 23:04:58.144434 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:58.144392 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e2e4597-f643-42fb-a968-1dd7f39407fe-istio-data\") pod \"e2e-ipp-a-1b7950-openshift-default-6cbd77447f-z2jqh\" (UID: \"7e2e4597-f643-42fb-a968-1dd7f39407fe\") " pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-z2jqh" Oct 29 23:04:58.144585 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:58.144564 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e2e4597-f643-42fb-a968-1dd7f39407fe-istiod-ca-cert\") pod \"e2e-ipp-a-1b7950-openshift-default-6cbd77447f-z2jqh\" (UID: \"7e2e4597-f643-42fb-a968-1dd7f39407fe\") " pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-z2jqh" Oct 29 23:04:58.145983 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:58.145964 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e2e4597-f643-42fb-a968-1dd7f39407fe-istio-envoy\") pod \"e2e-ipp-a-1b7950-openshift-default-6cbd77447f-z2jqh\" (UID: \"7e2e4597-f643-42fb-a968-1dd7f39407fe\") " pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-z2jqh" Oct 29 23:04:58.146559 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:58.146534 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e2e4597-f643-42fb-a968-1dd7f39407fe-istio-podinfo\") pod \"e2e-ipp-a-1b7950-openshift-default-6cbd77447f-z2jqh\" (UID: \"7e2e4597-f643-42fb-a968-1dd7f39407fe\") " pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-z2jqh" Oct 29 23:04:58.165110 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:58.165078 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e2e4597-f643-42fb-a968-1dd7f39407fe-istio-token\") pod \"e2e-ipp-a-1b7950-openshift-default-6cbd77447f-z2jqh\" (UID: \"7e2e4597-f643-42fb-a968-1dd7f39407fe\") " pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-z2jqh" Oct 29 23:04:58.165234 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:58.165140 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jq6vk\" (UniqueName: \"kubernetes.io/projected/7e2e4597-f643-42fb-a968-1dd7f39407fe-kube-api-access-jq6vk\") pod \"e2e-ipp-a-1b7950-openshift-default-6cbd77447f-z2jqh\" (UID: \"7e2e4597-f643-42fb-a968-1dd7f39407fe\") " pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-z2jqh" Oct 29 23:04:58.177287 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:58.177094 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-z2jqh" Oct 29 23:04:58.530564 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:58.530524 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-z2jqh"] Oct 29 23:04:58.530788 ip-10-0-140-74 kubenswrapper[5057]: W1029 23:04:58.530748 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e2e4597_f643_42fb_a968_1dd7f39407fe.slice/crio-f20a73ace8a30370ad1fb811d7ac69189b680bfe2502ce3808cb85251be41ef4 WatchSource:0}: Error finding container f20a73ace8a30370ad1fb811d7ac69189b680bfe2502ce3808cb85251be41ef4: Status 404 returned error can't find the container with id f20a73ace8a30370ad1fb811d7ac69189b680bfe2502ce3808cb85251be41ef4 Oct 29 23:04:58.543594 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:58.538894 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 23:04:58.543594 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:58.538993 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 23:04:58.543594 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:58.539033 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 23:04:58.589504 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:58.588884 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-z2jqh" event={"ID":"7e2e4597-f643-42fb-a968-1dd7f39407fe","Type":"ContainerStarted","Data":"f20a73ace8a30370ad1fb811d7ac69189b680bfe2502ce3808cb85251be41ef4"} Oct 29 23:04:59.520568 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:59.520527 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:04:59.615395 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:59.615306 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-z2jqh" event={"ID":"7e2e4597-f643-42fb-a968-1dd7f39407fe","Type":"ContainerStarted","Data":"c6589eeaa1d8ca92893c11400a4be947aa88b506dc64ace9bf3c6b7aa841ddfb"} Oct 29 23:04:59.667671 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:59.667611 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-z2jqh" podStartSLOduration=2.667597019 podStartE2EDuration="2.667597019s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:04:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:04:59.656283715 +0000 UTC m=+1880.439497143" watchObservedRunningTime="2026-10-29 23:04:59.667597019 +0000 UTC m=+1880.450810445" Oct 29 23:04:59.851650 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:59.851561 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:04:59.975048 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:04:59.975004 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:05:00.177769 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:05:00.177686 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-z2jqh" Oct 29 23:05:00.186571 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:05:00.186468 5057 patch_prober.go:29] interesting pod/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-z2jqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body= Oct 29 23:05:00.194349 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:05:00.186828 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-z2jqh" podUID="7e2e4597-f643-42fb-a968-1dd7f39407fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" Oct 29 23:05:00.553359 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:05:00.550451 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-4f2e3a-697864b755-jcjzh" Oct 29 23:05:00.964175 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:05:00.964141 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:05:01.182795 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:05:01.182755 5057 patch_prober.go:29] interesting pod/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-z2jqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body= Oct 29 23:05:01.183144 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:05:01.183108 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-z2jqh" podUID="7e2e4597-f643-42fb-a968-1dd7f39407fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" Oct 29 23:05:02.196900 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:05:02.196866 5057 patch_prober.go:29] interesting pod/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-z2jqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body= Oct 29 23:05:02.197599 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:05:02.197565 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-z2jqh" podUID="7e2e4597-f643-42fb-a968-1dd7f39407fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" Oct 29 23:05:03.195351 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:05:03.192727 5057 patch_prober.go:29] interesting pod/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-z2jqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body= Oct 29 23:05:03.195351 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:05:03.192785 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-z2jqh" podUID="7e2e4597-f643-42fb-a968-1dd7f39407fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" Oct 29 23:05:03.529399 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:05:03.529354 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:05:04.181361 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:05:04.180693 5057 patch_prober.go:29] interesting pod/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-z2jqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body= Oct 29 23:05:04.181361 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:05:04.180768 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-z2jqh" podUID="7e2e4597-f643-42fb-a968-1dd7f39407fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" Oct 29 23:05:05.209556 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:05:05.209506 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-z2jqh" Oct 29 23:05:05.210752 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:05:05.210730 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-z2jqh" Oct 29 23:05:05.211591 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:05:05.211567 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-z2jqh" Oct 29 23:05:05.383134 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:05:05.383082 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:05:05.501308 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:05:05.501220 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:05:06.199523 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:05:06.199488 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:05:06.871746 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:05:06.871710 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:05:09.221458 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:05:09.221422 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:05:09.448453 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:05:09.448415 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:05:11.019315 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:05:11.019269 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:05:12.721052 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:05:12.721013 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:05:14.410641 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:05:14.410605 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:05:15.764573 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:05:15.764533 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:05:16.344353 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:05:16.334866 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:05:16.525076 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:05:16.525043 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:05:16.732936 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:05:16.732896 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:05:17.088505 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:05:17.088412 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:05:18.459850 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:05:18.459808 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:05:19.030177 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:05:19.030138 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d2dnn" containerName="extractor" Oct 29 23:05:19.981764 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:05:19.981712 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:05:20.108171 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:05:20.108127 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:05:21.736876 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:05:21.736842 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:05:23.330872 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:05:23.330833 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:05:23.441407 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:05:23.441370 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:05:23.723717 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:05:23.721408 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:05:25.849974 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:05:25.849934 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:05:27.121609 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:05:27.121571 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:05:27.311800 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:05:27.311759 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:05:27.541436 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:05:27.541401 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-1b7950-684586b68c-lr9zc"] Oct 29 23:05:27.549870 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:05:27.549839 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1b7950-684586b68c-lr9zc" Oct 29 23:05:27.572999 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:05:27.572115 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-1b7950-dockercfg-q4s6g\"" Oct 29 23:05:27.573210 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:05:27.572495 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-1b7950\"" Oct 29 23:05:27.610420 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:05:27.610385 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-1b7950-684586b68c-lr9zc"] Oct 29 23:05:27.639683 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:05:27.639639 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d30e7e8-5647-43a3-b15f-074537fa73ba-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-1b7950-684586b68c-lr9zc\" (UID: \"8d30e7e8-5647-43a3-b15f-074537fa73ba\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1b7950-684586b68c-lr9zc" Oct 29 23:05:27.640148 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:05:27.640123 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4b4h\" (UniqueName: \"kubernetes.io/projected/8d30e7e8-5647-43a3-b15f-074537fa73ba-kube-api-access-l4b4h\") pod \"payload-pre-processing-e2e-ipp-a-1b7950-684586b68c-lr9zc\" (UID: \"8d30e7e8-5647-43a3-b15f-074537fa73ba\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1b7950-684586b68c-lr9zc" Oct 29 23:05:27.754600 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:05:27.754550 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d30e7e8-5647-43a3-b15f-074537fa73ba-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-1b7950-684586b68c-lr9zc\" (UID: \"8d30e7e8-5647-43a3-b15f-074537fa73ba\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1b7950-684586b68c-lr9zc" Oct 29 23:05:27.755161 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:05:27.755125 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4b4h\" (UniqueName: \"kubernetes.io/projected/8d30e7e8-5647-43a3-b15f-074537fa73ba-kube-api-access-l4b4h\") pod \"payload-pre-processing-e2e-ipp-a-1b7950-684586b68c-lr9zc\" (UID: \"8d30e7e8-5647-43a3-b15f-074537fa73ba\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1b7950-684586b68c-lr9zc" Oct 29 23:05:27.757091 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:05:27.757066 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d30e7e8-5647-43a3-b15f-074537fa73ba-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-1b7950-684586b68c-lr9zc\" (UID: \"8d30e7e8-5647-43a3-b15f-074537fa73ba\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1b7950-684586b68c-lr9zc" Oct 29 23:05:27.799025 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:05:27.797682 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4b4h\" (UniqueName: \"kubernetes.io/projected/8d30e7e8-5647-43a3-b15f-074537fa73ba-kube-api-access-l4b4h\") pod \"payload-pre-processing-e2e-ipp-a-1b7950-684586b68c-lr9zc\" (UID: \"8d30e7e8-5647-43a3-b15f-074537fa73ba\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1b7950-684586b68c-lr9zc" Oct 29 23:05:27.894733 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:05:27.894683 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1b7950-684586b68c-lr9zc" Oct 29 23:05:28.233073 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:05:28.233042 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-1b7950-684586b68c-lr9zc"] Oct 29 23:05:28.233605 ip-10-0-140-74 kubenswrapper[5057]: W1029 23:05:28.233582 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d30e7e8_5647_43a3_b15f_074537fa73ba.slice/crio-7e2735cfa58425dc3bf0d909890e5be63f136dd5654677c6d8e14aeb002b27a4 WatchSource:0}: Error finding container 7e2735cfa58425dc3bf0d909890e5be63f136dd5654677c6d8e14aeb002b27a4: Status 404 returned error can't find the container with id 7e2735cfa58425dc3bf0d909890e5be63f136dd5654677c6d8e14aeb002b27a4 Oct 29 23:05:28.311240 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:05:28.308628 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:05:28.912284 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:05:28.912247 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1b7950-684586b68c-lr9zc" event={"ID":"8d30e7e8-5647-43a3-b15f-074537fa73ba","Type":"ContainerStarted","Data":"6141a134d935be8ad74da349070fe665637f358449e01974ca83ff4883b3e8a4"} Oct 29 23:05:28.912501 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:05:28.912303 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1b7950-684586b68c-lr9zc" event={"ID":"8d30e7e8-5647-43a3-b15f-074537fa73ba","Type":"ContainerStarted","Data":"7e2735cfa58425dc3bf0d909890e5be63f136dd5654677c6d8e14aeb002b27a4"} Oct 29 23:05:28.912501 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:05:28.912491 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1b7950-684586b68c-lr9zc" Oct 29 23:05:28.957431 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:05:28.957369 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1b7950-684586b68c-lr9zc" podStartSLOduration=1.957349635 podStartE2EDuration="1.957349635s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:05:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:05:28.947679919 +0000 UTC m=+1909.730893346" watchObservedRunningTime="2026-10-29 23:05:28.957349635 +0000 UTC m=+1909.740563062" Oct 29 23:05:29.206291 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:05:29.206257 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v2tfm"] Oct 29 23:05:30.008899 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:05:30.008862 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:05:30.121834 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:05:30.121799 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:05:32.072789 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:05:32.072712 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:05:33.035953 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:05:33.035909 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:05:33.718230 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:05:33.718195 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:05:34.816038 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:05:34.816003 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:05:34.926732 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:05:34.926693 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:05:35.451530 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:05:35.451494 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:05:37.158682 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:05:37.158640 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:05:37.169252 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:05:37.169225 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-7d18f7-595cb9f868-ps49n_ac11d963-38a4-41b4-ac9d-0eeadb3a2547/payload-processing/0.log" Oct 29 23:05:38.447320 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:05:38.447278 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:05:38.789260 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:05:38.789161 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:05:39.928051 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:05:39.928024 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1b7950-684586b68c-lr9zc" Oct 29 23:05:40.824184 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:05:40.824144 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:05:41.126178 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:05:41.126082 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:05:41.326227 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:05:41.326188 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:05:41.487350 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:05:41.481539 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:05:42.884433 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:05:42.884395 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:05:44.625298 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:05:44.625258 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:05:45.457260 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:05:45.457227 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:05:45.573912 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:05:45.573882 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:05:46.638474 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:05:46.638420 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:05:47.138992 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:05:47.138955 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:05:47.255642 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:05:47.255598 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:05:49.001590 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:05:49.001549 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:05:49.031462 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:05:49.031431 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d2dnn" containerName="extractor" Oct 29 23:05:50.866944 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:05:50.866734 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:05:51.610487 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:05:51.610321 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:05:51.860639 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:05:51.860548 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:05:51.999198 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:05:51.999160 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:05:52.409918 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:05:52.409876 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:05:54.506896 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:05:54.506856 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:05:56.330048 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:05:56.330007 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:05:57.549042 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:05:57.549007 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:05:58.543226 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:05:58.543184 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:05:58.655859 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:05:58.655821 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:05:58.963873 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:05:58.963833 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:06:00.813680 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:06:00.813632 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:06:01.725515 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:06:01.725477 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:06:02.611913 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:06:02.611877 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:06:04.327175 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:06:04.327133 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:06:06.125342 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:06:06.125288 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:06:07.809971 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:06:07.809937 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:06:09.579722 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:06:09.579687 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:06:09.695816 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:06:09.695765 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:06:10.932214 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:06:10.932178 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:06:11.416858 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:06:11.416824 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:06:11.540192 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:06:11.540156 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:06:11.659920 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:06:11.659880 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:06:11.775686 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:06:11.775652 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:06:13.834828 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:06:13.834788 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:06:15.815283 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:06:15.815253 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:06:15.936695 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:06:15.936654 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:06:16.051626 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:06:16.051589 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:06:17.560270 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:06:17.560217 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:06:17.767472 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:06:17.767435 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:06:17.944181 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:06:17.943930 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:06:18.880939 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:06:18.880904 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:06:19.030502 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:06:19.030466 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d2dnn" containerName="extractor" Oct 29 23:06:19.061892 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:06:19.061855 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:06:19.181164 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:06:19.181071 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:06:19.605367 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:06:19.605264 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:06:20.881485 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:06:20.881446 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:06:22.461871 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:06:22.461830 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:06:22.575952 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:06:22.575915 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:06:24.169532 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:06:24.169464 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:06:26.047202 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:06:26.047165 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:06:26.166622 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:06:26.166584 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:06:26.391294 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:06:26.391182 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:06:29.112073 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:06:29.112037 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:06:31.312764 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:06:31.312726 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:06:32.924294 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:06:32.924252 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:06:33.173654 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:06:33.173616 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:06:34.446747 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:06:34.446706 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:06:35.967232 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:06:35.967187 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:06:37.468213 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:06:37.468181 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-4f2e3a-697864b755-jcjzh"] Oct 29 23:06:37.522012 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:06:37.521971 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:06:37.632912 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:06:37.632875 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:06:39.332571 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:06:39.332535 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:06:40.648865 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:06:40.648823 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:06:40.738056 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:06:40.738023 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7d18f7-595cb9f868-ps49n"] Oct 29 23:06:40.738525 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:06:40.738499 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-7d18f7-595cb9f868-ps49n" podUID="ac11d963-38a4-41b4-ac9d-0eeadb3a2547" containerName="payload-processing" containerID="cri-o://bb19401173d4565d9c6d4fa5177cccb4113b56292cf2c58adbe864dd59537725" gracePeriod=30 Oct 29 23:06:40.757221 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:06:40.757195 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7d18f7-9fb86cbf9-wshhh"] Oct 29 23:06:40.757460 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:06:40.757430 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7d18f7-9fb86cbf9-wshhh" podUID="4a6b6875-8ebc-405a-960e-45cdb0073bcb" containerName="payload-pre-processing" containerID="cri-o://d95eeab1a2821302ac42571aa281ae35a94774696349a6b298e50eaf6185ec82" gracePeriod=30 Oct 29 23:06:40.826684 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:06:40.826286 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:06:40.942279 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:06:40.942175 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:06:41.287565 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:06:41.287528 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-7d18f7-595cb9f868-ps49n" Oct 29 23:06:41.382046 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:06:41.381438 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac11d963-38a4-41b4-ac9d-0eeadb3a2547-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac11d963-38a4-41b4-ac9d-0eeadb3a2547-plugins-config-volume\") pod \"ac11d963-38a4-41b4-ac9d-0eeadb3a2547\" (UID: \"ac11d963-38a4-41b4-ac9d-0eeadb3a2547\") " Oct 29 23:06:41.382046 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:06:41.381719 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac11d963-38a4-41b4-ac9d-0eeadb3a2547-kube-api-access-75272\" (UniqueName: \"kubernetes.io/projected/ac11d963-38a4-41b4-ac9d-0eeadb3a2547-kube-api-access-75272\") pod \"ac11d963-38a4-41b4-ac9d-0eeadb3a2547\" (UID: \"ac11d963-38a4-41b4-ac9d-0eeadb3a2547\") " Oct 29 23:06:41.382968 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:06:41.382701 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac11d963-38a4-41b4-ac9d-0eeadb3a2547-plugins-config-volume" pod "ac11d963-38a4-41b4-ac9d-0eeadb3a2547" (UID: "ac11d963-38a4-41b4-ac9d-0eeadb3a2547"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:06:41.382968 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:06:41.382938 5057 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac11d963-38a4-41b4-ac9d-0eeadb3a2547-plugins-config-volume\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:06:41.397405 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:06:41.393037 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac11d963-38a4-41b4-ac9d-0eeadb3a2547-kube-api-access-75272" pod "ac11d963-38a4-41b4-ac9d-0eeadb3a2547" (UID: "ac11d963-38a4-41b4-ac9d-0eeadb3a2547"). InnerVolumeSpecName "kube-api-access-75272". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:06:41.405366 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:06:41.400892 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7d18f7-9fb86cbf9-wshhh" Oct 29 23:06:41.484004 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:06:41.483965 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a6b6875-8ebc-405a-960e-45cdb0073bcb-kube-api-access-7r6ls\" (UniqueName: \"kubernetes.io/projected/4a6b6875-8ebc-405a-960e-45cdb0073bcb-kube-api-access-7r6ls\") pod \"4a6b6875-8ebc-405a-960e-45cdb0073bcb\" (UID: \"4a6b6875-8ebc-405a-960e-45cdb0073bcb\") " Oct 29 23:06:41.484208 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:06:41.484079 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a6b6875-8ebc-405a-960e-45cdb0073bcb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a6b6875-8ebc-405a-960e-45cdb0073bcb-plugins-config-volume\") pod \"4a6b6875-8ebc-405a-960e-45cdb0073bcb\" (UID: \"4a6b6875-8ebc-405a-960e-45cdb0073bcb\") " Oct 29 23:06:41.484761 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:06:41.484727 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a6b6875-8ebc-405a-960e-45cdb0073bcb-plugins-config-volume" pod "4a6b6875-8ebc-405a-960e-45cdb0073bcb" (UID: "4a6b6875-8ebc-405a-960e-45cdb0073bcb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:06:41.485398 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:06:41.485370 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-75272\" (UniqueName: \"kubernetes.io/projected/ac11d963-38a4-41b4-ac9d-0eeadb3a2547-kube-api-access-75272\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:06:41.485551 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:06:41.485400 5057 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a6b6875-8ebc-405a-960e-45cdb0073bcb-plugins-config-volume\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:06:41.490011 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:06:41.489981 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a6b6875-8ebc-405a-960e-45cdb0073bcb-kube-api-access-7r6ls" pod "4a6b6875-8ebc-405a-960e-45cdb0073bcb" (UID: "4a6b6875-8ebc-405a-960e-45cdb0073bcb"). InnerVolumeSpecName "kube-api-access-7r6ls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:06:41.510160 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:06:41.510018 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:06:41.587537 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:06:41.587454 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7r6ls\" (UniqueName: \"kubernetes.io/projected/4a6b6875-8ebc-405a-960e-45cdb0073bcb-kube-api-access-7r6ls\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:06:41.619971 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:06:41.619934 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:06:41.803203 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:06:41.803164 5057 generic.go:356] "Generic (PLEG): container finished" podID="4a6b6875-8ebc-405a-960e-45cdb0073bcb" containerID="d95eeab1a2821302ac42571aa281ae35a94774696349a6b298e50eaf6185ec82" exitCode=0 Oct 29 23:06:41.803696 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:06:41.803217 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7d18f7-9fb86cbf9-wshhh" Oct 29 23:06:41.803696 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:06:41.803248 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7d18f7-9fb86cbf9-wshhh" event={"ID":"4a6b6875-8ebc-405a-960e-45cdb0073bcb","Type":"ContainerDied","Data":"d95eeab1a2821302ac42571aa281ae35a94774696349a6b298e50eaf6185ec82"} Oct 29 23:06:41.803696 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:06:41.803310 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7d18f7-9fb86cbf9-wshhh" event={"ID":"4a6b6875-8ebc-405a-960e-45cdb0073bcb","Type":"ContainerDied","Data":"438f4f758fcb8e107c2f04212e827621e87bc602261ab356ffd89d95576155ff"} Oct 29 23:06:41.803696 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:06:41.803355 5057 scope.go:122] "RemoveContainer" containerID="d95eeab1a2821302ac42571aa281ae35a94774696349a6b298e50eaf6185ec82" Oct 29 23:06:41.805175 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:06:41.805153 5057 generic.go:356] "Generic (PLEG): container finished" podID="ac11d963-38a4-41b4-ac9d-0eeadb3a2547" containerID="bb19401173d4565d9c6d4fa5177cccb4113b56292cf2c58adbe864dd59537725" exitCode=0 Oct 29 23:06:41.805273 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:06:41.805206 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-7d18f7-595cb9f868-ps49n" Oct 29 23:06:41.805273 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:06:41.805240 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-7d18f7-595cb9f868-ps49n" event={"ID":"ac11d963-38a4-41b4-ac9d-0eeadb3a2547","Type":"ContainerDied","Data":"bb19401173d4565d9c6d4fa5177cccb4113b56292cf2c58adbe864dd59537725"} Oct 29 23:06:41.805412 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:06:41.805291 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-7d18f7-595cb9f868-ps49n" event={"ID":"ac11d963-38a4-41b4-ac9d-0eeadb3a2547","Type":"ContainerDied","Data":"f5c09d9cacb4d960dc288d8611653dc3a47e5260a08d8b16299a515ed06595b0"} Oct 29 23:06:41.818175 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:06:41.818156 5057 scope.go:122] "RemoveContainer" containerID="d95eeab1a2821302ac42571aa281ae35a94774696349a6b298e50eaf6185ec82" Oct 29 23:06:41.818716 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:06:41.818577 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d95eeab1a2821302ac42571aa281ae35a94774696349a6b298e50eaf6185ec82\": container with ID starting with d95eeab1a2821302ac42571aa281ae35a94774696349a6b298e50eaf6185ec82 not found: ID does not exist" containerID="d95eeab1a2821302ac42571aa281ae35a94774696349a6b298e50eaf6185ec82" Oct 29 23:06:41.818716 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:06:41.818617 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d95eeab1a2821302ac42571aa281ae35a94774696349a6b298e50eaf6185ec82"} err="failed to get container status \"d95eeab1a2821302ac42571aa281ae35a94774696349a6b298e50eaf6185ec82\": rpc error: code = NotFound desc = could not find container \"d95eeab1a2821302ac42571aa281ae35a94774696349a6b298e50eaf6185ec82\": container with ID starting with d95eeab1a2821302ac42571aa281ae35a94774696349a6b298e50eaf6185ec82 not found: ID does not exist" Oct 29 23:06:41.818716 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:06:41.818636 5057 scope.go:122] "RemoveContainer" containerID="bb19401173d4565d9c6d4fa5177cccb4113b56292cf2c58adbe864dd59537725" Oct 29 23:06:41.832154 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:06:41.832135 5057 scope.go:122] "RemoveContainer" containerID="bb19401173d4565d9c6d4fa5177cccb4113b56292cf2c58adbe864dd59537725" Oct 29 23:06:41.832435 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:06:41.832412 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb19401173d4565d9c6d4fa5177cccb4113b56292cf2c58adbe864dd59537725\": container with ID starting with bb19401173d4565d9c6d4fa5177cccb4113b56292cf2c58adbe864dd59537725 not found: ID does not exist" containerID="bb19401173d4565d9c6d4fa5177cccb4113b56292cf2c58adbe864dd59537725" Oct 29 23:06:41.832520 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:06:41.832441 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb19401173d4565d9c6d4fa5177cccb4113b56292cf2c58adbe864dd59537725"} err="failed to get container status \"bb19401173d4565d9c6d4fa5177cccb4113b56292cf2c58adbe864dd59537725\": rpc error: code = NotFound desc = could not find container \"bb19401173d4565d9c6d4fa5177cccb4113b56292cf2c58adbe864dd59537725\": container with ID starting with bb19401173d4565d9c6d4fa5177cccb4113b56292cf2c58adbe864dd59537725 not found: ID does not exist" Oct 29 23:06:41.863064 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:06:41.862994 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7d18f7-9fb86cbf9-wshhh"] Oct 29 23:06:41.867636 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:06:41.867605 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7d18f7-9fb86cbf9-wshhh"] Oct 29 23:06:41.899019 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:06:41.898993 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7d18f7-595cb9f868-ps49n"] Oct 29 23:06:41.902947 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:06:41.902927 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7d18f7-595cb9f868-ps49n"] Oct 29 23:06:42.139207 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:06:42.139119 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:06:42.469113 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:06:42.469075 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-4f2e3a-697864b755-jcjzh" podUID="c11bc924-7135-4185-a52b-26baa3ae934e" containerName="payload-pre-processing" containerID="cri-o://f8769c73066f18fd051e4f2b948ea960d3767c54b24f174f5fc6fd04ac85ce06" gracePeriod=25 Oct 29 23:06:42.798162 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:06:42.798109 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-4f2e3a-697864b755-jcjzh" Oct 29 23:06:42.820161 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:06:42.820130 5057 generic.go:356] "Generic (PLEG): container finished" podID="c11bc924-7135-4185-a52b-26baa3ae934e" containerID="f8769c73066f18fd051e4f2b948ea960d3767c54b24f174f5fc6fd04ac85ce06" exitCode=0 Oct 29 23:06:42.820720 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:06:42.820686 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-4f2e3a-697864b755-jcjzh" event={"ID":"c11bc924-7135-4185-a52b-26baa3ae934e","Type":"ContainerDied","Data":"f8769c73066f18fd051e4f2b948ea960d3767c54b24f174f5fc6fd04ac85ce06"} Oct 29 23:06:42.820865 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:06:42.820754 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-4f2e3a-697864b755-jcjzh" event={"ID":"c11bc924-7135-4185-a52b-26baa3ae934e","Type":"ContainerDied","Data":"114f32cf83e54c66d3c401b80ee5bcf1a70b494a0b46ae7881fa59f271dee53d"} Oct 29 23:06:42.820865 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:06:42.820785 5057 scope.go:122] "RemoveContainer" containerID="f8769c73066f18fd051e4f2b948ea960d3767c54b24f174f5fc6fd04ac85ce06" Oct 29 23:06:42.820998 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:06:42.820890 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-4f2e3a-697864b755-jcjzh" Oct 29 23:06:42.836827 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:06:42.836776 5057 scope.go:122] "RemoveContainer" containerID="f8769c73066f18fd051e4f2b948ea960d3767c54b24f174f5fc6fd04ac85ce06" Oct 29 23:06:42.837309 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:06:42.837277 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8769c73066f18fd051e4f2b948ea960d3767c54b24f174f5fc6fd04ac85ce06\": container with ID starting with f8769c73066f18fd051e4f2b948ea960d3767c54b24f174f5fc6fd04ac85ce06 not found: ID does not exist" containerID="f8769c73066f18fd051e4f2b948ea960d3767c54b24f174f5fc6fd04ac85ce06" Oct 29 23:06:42.837493 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:06:42.837462 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8769c73066f18fd051e4f2b948ea960d3767c54b24f174f5fc6fd04ac85ce06"} err="failed to get container status \"f8769c73066f18fd051e4f2b948ea960d3767c54b24f174f5fc6fd04ac85ce06\": rpc error: code = NotFound desc = could not find container \"f8769c73066f18fd051e4f2b948ea960d3767c54b24f174f5fc6fd04ac85ce06\": container with ID starting with f8769c73066f18fd051e4f2b948ea960d3767c54b24f174f5fc6fd04ac85ce06 not found: ID does not exist" Oct 29 23:06:42.908847 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:06:42.908818 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c11bc924-7135-4185-a52b-26baa3ae934e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c11bc924-7135-4185-a52b-26baa3ae934e-plugins-config-volume\") pod \"c11bc924-7135-4185-a52b-26baa3ae934e\" (UID: \"c11bc924-7135-4185-a52b-26baa3ae934e\") " Oct 29 23:06:42.909050 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:06:42.908985 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c11bc924-7135-4185-a52b-26baa3ae934e-kube-api-access-9djkf\" (UniqueName: \"kubernetes.io/projected/c11bc924-7135-4185-a52b-26baa3ae934e-kube-api-access-9djkf\") pod \"c11bc924-7135-4185-a52b-26baa3ae934e\" (UID: \"c11bc924-7135-4185-a52b-26baa3ae934e\") " Oct 29 23:06:42.909254 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:06:42.909224 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c11bc924-7135-4185-a52b-26baa3ae934e-plugins-config-volume" pod "c11bc924-7135-4185-a52b-26baa3ae934e" (UID: "c11bc924-7135-4185-a52b-26baa3ae934e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:06:42.909958 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:06:42.909933 5057 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c11bc924-7135-4185-a52b-26baa3ae934e-plugins-config-volume\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:06:42.911915 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:06:42.911891 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c11bc924-7135-4185-a52b-26baa3ae934e-kube-api-access-9djkf" pod "c11bc924-7135-4185-a52b-26baa3ae934e" (UID: "c11bc924-7135-4185-a52b-26baa3ae934e"). InnerVolumeSpecName "kube-api-access-9djkf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:06:43.011804 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:06:43.011741 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9djkf\" (UniqueName: \"kubernetes.io/projected/c11bc924-7135-4185-a52b-26baa3ae934e-kube-api-access-9djkf\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:06:43.163509 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:06:43.163370 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-4f2e3a-697864b755-jcjzh"] Oct 29 23:06:43.167847 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:06:43.167822 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-4f2e3a-697864b755-jcjzh"] Oct 29 23:06:43.331243 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:06:43.331165 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a6b6875-8ebc-405a-960e-45cdb0073bcb" path="/var/lib/kubelet/pods/4a6b6875-8ebc-405a-960e-45cdb0073bcb/volumes" Oct 29 23:06:43.331695 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:06:43.331680 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac11d963-38a4-41b4-ac9d-0eeadb3a2547" path="/var/lib/kubelet/pods/ac11d963-38a4-41b4-ac9d-0eeadb3a2547/volumes" Oct 29 23:06:43.332083 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:06:43.332071 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c11bc924-7135-4185-a52b-26baa3ae934e" path="/var/lib/kubelet/pods/c11bc924-7135-4185-a52b-26baa3ae934e/volumes" Oct 29 23:06:43.359373 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:06:43.359346 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:06:44.983267 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:06:44.983220 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:06:45.092846 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:06:45.092760 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:06:46.849897 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:06:46.849843 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:06:46.933007 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:06:46.932973 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v2tfm"] Oct 29 23:06:47.291442 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:06:47.291399 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:06:49.042355 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:06:49.041824 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:06:49.070949 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:06:49.070914 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d2dnn" containerName="extractor" Oct 29 23:06:51.043550 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:06:51.043517 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:06:52.092099 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:06:52.092064 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:06:52.290727 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:06:52.290688 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:06:52.642988 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:06:52.642758 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:06:53.691309 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:06:53.691273 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:06:53.804854 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:06:53.804815 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:06:54.163476 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:06:54.163427 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:06:54.277835 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:06:54.277801 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:06:54.742003 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:06:54.741972 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-vmj4r"] Oct 29 23:06:54.742571 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:06:54.742180 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-vmj4r" podUID="5954f1f5-c478-4f49-877e-9d3300568c2d" containerName="istio-proxy" containerID="cri-o://90020358042d221b88ea87fbb4cccb904f0836c0747ad5df003478138a7ec83a" gracePeriod=30 Oct 29 23:06:54.753596 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:06:54.753566 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-hfrrx"] Oct 29 23:06:54.753987 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:06:54.753953 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-hfrrx" podUID="43a9077f-2bb9-4801-abba-687de8a45076" containerName="istio-proxy" containerID="cri-o://cb4d8eb96cd3773d35f2f2bb1db5dc07f45ab4cfdec4e543c53e1253025cff5a" gracePeriod=30 Oct 29 23:06:54.829616 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:06:54.829577 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:06:56.028714 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:06:56.028666 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:06:56.191122 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:06:56.191071 5057 patch_prober.go:29] interesting pod/e2e-worker-w5-b2921b-openshift-default-5547c88fff-hfrrx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 23:06:56.191313 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:06:56.191157 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-hfrrx" podUID="43a9077f-2bb9-4801-abba-687de8a45076" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 23:06:56.611949 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:06:56.611590 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-k5t2b"] Oct 29 23:06:56.613049 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:06:56.612927 5057 cpu_manager.go:414] "containerMap: removing container" podUID="4a6b6875-8ebc-405a-960e-45cdb0073bcb" containerName="payload-pre-processing" Oct 29 23:06:56.613049 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:06:56.612953 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a6b6875-8ebc-405a-960e-45cdb0073bcb" containerName="payload-pre-processing" Oct 29 23:06:56.613049 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:06:56.612968 5057 cpu_manager.go:414] "containerMap: removing container" podUID="c11bc924-7135-4185-a52b-26baa3ae934e" containerName="payload-pre-processing" Oct 29 23:06:56.613049 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:06:56.612977 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c11bc924-7135-4185-a52b-26baa3ae934e" containerName="payload-pre-processing" Oct 29 23:06:56.613049 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:06:56.613009 5057 cpu_manager.go:414] "containerMap: removing container" podUID="ac11d963-38a4-41b4-ac9d-0eeadb3a2547" containerName="payload-processing" Oct 29 23:06:56.613049 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:06:56.613017 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac11d963-38a4-41b4-ac9d-0eeadb3a2547" containerName="payload-processing" Oct 29 23:06:56.613515 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:06:56.613200 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a6b6875-8ebc-405a-960e-45cdb0073bcb" containerName="payload-pre-processing" Oct 29 23:06:56.613515 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:06:56.613210 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="c11bc924-7135-4185-a52b-26baa3ae934e" containerName="payload-pre-processing" Oct 29 23:06:56.613515 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:06:56.613223 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac11d963-38a4-41b4-ac9d-0eeadb3a2547" containerName="payload-processing" Oct 29 23:06:56.635640 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:06:56.635590 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-k5t2b"] Oct 29 23:06:56.635640 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:06:56.635623 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-k5t2b" Oct 29 23:06:56.683661 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:06:56.683624 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-77238f-openshift-default-dockercfg-24287\"" Oct 29 23:06:56.752954 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:06:56.752923 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8dc3f31-0a4c-44e6-a3e9-4c45e238d1bc-istio-data\") pod \"e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-k5t2b\" (UID: \"f8dc3f31-0a4c-44e6-a3e9-4c45e238d1bc\") " pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-k5t2b" Oct 29 23:06:56.753129 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:06:56.752960 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8dc3f31-0a4c-44e6-a3e9-4c45e238d1bc-workload-certs\") pod \"e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-k5t2b\" (UID: \"f8dc3f31-0a4c-44e6-a3e9-4c45e238d1bc\") " pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-k5t2b" Oct 29 23:06:56.753129 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:06:56.752994 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8dc3f31-0a4c-44e6-a3e9-4c45e238d1bc-istio-token\") pod \"e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-k5t2b\" (UID: \"f8dc3f31-0a4c-44e6-a3e9-4c45e238d1bc\") " pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-k5t2b" Oct 29 23:06:56.753129 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:06:56.753104 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xmfv\" (UniqueName: \"kubernetes.io/projected/f8dc3f31-0a4c-44e6-a3e9-4c45e238d1bc-kube-api-access-9xmfv\") pod \"e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-k5t2b\" (UID: \"f8dc3f31-0a4c-44e6-a3e9-4c45e238d1bc\") " pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-k5t2b" Oct 29 23:06:56.753241 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:06:56.753143 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8dc3f31-0a4c-44e6-a3e9-4c45e238d1bc-istio-podinfo\") pod \"e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-k5t2b\" (UID: \"f8dc3f31-0a4c-44e6-a3e9-4c45e238d1bc\") " pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-k5t2b" Oct 29 23:06:56.753280 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:06:56.753249 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8dc3f31-0a4c-44e6-a3e9-4c45e238d1bc-workload-socket\") pod \"e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-k5t2b\" (UID: \"f8dc3f31-0a4c-44e6-a3e9-4c45e238d1bc\") " pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-k5t2b" Oct 29 23:06:56.753312 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:06:56.753289 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8dc3f31-0a4c-44e6-a3e9-4c45e238d1bc-istio-envoy\") pod \"e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-k5t2b\" (UID: \"f8dc3f31-0a4c-44e6-a3e9-4c45e238d1bc\") " pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-k5t2b" Oct 29 23:06:56.753401 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:06:56.753383 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8dc3f31-0a4c-44e6-a3e9-4c45e238d1bc-credential-socket\") pod \"e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-k5t2b\" (UID: \"f8dc3f31-0a4c-44e6-a3e9-4c45e238d1bc\") " pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-k5t2b" Oct 29 23:06:56.753442 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:06:56.753434 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8dc3f31-0a4c-44e6-a3e9-4c45e238d1bc-istiod-ca-cert\") pod \"e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-k5t2b\" (UID: \"f8dc3f31-0a4c-44e6-a3e9-4c45e238d1bc\") " pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-k5t2b" Oct 29 23:06:56.855132 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:06:56.855098 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8dc3f31-0a4c-44e6-a3e9-4c45e238d1bc-istio-data\") pod \"e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-k5t2b\" (UID: \"f8dc3f31-0a4c-44e6-a3e9-4c45e238d1bc\") " pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-k5t2b" Oct 29 23:06:56.855327 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:06:56.855143 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8dc3f31-0a4c-44e6-a3e9-4c45e238d1bc-workload-certs\") pod \"e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-k5t2b\" (UID: \"f8dc3f31-0a4c-44e6-a3e9-4c45e238d1bc\") " pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-k5t2b" Oct 29 23:06:56.855327 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:06:56.855188 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8dc3f31-0a4c-44e6-a3e9-4c45e238d1bc-istio-token\") pod \"e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-k5t2b\" (UID: \"f8dc3f31-0a4c-44e6-a3e9-4c45e238d1bc\") " pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-k5t2b" Oct 29 23:06:56.855327 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:06:56.855231 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xmfv\" (UniqueName: \"kubernetes.io/projected/f8dc3f31-0a4c-44e6-a3e9-4c45e238d1bc-kube-api-access-9xmfv\") pod \"e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-k5t2b\" (UID: \"f8dc3f31-0a4c-44e6-a3e9-4c45e238d1bc\") " pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-k5t2b" Oct 29 23:06:56.855327 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:06:56.855266 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8dc3f31-0a4c-44e6-a3e9-4c45e238d1bc-istio-podinfo\") pod \"e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-k5t2b\" (UID: \"f8dc3f31-0a4c-44e6-a3e9-4c45e238d1bc\") " pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-k5t2b" Oct 29 23:06:56.855704 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:06:56.855427 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8dc3f31-0a4c-44e6-a3e9-4c45e238d1bc-workload-socket\") pod \"e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-k5t2b\" (UID: \"f8dc3f31-0a4c-44e6-a3e9-4c45e238d1bc\") " pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-k5t2b" Oct 29 23:06:56.855704 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:06:56.855493 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8dc3f31-0a4c-44e6-a3e9-4c45e238d1bc-istio-envoy\") pod \"e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-k5t2b\" (UID: \"f8dc3f31-0a4c-44e6-a3e9-4c45e238d1bc\") " pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-k5t2b" Oct 29 23:06:56.855704 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:06:56.855526 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8dc3f31-0a4c-44e6-a3e9-4c45e238d1bc-istio-data\") pod \"e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-k5t2b\" (UID: \"f8dc3f31-0a4c-44e6-a3e9-4c45e238d1bc\") " pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-k5t2b" Oct 29 23:06:56.855704 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:06:56.855577 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8dc3f31-0a4c-44e6-a3e9-4c45e238d1bc-credential-socket\") pod \"e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-k5t2b\" (UID: \"f8dc3f31-0a4c-44e6-a3e9-4c45e238d1bc\") " pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-k5t2b" Oct 29 23:06:56.855704 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:06:56.855645 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8dc3f31-0a4c-44e6-a3e9-4c45e238d1bc-istiod-ca-cert\") pod \"e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-k5t2b\" (UID: \"f8dc3f31-0a4c-44e6-a3e9-4c45e238d1bc\") " pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-k5t2b" Oct 29 23:06:56.855985 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:06:56.855775 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8dc3f31-0a4c-44e6-a3e9-4c45e238d1bc-workload-socket\") pod \"e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-k5t2b\" (UID: \"f8dc3f31-0a4c-44e6-a3e9-4c45e238d1bc\") " pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-k5t2b" Oct 29 23:06:56.855985 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:06:56.855571 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8dc3f31-0a4c-44e6-a3e9-4c45e238d1bc-workload-certs\") pod \"e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-k5t2b\" (UID: \"f8dc3f31-0a4c-44e6-a3e9-4c45e238d1bc\") " pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-k5t2b" Oct 29 23:06:56.855985 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:06:56.855961 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8dc3f31-0a4c-44e6-a3e9-4c45e238d1bc-credential-socket\") pod \"e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-k5t2b\" (UID: \"f8dc3f31-0a4c-44e6-a3e9-4c45e238d1bc\") " pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-k5t2b" Oct 29 23:06:56.856309 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:06:56.856289 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8dc3f31-0a4c-44e6-a3e9-4c45e238d1bc-istiod-ca-cert\") pod \"e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-k5t2b\" (UID: \"f8dc3f31-0a4c-44e6-a3e9-4c45e238d1bc\") " pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-k5t2b" Oct 29 23:06:56.857988 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:06:56.857963 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8dc3f31-0a4c-44e6-a3e9-4c45e238d1bc-istio-envoy\") pod \"e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-k5t2b\" (UID: \"f8dc3f31-0a4c-44e6-a3e9-4c45e238d1bc\") " pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-k5t2b" Oct 29 23:06:56.858256 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:06:56.858234 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8dc3f31-0a4c-44e6-a3e9-4c45e238d1bc-istio-podinfo\") pod \"e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-k5t2b\" (UID: \"f8dc3f31-0a4c-44e6-a3e9-4c45e238d1bc\") " pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-k5t2b" Oct 29 23:06:56.883907 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:06:56.883849 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8dc3f31-0a4c-44e6-a3e9-4c45e238d1bc-istio-token\") pod \"e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-k5t2b\" (UID: \"f8dc3f31-0a4c-44e6-a3e9-4c45e238d1bc\") " pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-k5t2b" Oct 29 23:06:56.885564 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:06:56.885543 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xmfv\" (UniqueName: \"kubernetes.io/projected/f8dc3f31-0a4c-44e6-a3e9-4c45e238d1bc-kube-api-access-9xmfv\") pod \"e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-k5t2b\" (UID: \"f8dc3f31-0a4c-44e6-a3e9-4c45e238d1bc\") " pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-k5t2b" Oct 29 23:06:56.952570 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:06:56.952538 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-k5t2b" Oct 29 23:06:57.280935 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:06:57.280896 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-k5t2b"] Oct 29 23:06:57.282417 ip-10-0-140-74 kubenswrapper[5057]: W1029 23:06:57.282380 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8dc3f31_0a4c_44e6_a3e9_4c45e238d1bc.slice/crio-942b81c58f2a95c946999a14e9d3948ce75d5b61d2ed3c0b0cb7246d53f707c2 WatchSource:0}: Error finding container 942b81c58f2a95c946999a14e9d3948ce75d5b61d2ed3c0b0cb7246d53f707c2: Status 404 returned error can't find the container with id 942b81c58f2a95c946999a14e9d3948ce75d5b61d2ed3c0b0cb7246d53f707c2 Oct 29 23:06:57.285314 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:06:57.285280 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 23:06:57.285395 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:06:57.285370 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 23:06:57.285438 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:06:57.285402 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 23:06:58.012088 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:06:58.012045 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-k5t2b" event={"ID":"f8dc3f31-0a4c-44e6-a3e9-4c45e238d1bc","Type":"ContainerStarted","Data":"4ed085418f0efde7366ddf939dc16698831a8396b9388731e4e729ab98cc681c"} Oct 29 23:06:58.012278 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:06:58.012108 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-k5t2b" event={"ID":"f8dc3f31-0a4c-44e6-a3e9-4c45e238d1bc","Type":"ContainerStarted","Data":"942b81c58f2a95c946999a14e9d3948ce75d5b61d2ed3c0b0cb7246d53f707c2"} Oct 29 23:06:58.074092 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:06:58.074041 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-k5t2b" podStartSLOduration=2.074026133 podStartE2EDuration="2.074026133s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:06:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:06:58.072528655 +0000 UTC m=+1998.855742080" watchObservedRunningTime="2026-10-29 23:06:58.074026133 +0000 UTC m=+1998.857239559" Oct 29 23:06:58.274344 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:06:58.274242 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:06:58.953511 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:06:58.953473 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-k5t2b" Oct 29 23:06:58.954866 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:06:58.954841 5057 patch_prober.go:29] interesting pod/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-k5t2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" start-of-body= Oct 29 23:06:58.954943 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:06:58.954891 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-k5t2b" podUID="f8dc3f31-0a4c-44e6-a3e9-4c45e238d1bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" Oct 29 23:06:59.953312 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:06:59.953280 5057 patch_prober.go:29] interesting pod/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-k5t2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" start-of-body= Oct 29 23:06:59.953500 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:06:59.953364 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-k5t2b" podUID="f8dc3f31-0a4c-44e6-a3e9-4c45e238d1bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" Oct 29 23:07:00.047458 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:00.047382 5057 generic.go:356] "Generic (PLEG): container finished" podID="5954f1f5-c478-4f49-877e-9d3300568c2d" containerID="90020358042d221b88ea87fbb4cccb904f0836c0747ad5df003478138a7ec83a" exitCode=0 Oct 29 23:07:00.047979 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:00.047958 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-vmj4r" event={"ID":"5954f1f5-c478-4f49-877e-9d3300568c2d","Type":"ContainerDied","Data":"90020358042d221b88ea87fbb4cccb904f0836c0747ad5df003478138a7ec83a"} Oct 29 23:07:00.050089 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:00.050065 5057 generic.go:356] "Generic (PLEG): container finished" podID="43a9077f-2bb9-4801-abba-687de8a45076" containerID="cb4d8eb96cd3773d35f2f2bb1db5dc07f45ab4cfdec4e543c53e1253025cff5a" exitCode=0 Oct 29 23:07:00.051094 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:00.051070 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-hfrrx" event={"ID":"43a9077f-2bb9-4801-abba-687de8a45076","Type":"ContainerDied","Data":"cb4d8eb96cd3773d35f2f2bb1db5dc07f45ab4cfdec4e543c53e1253025cff5a"} Oct 29 23:07:00.172022 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:00.171993 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-vmj4r" Oct 29 23:07:00.228935 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:00.228905 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-hfrrx" Oct 29 23:07:00.249056 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:00.249026 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5954f1f5-c478-4f49-877e-9d3300568c2d-istio-token\" (UniqueName: \"kubernetes.io/projected/5954f1f5-c478-4f49-877e-9d3300568c2d-istio-token\") pod \"5954f1f5-c478-4f49-877e-9d3300568c2d\" (UID: \"5954f1f5-c478-4f49-877e-9d3300568c2d\") " Oct 29 23:07:00.249220 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:00.249080 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5954f1f5-c478-4f49-877e-9d3300568c2d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5954f1f5-c478-4f49-877e-9d3300568c2d-workload-socket\") pod \"5954f1f5-c478-4f49-877e-9d3300568c2d\" (UID: \"5954f1f5-c478-4f49-877e-9d3300568c2d\") " Oct 29 23:07:00.249220 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:00.249199 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5954f1f5-c478-4f49-877e-9d3300568c2d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5954f1f5-c478-4f49-877e-9d3300568c2d-credential-socket\") pod \"5954f1f5-c478-4f49-877e-9d3300568c2d\" (UID: \"5954f1f5-c478-4f49-877e-9d3300568c2d\") " Oct 29 23:07:00.249488 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:00.249461 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5954f1f5-c478-4f49-877e-9d3300568c2d-workload-socket" pod "5954f1f5-c478-4f49-877e-9d3300568c2d" (UID: "5954f1f5-c478-4f49-877e-9d3300568c2d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:07:00.249560 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:00.249503 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5954f1f5-c478-4f49-877e-9d3300568c2d-credential-socket" pod "5954f1f5-c478-4f49-877e-9d3300568c2d" (UID: "5954f1f5-c478-4f49-877e-9d3300568c2d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:07:00.249765 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:00.249748 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5954f1f5-c478-4f49-877e-9d3300568c2d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5954f1f5-c478-4f49-877e-9d3300568c2d-istio-envoy\") pod \"5954f1f5-c478-4f49-877e-9d3300568c2d\" (UID: \"5954f1f5-c478-4f49-877e-9d3300568c2d\") " Oct 29 23:07:00.249842 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:00.249800 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5954f1f5-c478-4f49-877e-9d3300568c2d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5954f1f5-c478-4f49-877e-9d3300568c2d-istiod-ca-cert\") pod \"5954f1f5-c478-4f49-877e-9d3300568c2d\" (UID: \"5954f1f5-c478-4f49-877e-9d3300568c2d\") " Oct 29 23:07:00.250155 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:00.250134 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5954f1f5-c478-4f49-877e-9d3300568c2d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5954f1f5-c478-4f49-877e-9d3300568c2d-istio-podinfo\") pod \"5954f1f5-c478-4f49-877e-9d3300568c2d\" (UID: \"5954f1f5-c478-4f49-877e-9d3300568c2d\") " Oct 29 23:07:00.250280 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:00.250135 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5954f1f5-c478-4f49-877e-9d3300568c2d-istiod-ca-cert" pod "5954f1f5-c478-4f49-877e-9d3300568c2d" (UID: "5954f1f5-c478-4f49-877e-9d3300568c2d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:07:00.250566 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:00.250319 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5954f1f5-c478-4f49-877e-9d3300568c2d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5954f1f5-c478-4f49-877e-9d3300568c2d-istio-data\") pod \"5954f1f5-c478-4f49-877e-9d3300568c2d\" (UID: \"5954f1f5-c478-4f49-877e-9d3300568c2d\") " Oct 29 23:07:00.250650 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:00.250608 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5954f1f5-c478-4f49-877e-9d3300568c2d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5954f1f5-c478-4f49-877e-9d3300568c2d-workload-certs\") pod \"5954f1f5-c478-4f49-877e-9d3300568c2d\" (UID: \"5954f1f5-c478-4f49-877e-9d3300568c2d\") " Oct 29 23:07:00.250650 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:00.250533 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5954f1f5-c478-4f49-877e-9d3300568c2d-istio-data" pod "5954f1f5-c478-4f49-877e-9d3300568c2d" (UID: "5954f1f5-c478-4f49-877e-9d3300568c2d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:07:00.250777 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:00.250711 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5954f1f5-c478-4f49-877e-9d3300568c2d-kube-api-access-rxwzp\" (UniqueName: \"kubernetes.io/projected/5954f1f5-c478-4f49-877e-9d3300568c2d-kube-api-access-rxwzp\") pod \"5954f1f5-c478-4f49-877e-9d3300568c2d\" (UID: \"5954f1f5-c478-4f49-877e-9d3300568c2d\") " Oct 29 23:07:00.250898 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:00.250876 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5954f1f5-c478-4f49-877e-9d3300568c2d-workload-certs" pod "5954f1f5-c478-4f49-877e-9d3300568c2d" (UID: "5954f1f5-c478-4f49-877e-9d3300568c2d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:07:00.253956 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:00.253584 5057 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5954f1f5-c478-4f49-877e-9d3300568c2d-istio-data\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:07:00.253956 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:00.253612 5057 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5954f1f5-c478-4f49-877e-9d3300568c2d-workload-certs\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:07:00.253956 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:00.253626 5057 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5954f1f5-c478-4f49-877e-9d3300568c2d-workload-socket\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:07:00.253956 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:00.253640 5057 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5954f1f5-c478-4f49-877e-9d3300568c2d-credential-socket\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:07:00.253956 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:00.253655 5057 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5954f1f5-c478-4f49-877e-9d3300568c2d-istiod-ca-cert\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:07:00.265963 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:00.265933 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5954f1f5-c478-4f49-877e-9d3300568c2d-istio-token" pod "5954f1f5-c478-4f49-877e-9d3300568c2d" (UID: "5954f1f5-c478-4f49-877e-9d3300568c2d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:07:00.269882 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:00.269855 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5954f1f5-c478-4f49-877e-9d3300568c2d-kube-api-access-rxwzp" pod "5954f1f5-c478-4f49-877e-9d3300568c2d" (UID: "5954f1f5-c478-4f49-877e-9d3300568c2d"). InnerVolumeSpecName "kube-api-access-rxwzp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:07:00.271757 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:00.271678 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5954f1f5-c478-4f49-877e-9d3300568c2d-istio-envoy" pod "5954f1f5-c478-4f49-877e-9d3300568c2d" (UID: "5954f1f5-c478-4f49-877e-9d3300568c2d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:07:00.273033 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:00.273012 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5954f1f5-c478-4f49-877e-9d3300568c2d-istio-podinfo" pod "5954f1f5-c478-4f49-877e-9d3300568c2d" (UID: "5954f1f5-c478-4f49-877e-9d3300568c2d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:07:00.356061 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:00.355244 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43a9077f-2bb9-4801-abba-687de8a45076-kube-api-access-xx56t\" (UniqueName: \"kubernetes.io/projected/43a9077f-2bb9-4801-abba-687de8a45076-kube-api-access-xx56t\") pod \"43a9077f-2bb9-4801-abba-687de8a45076\" (UID: \"43a9077f-2bb9-4801-abba-687de8a45076\") " Oct 29 23:07:00.356061 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:00.355312 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43a9077f-2bb9-4801-abba-687de8a45076-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43a9077f-2bb9-4801-abba-687de8a45076-workload-socket\") pod \"43a9077f-2bb9-4801-abba-687de8a45076\" (UID: \"43a9077f-2bb9-4801-abba-687de8a45076\") " Oct 29 23:07:00.356061 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:00.355392 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43a9077f-2bb9-4801-abba-687de8a45076-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43a9077f-2bb9-4801-abba-687de8a45076-credential-socket\") pod \"43a9077f-2bb9-4801-abba-687de8a45076\" (UID: \"43a9077f-2bb9-4801-abba-687de8a45076\") " Oct 29 23:07:00.356061 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:00.355568 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43a9077f-2bb9-4801-abba-687de8a45076-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43a9077f-2bb9-4801-abba-687de8a45076-istiod-ca-cert\") pod \"43a9077f-2bb9-4801-abba-687de8a45076\" (UID: \"43a9077f-2bb9-4801-abba-687de8a45076\") " Oct 29 23:07:00.356061 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:00.355663 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43a9077f-2bb9-4801-abba-687de8a45076-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43a9077f-2bb9-4801-abba-687de8a45076-istio-data\") pod \"43a9077f-2bb9-4801-abba-687de8a45076\" (UID: \"43a9077f-2bb9-4801-abba-687de8a45076\") " Oct 29 23:07:00.356061 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:00.355694 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43a9077f-2bb9-4801-abba-687de8a45076-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43a9077f-2bb9-4801-abba-687de8a45076-istio-envoy\") pod \"43a9077f-2bb9-4801-abba-687de8a45076\" (UID: \"43a9077f-2bb9-4801-abba-687de8a45076\") " Oct 29 23:07:00.356061 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:00.355729 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/43a9077f-2bb9-4801-abba-687de8a45076-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43a9077f-2bb9-4801-abba-687de8a45076-istio-podinfo\") pod \"43a9077f-2bb9-4801-abba-687de8a45076\" (UID: \"43a9077f-2bb9-4801-abba-687de8a45076\") " Oct 29 23:07:00.356061 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:00.355911 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43a9077f-2bb9-4801-abba-687de8a45076-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43a9077f-2bb9-4801-abba-687de8a45076-workload-certs\") pod \"43a9077f-2bb9-4801-abba-687de8a45076\" (UID: \"43a9077f-2bb9-4801-abba-687de8a45076\") " Oct 29 23:07:00.356061 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:00.355966 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43a9077f-2bb9-4801-abba-687de8a45076-credential-socket" pod "43a9077f-2bb9-4801-abba-687de8a45076" (UID: "43a9077f-2bb9-4801-abba-687de8a45076"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:07:00.356061 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:00.355982 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43a9077f-2bb9-4801-abba-687de8a45076-istio-token\" (UniqueName: \"kubernetes.io/projected/43a9077f-2bb9-4801-abba-687de8a45076-istio-token\") pod \"43a9077f-2bb9-4801-abba-687de8a45076\" (UID: \"43a9077f-2bb9-4801-abba-687de8a45076\") " Oct 29 23:07:00.356948 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:00.356162 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43a9077f-2bb9-4801-abba-687de8a45076-workload-socket" pod "43a9077f-2bb9-4801-abba-687de8a45076" (UID: "43a9077f-2bb9-4801-abba-687de8a45076"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:07:00.356948 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:00.356845 5057 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5954f1f5-c478-4f49-877e-9d3300568c2d-istio-envoy\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:07:00.356948 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:00.356866 5057 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43a9077f-2bb9-4801-abba-687de8a45076-workload-socket\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:07:00.356948 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:00.356880 5057 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43a9077f-2bb9-4801-abba-687de8a45076-credential-socket\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:07:00.356948 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:00.356886 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43a9077f-2bb9-4801-abba-687de8a45076-workload-certs" pod "43a9077f-2bb9-4801-abba-687de8a45076" (UID: "43a9077f-2bb9-4801-abba-687de8a45076"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:07:00.356948 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:00.356894 5057 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5954f1f5-c478-4f49-877e-9d3300568c2d-istio-podinfo\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:07:00.357272 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:00.356957 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rxwzp\" (UniqueName: \"kubernetes.io/projected/5954f1f5-c478-4f49-877e-9d3300568c2d-kube-api-access-rxwzp\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:07:00.357272 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:00.356977 5057 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5954f1f5-c478-4f49-877e-9d3300568c2d-istio-token\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:07:00.357407 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:00.357280 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43a9077f-2bb9-4801-abba-687de8a45076-istiod-ca-cert" pod "43a9077f-2bb9-4801-abba-687de8a45076" (UID: "43a9077f-2bb9-4801-abba-687de8a45076"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:07:00.357616 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:00.357588 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43a9077f-2bb9-4801-abba-687de8a45076-istio-data" pod "43a9077f-2bb9-4801-abba-687de8a45076" (UID: "43a9077f-2bb9-4801-abba-687de8a45076"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:07:00.359521 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:00.359494 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43a9077f-2bb9-4801-abba-687de8a45076-istio-token" pod "43a9077f-2bb9-4801-abba-687de8a45076" (UID: "43a9077f-2bb9-4801-abba-687de8a45076"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:07:00.359673 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:00.359648 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/43a9077f-2bb9-4801-abba-687de8a45076-istio-podinfo" pod "43a9077f-2bb9-4801-abba-687de8a45076" (UID: "43a9077f-2bb9-4801-abba-687de8a45076"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:07:00.359856 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:00.359837 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43a9077f-2bb9-4801-abba-687de8a45076-kube-api-access-xx56t" pod "43a9077f-2bb9-4801-abba-687de8a45076" (UID: "43a9077f-2bb9-4801-abba-687de8a45076"). InnerVolumeSpecName "kube-api-access-xx56t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:07:00.361386 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:00.361363 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43a9077f-2bb9-4801-abba-687de8a45076-istio-envoy" pod "43a9077f-2bb9-4801-abba-687de8a45076" (UID: "43a9077f-2bb9-4801-abba-687de8a45076"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:07:00.422362 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:00.422310 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:07:00.460232 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:00.458843 5057 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43a9077f-2bb9-4801-abba-687de8a45076-workload-certs\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:07:00.460232 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:00.458878 5057 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43a9077f-2bb9-4801-abba-687de8a45076-istio-token\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:07:00.460232 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:00.458899 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xx56t\" (UniqueName: \"kubernetes.io/projected/43a9077f-2bb9-4801-abba-687de8a45076-kube-api-access-xx56t\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:07:00.460232 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:00.458910 5057 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43a9077f-2bb9-4801-abba-687de8a45076-istiod-ca-cert\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:07:00.460232 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:00.458921 5057 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43a9077f-2bb9-4801-abba-687de8a45076-istio-data\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:07:00.460232 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:00.458929 5057 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43a9077f-2bb9-4801-abba-687de8a45076-istio-envoy\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:07:00.460232 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:00.458938 5057 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43a9077f-2bb9-4801-abba-687de8a45076-istio-podinfo\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:07:00.953019 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:00.952981 5057 patch_prober.go:29] interesting pod/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-k5t2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" start-of-body= Oct 29 23:07:00.953184 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:00.953044 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-k5t2b" podUID="f8dc3f31-0a4c-44e6-a3e9-4c45e238d1bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" Oct 29 23:07:01.031691 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:01.031584 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:07:01.060092 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:01.060058 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-vmj4r" Oct 29 23:07:01.060520 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:01.060096 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-vmj4r" event={"ID":"5954f1f5-c478-4f49-877e-9d3300568c2d","Type":"ContainerDied","Data":"438b58ac6bf6ef644c08c6d89e00527781bd576c4079e4d1e7ee8a109f980186"} Oct 29 23:07:01.060520 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:01.060159 5057 scope.go:122] "RemoveContainer" containerID="90020358042d221b88ea87fbb4cccb904f0836c0747ad5df003478138a7ec83a" Oct 29 23:07:01.061876 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:01.061856 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-hfrrx" Oct 29 23:07:01.062024 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:01.061887 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-hfrrx" event={"ID":"43a9077f-2bb9-4801-abba-687de8a45076","Type":"ContainerDied","Data":"a855f564baf16681264119ad39dc6b27c474133d10eee8ba886f2dd148ad58a3"} Oct 29 23:07:01.072220 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:01.072199 5057 scope.go:122] "RemoveContainer" containerID="cb4d8eb96cd3773d35f2f2bb1db5dc07f45ab4cfdec4e543c53e1253025cff5a" Oct 29 23:07:01.125434 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:01.123585 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-vmj4r"] Oct 29 23:07:01.127923 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:01.127897 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-vmj4r"] Oct 29 23:07:01.169144 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:01.169109 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-hfrrx"] Oct 29 23:07:01.198356 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:01.193885 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:07:01.217416 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:01.210378 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-hfrrx"] Oct 29 23:07:01.340628 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:01.340541 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43a9077f-2bb9-4801-abba-687de8a45076" path="/var/lib/kubelet/pods/43a9077f-2bb9-4801-abba-687de8a45076/volumes" Oct 29 23:07:01.341490 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:01.341384 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5954f1f5-c478-4f49-877e-9d3300568c2d" path="/var/lib/kubelet/pods/5954f1f5-c478-4f49-877e-9d3300568c2d/volumes" Oct 29 23:07:01.422715 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:01.422570 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:07:01.952991 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:01.952955 5057 patch_prober.go:29] interesting pod/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-k5t2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" start-of-body= Oct 29 23:07:01.953170 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:01.953006 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-k5t2b" podUID="f8dc3f31-0a4c-44e6-a3e9-4c45e238d1bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" Oct 29 23:07:02.212421 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:02.207185 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-d67deb-8598cdf4b8-jr6n7"] Oct 29 23:07:02.235970 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:02.235937 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-d67deb-d675bc8c9-v9tg7"] Oct 29 23:07:02.373138 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:02.373002 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:07:02.513588 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:02.512732 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:07:02.960352 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:02.958785 5057 patch_prober.go:29] interesting pod/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-k5t2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" start-of-body= Oct 29 23:07:02.960352 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:02.958860 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-k5t2b" podUID="f8dc3f31-0a4c-44e6-a3e9-4c45e238d1bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" Oct 29 23:07:03.957570 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:03.957536 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-k5t2b" Oct 29 23:07:03.958125 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:03.958103 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-k5t2b" Oct 29 23:07:03.958913 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:03.958894 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-k5t2b" Oct 29 23:07:04.236479 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:04.236381 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:07:05.944927 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:05.944891 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:07:07.215826 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:07.215744 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-d67deb-8598cdf4b8-jr6n7" podUID="b4535855-9770-43ca-abb3-3ee2a56d03d3" containerName="payload-processing" containerID="cri-o://adbc57b07e9826106e6717704ae2c272e710f7fa49dede434bc3693b332f16c2" gracePeriod=25 Oct 29 23:07:07.237135 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:07.237088 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d67deb-d675bc8c9-v9tg7" podUID="cd71b31d-7725-420c-94ab-f94eb006b35f" containerName="payload-pre-processing" containerID="cri-o://63e632a5435d8c6d99e09bb85d22e31951cec29bf78cfc7895fb4617c581ba52" gracePeriod=25 Oct 29 23:07:07.617844 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:07.617742 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:07:09.410264 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:09.410225 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:07:11.082755 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:11.082716 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:07:12.624322 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:12.624282 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:07:13.713713 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:13.713672 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d67deb-d675bc8c9-v9tg7" podUID="cd71b31d-7725-420c-94ab-f94eb006b35f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:07:13.714227 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:13.713677 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-praxis-a-d67deb-8598cdf4b8-jr6n7" podUID="b4535855-9770-43ca-abb3-3ee2a56d03d3" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:07:14.200362 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:14.200291 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:07:16.086212 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:16.086152 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:07:16.732152 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:16.732105 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:07:17.339566 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:17.339530 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:07:17.487025 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:17.486805 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:07:17.778775 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:17.778738 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-77238f-5db9746b9f-7cvgq"] Oct 29 23:07:17.780344 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:17.780294 5057 cpu_manager.go:414] "containerMap: removing container" podUID="5954f1f5-c478-4f49-877e-9d3300568c2d" containerName="istio-proxy" Oct 29 23:07:17.780591 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:17.780540 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5954f1f5-c478-4f49-877e-9d3300568c2d" containerName="istio-proxy" Oct 29 23:07:17.780770 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:17.780748 5057 cpu_manager.go:414] "containerMap: removing container" podUID="43a9077f-2bb9-4801-abba-687de8a45076" containerName="istio-proxy" Oct 29 23:07:17.780871 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:17.780859 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43a9077f-2bb9-4801-abba-687de8a45076" containerName="istio-proxy" Oct 29 23:07:17.788365 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:17.781306 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="5954f1f5-c478-4f49-877e-9d3300568c2d" containerName="istio-proxy" Oct 29 23:07:17.788554 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:17.788537 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="43a9077f-2bb9-4801-abba-687de8a45076" containerName="istio-proxy" Oct 29 23:07:17.804865 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:17.804836 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-77238f-5db9746b9f-7cvgq"] Oct 29 23:07:17.805089 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:17.804977 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-77238f-5db9746b9f-7cvgq" Oct 29 23:07:17.825404 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:17.825374 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-77238f\"" Oct 29 23:07:17.884355 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:17.841620 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-77238f-dockercfg-fb5pv\"" Oct 29 23:07:17.955876 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:17.919025 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-77238f-5c9c8b447d-ksqsl"] Oct 29 23:07:17.955876 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:17.930989 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-77238f-5c9c8b447d-ksqsl" Oct 29 23:07:17.996152 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:17.995512 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qtgsz\" (UniqueName: \"kubernetes.io/projected/96d53b6e-441d-411e-aa0c-efcb79e89983-kube-api-access-qtgsz\") pod \"payload-pre-processing-e2e-ipp-b-77238f-5db9746b9f-7cvgq\" (UID: \"96d53b6e-441d-411e-aa0c-efcb79e89983\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-77238f-5db9746b9f-7cvgq" Oct 29 23:07:17.996152 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:17.995663 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96d53b6e-441d-411e-aa0c-efcb79e89983-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-77238f-5db9746b9f-7cvgq\" (UID: \"96d53b6e-441d-411e-aa0c-efcb79e89983\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-77238f-5db9746b9f-7cvgq" Oct 29 23:07:17.996152 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:17.995833 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cq2gv\" (UniqueName: \"kubernetes.io/projected/27e3d9c6-89a4-42fd-9a98-c030f6a22990-kube-api-access-cq2gv\") pod \"payload-processing-e2e-ipp-b-77238f-5c9c8b447d-ksqsl\" (UID: \"27e3d9c6-89a4-42fd-9a98-c030f6a22990\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-77238f-5c9c8b447d-ksqsl" Oct 29 23:07:17.996152 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:17.995916 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27e3d9c6-89a4-42fd-9a98-c030f6a22990-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-77238f-5c9c8b447d-ksqsl\" (UID: \"27e3d9c6-89a4-42fd-9a98-c030f6a22990\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-77238f-5c9c8b447d-ksqsl" Oct 29 23:07:18.041491 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:18.041398 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-77238f-5c9c8b447d-ksqsl"] Oct 29 23:07:18.097018 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:18.096986 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96d53b6e-441d-411e-aa0c-efcb79e89983-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-77238f-5db9746b9f-7cvgq\" (UID: \"96d53b6e-441d-411e-aa0c-efcb79e89983\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-77238f-5db9746b9f-7cvgq" Oct 29 23:07:18.097206 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:18.097140 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cq2gv\" (UniqueName: \"kubernetes.io/projected/27e3d9c6-89a4-42fd-9a98-c030f6a22990-kube-api-access-cq2gv\") pod \"payload-processing-e2e-ipp-b-77238f-5c9c8b447d-ksqsl\" (UID: \"27e3d9c6-89a4-42fd-9a98-c030f6a22990\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-77238f-5c9c8b447d-ksqsl" Oct 29 23:07:18.097206 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:18.097184 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27e3d9c6-89a4-42fd-9a98-c030f6a22990-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-77238f-5c9c8b447d-ksqsl\" (UID: \"27e3d9c6-89a4-42fd-9a98-c030f6a22990\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-77238f-5c9c8b447d-ksqsl" Oct 29 23:07:18.097360 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:18.097278 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtgsz\" (UniqueName: \"kubernetes.io/projected/96d53b6e-441d-411e-aa0c-efcb79e89983-kube-api-access-qtgsz\") pod \"payload-pre-processing-e2e-ipp-b-77238f-5db9746b9f-7cvgq\" (UID: \"96d53b6e-441d-411e-aa0c-efcb79e89983\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-77238f-5db9746b9f-7cvgq" Oct 29 23:07:18.097970 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:18.097936 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96d53b6e-441d-411e-aa0c-efcb79e89983-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-77238f-5db9746b9f-7cvgq\" (UID: \"96d53b6e-441d-411e-aa0c-efcb79e89983\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-77238f-5db9746b9f-7cvgq" Oct 29 23:07:18.098162 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:18.098146 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27e3d9c6-89a4-42fd-9a98-c030f6a22990-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-77238f-5c9c8b447d-ksqsl\" (UID: \"27e3d9c6-89a4-42fd-9a98-c030f6a22990\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-77238f-5c9c8b447d-ksqsl" Oct 29 23:07:18.119081 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:18.118950 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtgsz\" (UniqueName: \"kubernetes.io/projected/96d53b6e-441d-411e-aa0c-efcb79e89983-kube-api-access-qtgsz\") pod \"payload-pre-processing-e2e-ipp-b-77238f-5db9746b9f-7cvgq\" (UID: \"96d53b6e-441d-411e-aa0c-efcb79e89983\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-77238f-5db9746b9f-7cvgq" Oct 29 23:07:18.119081 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:18.119024 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cq2gv\" (UniqueName: \"kubernetes.io/projected/27e3d9c6-89a4-42fd-9a98-c030f6a22990-kube-api-access-cq2gv\") pod \"payload-processing-e2e-ipp-b-77238f-5c9c8b447d-ksqsl\" (UID: \"27e3d9c6-89a4-42fd-9a98-c030f6a22990\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-77238f-5c9c8b447d-ksqsl" Oct 29 23:07:18.305698 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:18.305614 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-77238f-5db9746b9f-7cvgq" Oct 29 23:07:18.337828 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:18.337788 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-77238f-5c9c8b447d-ksqsl" Oct 29 23:07:18.552054 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:18.552014 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:07:18.631868 ip-10-0-140-74 kubenswrapper[5057]: W1029 23:07:18.631829 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96d53b6e_441d_411e_aa0c_efcb79e89983.slice/crio-3cc7bfc93ffe0084eed517575e4a94bc2441aa34fee6d3479026fc2fd03d0896 WatchSource:0}: Error finding container 3cc7bfc93ffe0084eed517575e4a94bc2441aa34fee6d3479026fc2fd03d0896: Status 404 returned error can't find the container with id 3cc7bfc93ffe0084eed517575e4a94bc2441aa34fee6d3479026fc2fd03d0896 Oct 29 23:07:18.634256 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:18.634233 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-77238f-5db9746b9f-7cvgq"] Oct 29 23:07:18.687534 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:18.687508 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-77238f-5c9c8b447d-ksqsl"] Oct 29 23:07:18.690984 ip-10-0-140-74 kubenswrapper[5057]: W1029 23:07:18.690954 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27e3d9c6_89a4_42fd_9a98_c030f6a22990.slice/crio-8a234877436ec56d87b6ecbafd69ab7b984179a5f955c809c0f2c4ab6e699d53 WatchSource:0}: Error finding container 8a234877436ec56d87b6ecbafd69ab7b984179a5f955c809c0f2c4ab6e699d53: Status 404 returned error can't find the container with id 8a234877436ec56d87b6ecbafd69ab7b984179a5f955c809c0f2c4ab6e699d53 Oct 29 23:07:19.035396 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:07:19.035362 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d2dnn" containerName="extractor" Oct 29 23:07:19.282294 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:19.282257 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-77238f-5c9c8b447d-ksqsl" event={"ID":"27e3d9c6-89a4-42fd-9a98-c030f6a22990","Type":"ContainerStarted","Data":"a2c731e87b3a2c106692ea57c197e10939529d0a4967ee09ae45faa632e50518"} Oct 29 23:07:19.282507 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:19.282319 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-77238f-5c9c8b447d-ksqsl" event={"ID":"27e3d9c6-89a4-42fd-9a98-c030f6a22990","Type":"ContainerStarted","Data":"8a234877436ec56d87b6ecbafd69ab7b984179a5f955c809c0f2c4ab6e699d53"} Oct 29 23:07:19.283293 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:19.283270 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-77238f-5c9c8b447d-ksqsl" Oct 29 23:07:19.285680 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:19.285606 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-77238f-5db9746b9f-7cvgq" event={"ID":"96d53b6e-441d-411e-aa0c-efcb79e89983","Type":"ContainerStarted","Data":"f325daa6f7d43b2c19c6b89804ce93ded4d488befe209d70e81b8a31671f885f"} Oct 29 23:07:19.285800 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:19.285701 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-77238f-5db9746b9f-7cvgq" Oct 29 23:07:19.285800 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:19.285730 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-77238f-5db9746b9f-7cvgq" event={"ID":"96d53b6e-441d-411e-aa0c-efcb79e89983","Type":"ContainerStarted","Data":"3cc7bfc93ffe0084eed517575e4a94bc2441aa34fee6d3479026fc2fd03d0896"} Oct 29 23:07:19.322075 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:19.322026 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-77238f-5c9c8b447d-ksqsl" podStartSLOduration=2.32201037 podStartE2EDuration="2.32201037s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:07:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:07:19.318568946 +0000 UTC m=+2020.101782374" watchObservedRunningTime="2026-10-29 23:07:19.32201037 +0000 UTC m=+2020.105223797" Oct 29 23:07:19.358478 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:19.358426 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-77238f-5db9746b9f-7cvgq" podStartSLOduration=2.358414207 podStartE2EDuration="2.358414207s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:07:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:07:19.355568758 +0000 UTC m=+2020.138782183" watchObservedRunningTime="2026-10-29 23:07:19.358414207 +0000 UTC m=+2020.141627632" Oct 29 23:07:20.376548 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:20.376489 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:07:22.143607 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:22.143569 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:07:23.713504 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:23.713461 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-praxis-a-d67deb-8598cdf4b8-jr6n7" podUID="b4535855-9770-43ca-abb3-3ee2a56d03d3" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:07:23.713504 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:23.713481 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d67deb-d675bc8c9-v9tg7" podUID="cd71b31d-7725-420c-94ab-f94eb006b35f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:07:24.060429 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:24.059937 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:07:24.636799 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:24.636764 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:07:24.995268 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:24.995229 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:07:26.677927 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:26.677885 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-2mcsr"] Oct 29 23:07:26.683841 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:26.683816 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-2mcsr" Oct 29 23:07:26.712080 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:26.712045 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-2mcsr"] Oct 29 23:07:26.782262 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:26.782224 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e61a681b-48a0-4b15-bea3-013044a07714-credential-socket\") pod \"e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-2mcsr\" (UID: \"e61a681b-48a0-4b15-bea3-013044a07714\") " pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-2mcsr" Oct 29 23:07:26.782547 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:26.782519 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e61a681b-48a0-4b15-bea3-013044a07714-istiod-ca-cert\") pod \"e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-2mcsr\" (UID: \"e61a681b-48a0-4b15-bea3-013044a07714\") " pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-2mcsr" Oct 29 23:07:26.782702 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:26.782596 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e61a681b-48a0-4b15-bea3-013044a07714-workload-certs\") pod \"e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-2mcsr\" (UID: \"e61a681b-48a0-4b15-bea3-013044a07714\") " pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-2mcsr" Oct 29 23:07:26.782771 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:26.782709 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e61a681b-48a0-4b15-bea3-013044a07714-workload-socket\") pod \"e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-2mcsr\" (UID: \"e61a681b-48a0-4b15-bea3-013044a07714\") " pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-2mcsr" Oct 29 23:07:26.782771 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:26.782753 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e61a681b-48a0-4b15-bea3-013044a07714-istio-envoy\") pod \"e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-2mcsr\" (UID: \"e61a681b-48a0-4b15-bea3-013044a07714\") " pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-2mcsr" Oct 29 23:07:26.782877 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:26.782797 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dntns\" (UniqueName: \"kubernetes.io/projected/e61a681b-48a0-4b15-bea3-013044a07714-kube-api-access-dntns\") pod \"e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-2mcsr\" (UID: \"e61a681b-48a0-4b15-bea3-013044a07714\") " pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-2mcsr" Oct 29 23:07:26.782877 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:26.782841 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e61a681b-48a0-4b15-bea3-013044a07714-istio-data\") pod \"e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-2mcsr\" (UID: \"e61a681b-48a0-4b15-bea3-013044a07714\") " pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-2mcsr" Oct 29 23:07:26.782981 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:26.782890 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e61a681b-48a0-4b15-bea3-013044a07714-istio-podinfo\") pod \"e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-2mcsr\" (UID: \"e61a681b-48a0-4b15-bea3-013044a07714\") " pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-2mcsr" Oct 29 23:07:26.783079 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:26.783064 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e61a681b-48a0-4b15-bea3-013044a07714-istio-token\") pod \"e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-2mcsr\" (UID: \"e61a681b-48a0-4b15-bea3-013044a07714\") " pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-2mcsr" Oct 29 23:07:26.853016 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:26.852980 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:07:26.884362 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:26.884316 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e61a681b-48a0-4b15-bea3-013044a07714-istiod-ca-cert\") pod \"e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-2mcsr\" (UID: \"e61a681b-48a0-4b15-bea3-013044a07714\") " pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-2mcsr" Oct 29 23:07:26.884557 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:26.884407 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e61a681b-48a0-4b15-bea3-013044a07714-workload-certs\") pod \"e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-2mcsr\" (UID: \"e61a681b-48a0-4b15-bea3-013044a07714\") " pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-2mcsr" Oct 29 23:07:26.884557 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:26.884497 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e61a681b-48a0-4b15-bea3-013044a07714-workload-socket\") pod \"e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-2mcsr\" (UID: \"e61a681b-48a0-4b15-bea3-013044a07714\") " pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-2mcsr" Oct 29 23:07:26.884557 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:26.884530 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e61a681b-48a0-4b15-bea3-013044a07714-istio-envoy\") pod \"e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-2mcsr\" (UID: \"e61a681b-48a0-4b15-bea3-013044a07714\") " pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-2mcsr" Oct 29 23:07:26.884741 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:26.884574 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dntns\" (UniqueName: \"kubernetes.io/projected/e61a681b-48a0-4b15-bea3-013044a07714-kube-api-access-dntns\") pod \"e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-2mcsr\" (UID: \"e61a681b-48a0-4b15-bea3-013044a07714\") " pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-2mcsr" Oct 29 23:07:26.884741 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:26.884612 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e61a681b-48a0-4b15-bea3-013044a07714-istio-data\") pod \"e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-2mcsr\" (UID: \"e61a681b-48a0-4b15-bea3-013044a07714\") " pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-2mcsr" Oct 29 23:07:26.884741 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:26.884645 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e61a681b-48a0-4b15-bea3-013044a07714-istio-podinfo\") pod \"e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-2mcsr\" (UID: \"e61a681b-48a0-4b15-bea3-013044a07714\") " pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-2mcsr" Oct 29 23:07:26.885315 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:26.884786 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e61a681b-48a0-4b15-bea3-013044a07714-istio-token\") pod \"e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-2mcsr\" (UID: \"e61a681b-48a0-4b15-bea3-013044a07714\") " pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-2mcsr" Oct 29 23:07:26.885315 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:26.884907 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e61a681b-48a0-4b15-bea3-013044a07714-credential-socket\") pod \"e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-2mcsr\" (UID: \"e61a681b-48a0-4b15-bea3-013044a07714\") " pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-2mcsr" Oct 29 23:07:26.885315 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:26.885196 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e61a681b-48a0-4b15-bea3-013044a07714-istio-data\") pod \"e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-2mcsr\" (UID: \"e61a681b-48a0-4b15-bea3-013044a07714\") " pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-2mcsr" Oct 29 23:07:26.886711 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:26.884864 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e61a681b-48a0-4b15-bea3-013044a07714-workload-certs\") pod \"e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-2mcsr\" (UID: \"e61a681b-48a0-4b15-bea3-013044a07714\") " pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-2mcsr" Oct 29 23:07:26.886711 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:26.886168 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e61a681b-48a0-4b15-bea3-013044a07714-istiod-ca-cert\") pod \"e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-2mcsr\" (UID: \"e61a681b-48a0-4b15-bea3-013044a07714\") " pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-2mcsr" Oct 29 23:07:26.886711 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:26.886200 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e61a681b-48a0-4b15-bea3-013044a07714-workload-socket\") pod \"e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-2mcsr\" (UID: \"e61a681b-48a0-4b15-bea3-013044a07714\") " pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-2mcsr" Oct 29 23:07:26.888583 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:26.888560 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e61a681b-48a0-4b15-bea3-013044a07714-istio-podinfo\") pod \"e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-2mcsr\" (UID: \"e61a681b-48a0-4b15-bea3-013044a07714\") " pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-2mcsr" Oct 29 23:07:26.893380 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:26.890097 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e61a681b-48a0-4b15-bea3-013044a07714-credential-socket\") pod \"e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-2mcsr\" (UID: \"e61a681b-48a0-4b15-bea3-013044a07714\") " pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-2mcsr" Oct 29 23:07:26.893380 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:26.892221 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e61a681b-48a0-4b15-bea3-013044a07714-istio-envoy\") pod \"e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-2mcsr\" (UID: \"e61a681b-48a0-4b15-bea3-013044a07714\") " pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-2mcsr" Oct 29 23:07:26.918773 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:26.917620 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dntns\" (UniqueName: \"kubernetes.io/projected/e61a681b-48a0-4b15-bea3-013044a07714-kube-api-access-dntns\") pod \"e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-2mcsr\" (UID: \"e61a681b-48a0-4b15-bea3-013044a07714\") " pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-2mcsr" Oct 29 23:07:26.919270 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:26.919244 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e61a681b-48a0-4b15-bea3-013044a07714-istio-token\") pod \"e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-2mcsr\" (UID: \"e61a681b-48a0-4b15-bea3-013044a07714\") " pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-2mcsr" Oct 29 23:07:27.008165 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:27.008124 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-2mcsr" Oct 29 23:07:27.344471 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:27.344438 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-2mcsr"] Oct 29 23:07:27.345195 ip-10-0-140-74 kubenswrapper[5057]: W1029 23:07:27.345169 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode61a681b_48a0_4b15_bea3_013044a07714.slice/crio-139b73a73512a5b10f5f8f997ee4fe307a47f8054f4696c73210bb0105ae83d2 WatchSource:0}: Error finding container 139b73a73512a5b10f5f8f997ee4fe307a47f8054f4696c73210bb0105ae83d2: Status 404 returned error can't find the container with id 139b73a73512a5b10f5f8f997ee4fe307a47f8054f4696c73210bb0105ae83d2 Oct 29 23:07:27.347892 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:27.347857 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 23:07:27.348010 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:27.347934 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 23:07:27.348010 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:27.347978 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 23:07:27.426832 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:27.426792 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-2mcsr" event={"ID":"e61a681b-48a0-4b15-bea3-013044a07714","Type":"ContainerStarted","Data":"139b73a73512a5b10f5f8f997ee4fe307a47f8054f4696c73210bb0105ae83d2"} Oct 29 23:07:27.430009 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:27.429977 5057 generic.go:356] "Generic (PLEG): container finished" podID="b4535855-9770-43ca-abb3-3ee2a56d03d3" containerID="adbc57b07e9826106e6717704ae2c272e710f7fa49dede434bc3693b332f16c2" exitCode=0 Oct 29 23:07:27.430153 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:27.430103 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-d67deb-8598cdf4b8-jr6n7" event={"ID":"b4535855-9770-43ca-abb3-3ee2a56d03d3","Type":"ContainerDied","Data":"adbc57b07e9826106e6717704ae2c272e710f7fa49dede434bc3693b332f16c2"} Oct 29 23:07:27.432349 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:27.432293 5057 generic.go:356] "Generic (PLEG): container finished" podID="cd71b31d-7725-420c-94ab-f94eb006b35f" containerID="63e632a5435d8c6d99e09bb85d22e31951cec29bf78cfc7895fb4617c581ba52" exitCode=0 Oct 29 23:07:27.432464 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:27.432397 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d67deb-d675bc8c9-v9tg7" event={"ID":"cd71b31d-7725-420c-94ab-f94eb006b35f","Type":"ContainerDied","Data":"63e632a5435d8c6d99e09bb85d22e31951cec29bf78cfc7895fb4617c581ba52"} Oct 29 23:07:27.537668 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:27.537605 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-d67deb-8598cdf4b8-jr6n7" Oct 29 23:07:27.619352 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:27.619024 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4535855-9770-43ca-abb3-3ee2a56d03d3-kube-api-access-nnvgk\" (UniqueName: \"kubernetes.io/projected/b4535855-9770-43ca-abb3-3ee2a56d03d3-kube-api-access-nnvgk\") pod \"b4535855-9770-43ca-abb3-3ee2a56d03d3\" (UID: \"b4535855-9770-43ca-abb3-3ee2a56d03d3\") " Oct 29 23:07:27.619352 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:27.619253 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4535855-9770-43ca-abb3-3ee2a56d03d3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4535855-9770-43ca-abb3-3ee2a56d03d3-plugins-config-volume\") pod \"b4535855-9770-43ca-abb3-3ee2a56d03d3\" (UID: \"b4535855-9770-43ca-abb3-3ee2a56d03d3\") " Oct 29 23:07:27.624387 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:27.620560 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4535855-9770-43ca-abb3-3ee2a56d03d3-plugins-config-volume" pod "b4535855-9770-43ca-abb3-3ee2a56d03d3" (UID: "b4535855-9770-43ca-abb3-3ee2a56d03d3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:07:27.630702 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:27.630563 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4535855-9770-43ca-abb3-3ee2a56d03d3-kube-api-access-nnvgk" pod "b4535855-9770-43ca-abb3-3ee2a56d03d3" (UID: "b4535855-9770-43ca-abb3-3ee2a56d03d3"). InnerVolumeSpecName "kube-api-access-nnvgk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:07:27.646670 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:27.646649 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d67deb-d675bc8c9-v9tg7" Oct 29 23:07:27.722432 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:27.721033 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd71b31d-7725-420c-94ab-f94eb006b35f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd71b31d-7725-420c-94ab-f94eb006b35f-plugins-config-volume\") pod \"cd71b31d-7725-420c-94ab-f94eb006b35f\" (UID: \"cd71b31d-7725-420c-94ab-f94eb006b35f\") " Oct 29 23:07:27.722432 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:27.721087 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd71b31d-7725-420c-94ab-f94eb006b35f-kube-api-access-m9gbg\" (UniqueName: \"kubernetes.io/projected/cd71b31d-7725-420c-94ab-f94eb006b35f-kube-api-access-m9gbg\") pod \"cd71b31d-7725-420c-94ab-f94eb006b35f\" (UID: \"cd71b31d-7725-420c-94ab-f94eb006b35f\") " Oct 29 23:07:27.722432 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:27.722032 5057 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4535855-9770-43ca-abb3-3ee2a56d03d3-plugins-config-volume\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:07:27.722432 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:27.722057 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nnvgk\" (UniqueName: \"kubernetes.io/projected/b4535855-9770-43ca-abb3-3ee2a56d03d3-kube-api-access-nnvgk\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:07:27.723577 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:27.723552 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd71b31d-7725-420c-94ab-f94eb006b35f-plugins-config-volume" pod "cd71b31d-7725-420c-94ab-f94eb006b35f" (UID: "cd71b31d-7725-420c-94ab-f94eb006b35f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:07:27.724813 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:27.724790 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd71b31d-7725-420c-94ab-f94eb006b35f-kube-api-access-m9gbg" pod "cd71b31d-7725-420c-94ab-f94eb006b35f" (UID: "cd71b31d-7725-420c-94ab-f94eb006b35f"). InnerVolumeSpecName "kube-api-access-m9gbg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:07:27.823755 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:27.823672 5057 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd71b31d-7725-420c-94ab-f94eb006b35f-plugins-config-volume\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:07:27.823755 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:27.823705 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m9gbg\" (UniqueName: \"kubernetes.io/projected/cd71b31d-7725-420c-94ab-f94eb006b35f-kube-api-access-m9gbg\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:07:28.447278 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:28.447248 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d67deb-d675bc8c9-v9tg7" Oct 29 23:07:28.447490 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:28.447279 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d67deb-d675bc8c9-v9tg7" event={"ID":"cd71b31d-7725-420c-94ab-f94eb006b35f","Type":"ContainerDied","Data":"6ee0a34a09fa045ee25d5f711f1de0a2e38f463ee3e7689c26f441069bb6b084"} Oct 29 23:07:28.447490 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:28.447355 5057 scope.go:122] "RemoveContainer" containerID="63e632a5435d8c6d99e09bb85d22e31951cec29bf78cfc7895fb4617c581ba52" Oct 29 23:07:28.449416 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:28.449392 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-2mcsr" event={"ID":"e61a681b-48a0-4b15-bea3-013044a07714","Type":"ContainerStarted","Data":"13c8406382cc22f056158c48a1139c2274b7aef44fe6a7b5d36231960ba8f688"} Oct 29 23:07:28.451164 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:28.451141 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-d67deb-8598cdf4b8-jr6n7" event={"ID":"b4535855-9770-43ca-abb3-3ee2a56d03d3","Type":"ContainerDied","Data":"211ca9c67377b85cdb5ac47fa801ff23c75b83230170592576b0443d848eceeb"} Oct 29 23:07:28.451259 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:28.451169 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-d67deb-8598cdf4b8-jr6n7" Oct 29 23:07:28.460141 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:28.460119 5057 scope.go:122] "RemoveContainer" containerID="adbc57b07e9826106e6717704ae2c272e710f7fa49dede434bc3693b332f16c2" Oct 29 23:07:28.530382 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:28.529424 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-d67deb-d675bc8c9-v9tg7"] Oct 29 23:07:28.533359 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:28.533014 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-d67deb-d675bc8c9-v9tg7"] Oct 29 23:07:28.557496 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:28.557448 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-2mcsr" podStartSLOduration=2.557433921 podStartE2EDuration="2.557433921s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:07:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:07:28.556772791 +0000 UTC m=+2029.339986217" watchObservedRunningTime="2026-10-29 23:07:28.557433921 +0000 UTC m=+2029.340647372" Oct 29 23:07:28.620005 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:28.619968 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-d67deb-8598cdf4b8-jr6n7"] Oct 29 23:07:28.627050 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:28.627016 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-d67deb-8598cdf4b8-jr6n7"] Oct 29 23:07:28.714349 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:28.714239 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:07:29.010614 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:29.010512 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-2mcsr" Oct 29 23:07:29.013530 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:29.013500 5057 patch_prober.go:29] interesting pod/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-2mcsr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body= Oct 29 23:07:29.013677 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:29.013549 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-2mcsr" podUID="e61a681b-48a0-4b15-bea3-013044a07714" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" Oct 29 23:07:29.345838 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:29.345760 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4535855-9770-43ca-abb3-3ee2a56d03d3" path="/var/lib/kubelet/pods/b4535855-9770-43ca-abb3-3ee2a56d03d3/volumes" Oct 29 23:07:29.346392 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:29.346372 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd71b31d-7725-420c-94ab-f94eb006b35f" path="/var/lib/kubelet/pods/cd71b31d-7725-420c-94ab-f94eb006b35f/volumes" Oct 29 23:07:29.540701 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:29.540660 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:07:29.657290 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:29.657114 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:07:30.011295 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:30.010779 5057 patch_prober.go:29] interesting pod/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-2mcsr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body= Oct 29 23:07:30.011295 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:30.010849 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-2mcsr" podUID="e61a681b-48a0-4b15-bea3-013044a07714" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" Oct 29 23:07:30.316410 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:30.310498 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-77238f-5db9746b9f-7cvgq" Oct 29 23:07:30.722624 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:30.722095 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:07:31.009482 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:31.009393 5057 patch_prober.go:29] interesting pod/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-2mcsr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body= Oct 29 23:07:31.009482 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:31.009447 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-2mcsr" podUID="e61a681b-48a0-4b15-bea3-013044a07714" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" Oct 29 23:07:31.338923 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:31.338838 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-77238f-5c9c8b447d-ksqsl" Oct 29 23:07:32.009129 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:32.009095 5057 patch_prober.go:29] interesting pod/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-2mcsr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body= Oct 29 23:07:32.009477 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:32.009431 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-2mcsr" podUID="e61a681b-48a0-4b15-bea3-013044a07714" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" Oct 29 23:07:32.626131 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:32.626096 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:07:32.863040 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:32.863002 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:07:33.011385 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:33.006292 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:07:33.011385 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:33.009743 5057 patch_prober.go:29] interesting pod/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-2mcsr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body= Oct 29 23:07:33.011385 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:33.009792 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-2mcsr" podUID="e61a681b-48a0-4b15-bea3-013044a07714" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" Oct 29 23:07:33.228958 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:33.228554 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:07:33.370184 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:33.370091 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:07:34.040465 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:34.040430 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-2mcsr" Oct 29 23:07:34.041027 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:34.040989 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-2mcsr" Oct 29 23:07:34.042008 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:34.041983 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-2mcsr" Oct 29 23:07:34.657864 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:34.657821 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:07:36.535755 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:36.535721 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:07:38.315867 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:38.315833 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:07:40.081776 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:40.081731 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:07:41.713538 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:41.713504 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:07:42.755625 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:42.755590 5057 scope.go:122] "RemoveContainer" containerID="3e3777a8bcef48c4f40a09c17bc4239c0b14bf4fac12cd958da3563e7e5ba814" Oct 29 23:07:43.349023 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:43.348983 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:07:45.406797 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:45.406750 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:07:47.008453 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:47.008413 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:07:48.351370 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:48.351289 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:07:48.999247 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:48.999197 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:07:49.039448 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:07:49.039409 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d2dnn" containerName="extractor" Oct 29 23:07:49.170084 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:49.170054 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:07:49.462969 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:49.462932 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:07:49.754119 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:49.754033 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:07:50.498133 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:50.498095 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v2tfm"] Oct 29 23:07:51.540793 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:51.540757 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:07:51.653943 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:51.653910 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-1b7950-684586b68c-lr9zc"] Oct 29 23:07:51.654163 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:51.654135 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1b7950-684586b68c-lr9zc" podUID="8d30e7e8-5647-43a3-b15f-074537fa73ba" containerName="payload-pre-processing" containerID="cri-o://6141a134d935be8ad74da349070fe665637f358449e01974ca83ff4883b3e8a4" gracePeriod=30 Oct 29 23:07:51.705230 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:51.705147 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:07:51.805553 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:51.805465 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-nz2fk"] Oct 29 23:07:51.805746 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:51.805718 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-nz2fk" podUID="03a26d04-61f6-4e70-b52d-f6263220a95d" containerName="istio-proxy" containerID="cri-o://b8576a0b55136d72e23dc28a90605488773503d9d1eb5adf4ed25f1f89274634" gracePeriod=30 Oct 29 23:07:51.826424 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:51.826394 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:07:52.167053 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:52.153893 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:07:52.206903 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:52.206873 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1b7950-684586b68c-lr9zc" Oct 29 23:07:52.280829 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:52.280793 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:07:52.344155 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:52.344061 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d30e7e8-5647-43a3-b15f-074537fa73ba-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d30e7e8-5647-43a3-b15f-074537fa73ba-plugins-config-volume\") pod \"8d30e7e8-5647-43a3-b15f-074537fa73ba\" (UID: \"8d30e7e8-5647-43a3-b15f-074537fa73ba\") " Oct 29 23:07:52.344341 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:52.344311 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d30e7e8-5647-43a3-b15f-074537fa73ba-kube-api-access-l4b4h\" (UniqueName: \"kubernetes.io/projected/8d30e7e8-5647-43a3-b15f-074537fa73ba-kube-api-access-l4b4h\") pod \"8d30e7e8-5647-43a3-b15f-074537fa73ba\" (UID: \"8d30e7e8-5647-43a3-b15f-074537fa73ba\") " Oct 29 23:07:52.344810 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:52.344783 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d30e7e8-5647-43a3-b15f-074537fa73ba-plugins-config-volume" pod "8d30e7e8-5647-43a3-b15f-074537fa73ba" (UID: "8d30e7e8-5647-43a3-b15f-074537fa73ba"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:07:52.346715 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:52.346691 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d30e7e8-5647-43a3-b15f-074537fa73ba-kube-api-access-l4b4h" pod "8d30e7e8-5647-43a3-b15f-074537fa73ba" (UID: "8d30e7e8-5647-43a3-b15f-074537fa73ba"). InnerVolumeSpecName "kube-api-access-l4b4h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:07:52.389286 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:52.389248 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:07:52.446646 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:52.446614 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l4b4h\" (UniqueName: \"kubernetes.io/projected/8d30e7e8-5647-43a3-b15f-074537fa73ba-kube-api-access-l4b4h\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:07:52.446646 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:52.446646 5057 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d30e7e8-5647-43a3-b15f-074537fa73ba-plugins-config-volume\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:07:52.750577 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:52.750546 5057 generic.go:356] "Generic (PLEG): container finished" podID="8d30e7e8-5647-43a3-b15f-074537fa73ba" containerID="6141a134d935be8ad74da349070fe665637f358449e01974ca83ff4883b3e8a4" exitCode=0 Oct 29 23:07:52.751010 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:52.750586 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1b7950-684586b68c-lr9zc" event={"ID":"8d30e7e8-5647-43a3-b15f-074537fa73ba","Type":"ContainerDied","Data":"6141a134d935be8ad74da349070fe665637f358449e01974ca83ff4883b3e8a4"} Oct 29 23:07:52.751010 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:52.750608 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1b7950-684586b68c-lr9zc" Oct 29 23:07:52.751010 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:52.750644 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1b7950-684586b68c-lr9zc" event={"ID":"8d30e7e8-5647-43a3-b15f-074537fa73ba","Type":"ContainerDied","Data":"7e2735cfa58425dc3bf0d909890e5be63f136dd5654677c6d8e14aeb002b27a4"} Oct 29 23:07:52.751010 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:52.750670 5057 scope.go:122] "RemoveContainer" containerID="6141a134d935be8ad74da349070fe665637f358449e01974ca83ff4883b3e8a4" Oct 29 23:07:52.762449 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:52.762432 5057 scope.go:122] "RemoveContainer" containerID="6141a134d935be8ad74da349070fe665637f358449e01974ca83ff4883b3e8a4" Oct 29 23:07:52.762713 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:07:52.762690 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6141a134d935be8ad74da349070fe665637f358449e01974ca83ff4883b3e8a4\": container with ID starting with 6141a134d935be8ad74da349070fe665637f358449e01974ca83ff4883b3e8a4 not found: ID does not exist" containerID="6141a134d935be8ad74da349070fe665637f358449e01974ca83ff4883b3e8a4" Oct 29 23:07:52.762784 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:52.762726 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6141a134d935be8ad74da349070fe665637f358449e01974ca83ff4883b3e8a4"} err="failed to get container status \"6141a134d935be8ad74da349070fe665637f358449e01974ca83ff4883b3e8a4\": rpc error: code = NotFound desc = could not find container \"6141a134d935be8ad74da349070fe665637f358449e01974ca83ff4883b3e8a4\": container with ID starting with 6141a134d935be8ad74da349070fe665637f358449e01974ca83ff4883b3e8a4 not found: ID does not exist" Oct 29 23:07:52.799402 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:52.799372 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-1b7950-684586b68c-lr9zc"] Oct 29 23:07:52.808213 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:52.808189 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-1b7950-684586b68c-lr9zc"] Oct 29 23:07:52.866629 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:52.866598 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:07:53.330439 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:53.330405 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d30e7e8-5647-43a3-b15f-074537fa73ba" path="/var/lib/kubelet/pods/8d30e7e8-5647-43a3-b15f-074537fa73ba/volumes" Oct 29 23:07:54.013177 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:54.013138 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:07:54.473767 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:54.473732 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:07:54.600432 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:54.599545 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:07:54.787917 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:54.787822 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:07:54.900572 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:54.900538 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:07:55.020766 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:55.020725 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:07:55.133004 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:55.132920 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:07:55.492041 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:55.492003 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:07:56.935518 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:56.935482 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:07:57.149240 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:57.149213 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-nz2fk" Oct 29 23:07:57.160643 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:57.160617 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03a26d04-61f6-4e70-b52d-f6263220a95d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03a26d04-61f6-4e70-b52d-f6263220a95d-istio-envoy\") pod \"03a26d04-61f6-4e70-b52d-f6263220a95d\" (UID: \"03a26d04-61f6-4e70-b52d-f6263220a95d\") " Oct 29 23:07:57.160758 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:57.160668 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03a26d04-61f6-4e70-b52d-f6263220a95d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03a26d04-61f6-4e70-b52d-f6263220a95d-credential-socket\") pod \"03a26d04-61f6-4e70-b52d-f6263220a95d\" (UID: \"03a26d04-61f6-4e70-b52d-f6263220a95d\") " Oct 29 23:07:57.160758 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:57.160706 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03a26d04-61f6-4e70-b52d-f6263220a95d-kube-api-access-8x2bd\" (UniqueName: \"kubernetes.io/projected/03a26d04-61f6-4e70-b52d-f6263220a95d-kube-api-access-8x2bd\") pod \"03a26d04-61f6-4e70-b52d-f6263220a95d\" (UID: \"03a26d04-61f6-4e70-b52d-f6263220a95d\") " Oct 29 23:07:57.160758 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:57.160734 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/03a26d04-61f6-4e70-b52d-f6263220a95d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03a26d04-61f6-4e70-b52d-f6263220a95d-istiod-ca-cert\") pod \"03a26d04-61f6-4e70-b52d-f6263220a95d\" (UID: \"03a26d04-61f6-4e70-b52d-f6263220a95d\") " Oct 29 23:07:57.160943 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:57.160845 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03a26d04-61f6-4e70-b52d-f6263220a95d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03a26d04-61f6-4e70-b52d-f6263220a95d-workload-certs\") pod \"03a26d04-61f6-4e70-b52d-f6263220a95d\" (UID: \"03a26d04-61f6-4e70-b52d-f6263220a95d\") " Oct 29 23:07:57.160943 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:57.160893 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03a26d04-61f6-4e70-b52d-f6263220a95d-istio-token\" (UniqueName: \"kubernetes.io/projected/03a26d04-61f6-4e70-b52d-f6263220a95d-istio-token\") pod \"03a26d04-61f6-4e70-b52d-f6263220a95d\" (UID: \"03a26d04-61f6-4e70-b52d-f6263220a95d\") " Oct 29 23:07:57.161052 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:57.161008 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03a26d04-61f6-4e70-b52d-f6263220a95d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03a26d04-61f6-4e70-b52d-f6263220a95d-workload-socket\") pod \"03a26d04-61f6-4e70-b52d-f6263220a95d\" (UID: \"03a26d04-61f6-4e70-b52d-f6263220a95d\") " Oct 29 23:07:57.161052 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:57.161016 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03a26d04-61f6-4e70-b52d-f6263220a95d-credential-socket" pod "03a26d04-61f6-4e70-b52d-f6263220a95d" (UID: "03a26d04-61f6-4e70-b52d-f6263220a95d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:07:57.161151 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:57.161086 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/03a26d04-61f6-4e70-b52d-f6263220a95d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03a26d04-61f6-4e70-b52d-f6263220a95d-istio-podinfo\") pod \"03a26d04-61f6-4e70-b52d-f6263220a95d\" (UID: \"03a26d04-61f6-4e70-b52d-f6263220a95d\") " Oct 29 23:07:57.161151 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:57.161120 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03a26d04-61f6-4e70-b52d-f6263220a95d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03a26d04-61f6-4e70-b52d-f6263220a95d-istio-data\") pod \"03a26d04-61f6-4e70-b52d-f6263220a95d\" (UID: \"03a26d04-61f6-4e70-b52d-f6263220a95d\") " Oct 29 23:07:57.161266 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:57.161180 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03a26d04-61f6-4e70-b52d-f6263220a95d-workload-certs" pod "03a26d04-61f6-4e70-b52d-f6263220a95d" (UID: "03a26d04-61f6-4e70-b52d-f6263220a95d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:07:57.161266 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:57.161183 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03a26d04-61f6-4e70-b52d-f6263220a95d-istiod-ca-cert" pod "03a26d04-61f6-4e70-b52d-f6263220a95d" (UID: "03a26d04-61f6-4e70-b52d-f6263220a95d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:07:57.161387 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:57.161308 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03a26d04-61f6-4e70-b52d-f6263220a95d-workload-socket" pod "03a26d04-61f6-4e70-b52d-f6263220a95d" (UID: "03a26d04-61f6-4e70-b52d-f6263220a95d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:07:57.161488 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:57.161457 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03a26d04-61f6-4e70-b52d-f6263220a95d-istio-data" pod "03a26d04-61f6-4e70-b52d-f6263220a95d" (UID: "03a26d04-61f6-4e70-b52d-f6263220a95d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:07:57.162032 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:57.162007 5057 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03a26d04-61f6-4e70-b52d-f6263220a95d-workload-socket\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:07:57.162155 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:57.162036 5057 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03a26d04-61f6-4e70-b52d-f6263220a95d-istio-data\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:07:57.162155 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:57.162052 5057 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03a26d04-61f6-4e70-b52d-f6263220a95d-credential-socket\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:07:57.162155 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:57.162069 5057 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03a26d04-61f6-4e70-b52d-f6263220a95d-istiod-ca-cert\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:07:57.162155 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:57.162083 5057 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03a26d04-61f6-4e70-b52d-f6263220a95d-workload-certs\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:07:57.162898 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:57.162871 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03a26d04-61f6-4e70-b52d-f6263220a95d-istio-envoy" pod "03a26d04-61f6-4e70-b52d-f6263220a95d" (UID: "03a26d04-61f6-4e70-b52d-f6263220a95d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:07:57.163178 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:57.163156 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03a26d04-61f6-4e70-b52d-f6263220a95d-istio-token" pod "03a26d04-61f6-4e70-b52d-f6263220a95d" (UID: "03a26d04-61f6-4e70-b52d-f6263220a95d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:07:57.163270 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:57.163189 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03a26d04-61f6-4e70-b52d-f6263220a95d-kube-api-access-8x2bd" pod "03a26d04-61f6-4e70-b52d-f6263220a95d" (UID: "03a26d04-61f6-4e70-b52d-f6263220a95d"). InnerVolumeSpecName "kube-api-access-8x2bd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:07:57.163477 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:57.163458 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/03a26d04-61f6-4e70-b52d-f6263220a95d-istio-podinfo" pod "03a26d04-61f6-4e70-b52d-f6263220a95d" (UID: "03a26d04-61f6-4e70-b52d-f6263220a95d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:07:57.263619 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:57.263582 5057 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03a26d04-61f6-4e70-b52d-f6263220a95d-istio-podinfo\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:07:57.263619 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:57.263620 5057 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03a26d04-61f6-4e70-b52d-f6263220a95d-istio-envoy\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:07:57.263898 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:57.263642 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8x2bd\" (UniqueName: \"kubernetes.io/projected/03a26d04-61f6-4e70-b52d-f6263220a95d-kube-api-access-8x2bd\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:07:57.263898 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:57.263657 5057 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03a26d04-61f6-4e70-b52d-f6263220a95d-istio-token\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:07:57.797884 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:57.797850 5057 generic.go:356] "Generic (PLEG): container finished" podID="03a26d04-61f6-4e70-b52d-f6263220a95d" containerID="b8576a0b55136d72e23dc28a90605488773503d9d1eb5adf4ed25f1f89274634" exitCode=0 Oct 29 23:07:57.798065 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:57.797913 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-nz2fk" Oct 29 23:07:57.798065 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:57.797926 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-nz2fk" event={"ID":"03a26d04-61f6-4e70-b52d-f6263220a95d","Type":"ContainerDied","Data":"b8576a0b55136d72e23dc28a90605488773503d9d1eb5adf4ed25f1f89274634"} Oct 29 23:07:57.798065 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:57.797983 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-nz2fk" event={"ID":"03a26d04-61f6-4e70-b52d-f6263220a95d","Type":"ContainerDied","Data":"ebe25b768396567f330019db6bf23634aa7d9f09a375d8764159eebe3faa7b89"} Oct 29 23:07:57.798065 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:57.798006 5057 scope.go:122] "RemoveContainer" containerID="b8576a0b55136d72e23dc28a90605488773503d9d1eb5adf4ed25f1f89274634" Oct 29 23:07:57.809400 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:57.809380 5057 scope.go:122] "RemoveContainer" containerID="b8576a0b55136d72e23dc28a90605488773503d9d1eb5adf4ed25f1f89274634" Oct 29 23:07:57.809685 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:07:57.809664 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8576a0b55136d72e23dc28a90605488773503d9d1eb5adf4ed25f1f89274634\": container with ID starting with b8576a0b55136d72e23dc28a90605488773503d9d1eb5adf4ed25f1f89274634 not found: ID does not exist" containerID="b8576a0b55136d72e23dc28a90605488773503d9d1eb5adf4ed25f1f89274634" Oct 29 23:07:57.809737 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:57.809692 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8576a0b55136d72e23dc28a90605488773503d9d1eb5adf4ed25f1f89274634"} err="failed to get container status \"b8576a0b55136d72e23dc28a90605488773503d9d1eb5adf4ed25f1f89274634\": rpc error: code = NotFound desc = could not find container \"b8576a0b55136d72e23dc28a90605488773503d9d1eb5adf4ed25f1f89274634\": container with ID starting with b8576a0b55136d72e23dc28a90605488773503d9d1eb5adf4ed25f1f89274634 not found: ID does not exist" Oct 29 23:07:57.859989 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:57.859962 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-nz2fk"] Oct 29 23:07:57.868089 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:57.868039 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-nz2fk"] Oct 29 23:07:58.602267 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:58.602229 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:07:59.330692 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:59.330660 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03a26d04-61f6-4e70-b52d-f6263220a95d" path="/var/lib/kubelet/pods/03a26d04-61f6-4e70-b52d-f6263220a95d/volumes" Oct 29 23:07:59.405146 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:59.405118 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-987xq"] Oct 29 23:07:59.405316 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:59.405286 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-987xq" podUID="21dce295-63ce-4d61-8cfa-86669417b525" containerName="istio-proxy" containerID="cri-o://e58c63cf8d89bd25e95ba2b9afb495e0840adae3c8ef59a7d9dfce95d9fadf59" gracePeriod=30 Oct 29 23:07:59.846119 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:07:59.846071 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:08:00.001443 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:00.001402 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:08:00.170096 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:00.170016 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-z2jqh"] Oct 29 23:08:00.170469 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:00.170384 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-z2jqh" podUID="7e2e4597-f643-42fb-a968-1dd7f39407fe" containerName="istio-proxy" containerID="cri-o://c6589eeaa1d8ca92893c11400a4be947aa88b506dc64ace9bf3c6b7aa841ddfb" gracePeriod=30 Oct 29 23:08:00.214948 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:00.214903 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:08:00.345741 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:00.337309 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:08:01.861439 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:01.861404 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:08:01.987637 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:01.987588 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:08:02.104067 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:02.104029 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:08:02.219136 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:02.219100 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:08:04.027715 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:04.027674 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:08:04.726156 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:04.726128 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-987xq" Oct 29 23:08:04.808358 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:04.806031 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21dce295-63ce-4d61-8cfa-86669417b525-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21dce295-63ce-4d61-8cfa-86669417b525-istio-envoy\") pod \"21dce295-63ce-4d61-8cfa-86669417b525\" (UID: \"21dce295-63ce-4d61-8cfa-86669417b525\") " Oct 29 23:08:04.808358 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:04.806080 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21dce295-63ce-4d61-8cfa-86669417b525-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21dce295-63ce-4d61-8cfa-86669417b525-istiod-ca-cert\") pod \"21dce295-63ce-4d61-8cfa-86669417b525\" (UID: \"21dce295-63ce-4d61-8cfa-86669417b525\") " Oct 29 23:08:04.808358 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:04.806253 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21dce295-63ce-4d61-8cfa-86669417b525-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21dce295-63ce-4d61-8cfa-86669417b525-workload-socket\") pod \"21dce295-63ce-4d61-8cfa-86669417b525\" (UID: \"21dce295-63ce-4d61-8cfa-86669417b525\") " Oct 29 23:08:04.808358 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:04.806307 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21dce295-63ce-4d61-8cfa-86669417b525-kube-api-access-9pkxv\" (UniqueName: \"kubernetes.io/projected/21dce295-63ce-4d61-8cfa-86669417b525-kube-api-access-9pkxv\") pod \"21dce295-63ce-4d61-8cfa-86669417b525\" (UID: \"21dce295-63ce-4d61-8cfa-86669417b525\") " Oct 29 23:08:04.808358 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:04.806547 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21dce295-63ce-4d61-8cfa-86669417b525-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21dce295-63ce-4d61-8cfa-86669417b525-istio-data\") pod \"21dce295-63ce-4d61-8cfa-86669417b525\" (UID: \"21dce295-63ce-4d61-8cfa-86669417b525\") " Oct 29 23:08:04.808358 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:04.806625 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21dce295-63ce-4d61-8cfa-86669417b525-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21dce295-63ce-4d61-8cfa-86669417b525-credential-socket\") pod \"21dce295-63ce-4d61-8cfa-86669417b525\" (UID: \"21dce295-63ce-4d61-8cfa-86669417b525\") " Oct 29 23:08:04.808358 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:04.806668 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/21dce295-63ce-4d61-8cfa-86669417b525-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21dce295-63ce-4d61-8cfa-86669417b525-istio-podinfo\") pod \"21dce295-63ce-4d61-8cfa-86669417b525\" (UID: \"21dce295-63ce-4d61-8cfa-86669417b525\") " Oct 29 23:08:04.808358 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:04.806726 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21dce295-63ce-4d61-8cfa-86669417b525-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21dce295-63ce-4d61-8cfa-86669417b525-workload-certs\") pod \"21dce295-63ce-4d61-8cfa-86669417b525\" (UID: \"21dce295-63ce-4d61-8cfa-86669417b525\") " Oct 29 23:08:04.808358 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:04.806775 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21dce295-63ce-4d61-8cfa-86669417b525-istio-token\" (UniqueName: \"kubernetes.io/projected/21dce295-63ce-4d61-8cfa-86669417b525-istio-token\") pod \"21dce295-63ce-4d61-8cfa-86669417b525\" (UID: \"21dce295-63ce-4d61-8cfa-86669417b525\") " Oct 29 23:08:04.821323 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:04.821276 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21dce295-63ce-4d61-8cfa-86669417b525-istio-data" pod "21dce295-63ce-4d61-8cfa-86669417b525" (UID: "21dce295-63ce-4d61-8cfa-86669417b525"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:08:04.821772 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:04.821753 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21dce295-63ce-4d61-8cfa-86669417b525-credential-socket" pod "21dce295-63ce-4d61-8cfa-86669417b525" (UID: "21dce295-63ce-4d61-8cfa-86669417b525"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:08:04.822136 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:04.822112 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21dce295-63ce-4d61-8cfa-86669417b525-workload-certs" pod "21dce295-63ce-4d61-8cfa-86669417b525" (UID: "21dce295-63ce-4d61-8cfa-86669417b525"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:08:04.822326 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:04.822307 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21dce295-63ce-4d61-8cfa-86669417b525-istiod-ca-cert" pod "21dce295-63ce-4d61-8cfa-86669417b525" (UID: "21dce295-63ce-4d61-8cfa-86669417b525"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:08:04.822626 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:04.822600 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21dce295-63ce-4d61-8cfa-86669417b525-workload-socket" pod "21dce295-63ce-4d61-8cfa-86669417b525" (UID: "21dce295-63ce-4d61-8cfa-86669417b525"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:08:04.823462 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:04.823438 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21dce295-63ce-4d61-8cfa-86669417b525-istio-token" pod "21dce295-63ce-4d61-8cfa-86669417b525" (UID: "21dce295-63ce-4d61-8cfa-86669417b525"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:08:04.825280 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:04.825245 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21dce295-63ce-4d61-8cfa-86669417b525-kube-api-access-9pkxv" pod "21dce295-63ce-4d61-8cfa-86669417b525" (UID: "21dce295-63ce-4d61-8cfa-86669417b525"). InnerVolumeSpecName "kube-api-access-9pkxv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:08:04.827368 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:04.827308 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21dce295-63ce-4d61-8cfa-86669417b525-istio-envoy" pod "21dce295-63ce-4d61-8cfa-86669417b525" (UID: "21dce295-63ce-4d61-8cfa-86669417b525"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:08:04.827670 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:04.827645 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/21dce295-63ce-4d61-8cfa-86669417b525-istio-podinfo" pod "21dce295-63ce-4d61-8cfa-86669417b525" (UID: "21dce295-63ce-4d61-8cfa-86669417b525"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:08:04.864571 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:04.864543 5057 generic.go:356] "Generic (PLEG): container finished" podID="21dce295-63ce-4d61-8cfa-86669417b525" containerID="e58c63cf8d89bd25e95ba2b9afb495e0840adae3c8ef59a7d9dfce95d9fadf59" exitCode=0 Oct 29 23:08:04.864696 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:04.864591 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-987xq" Oct 29 23:08:04.864696 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:04.864631 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-987xq" event={"ID":"21dce295-63ce-4d61-8cfa-86669417b525","Type":"ContainerDied","Data":"e58c63cf8d89bd25e95ba2b9afb495e0840adae3c8ef59a7d9dfce95d9fadf59"} Oct 29 23:08:04.864841 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:04.864693 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-987xq" event={"ID":"21dce295-63ce-4d61-8cfa-86669417b525","Type":"ContainerDied","Data":"4fdefaa28aaf8a1cf1e4257794518cb070ca79cf6526c9f95dade2a987e9f7b4"} Oct 29 23:08:04.864841 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:04.864727 5057 scope.go:122] "RemoveContainer" containerID="e58c63cf8d89bd25e95ba2b9afb495e0840adae3c8ef59a7d9dfce95d9fadf59" Oct 29 23:08:04.875996 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:04.875975 5057 scope.go:122] "RemoveContainer" containerID="e58c63cf8d89bd25e95ba2b9afb495e0840adae3c8ef59a7d9dfce95d9fadf59" Oct 29 23:08:04.876258 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:08:04.876236 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e58c63cf8d89bd25e95ba2b9afb495e0840adae3c8ef59a7d9dfce95d9fadf59\": container with ID starting with e58c63cf8d89bd25e95ba2b9afb495e0840adae3c8ef59a7d9dfce95d9fadf59 not found: ID does not exist" containerID="e58c63cf8d89bd25e95ba2b9afb495e0840adae3c8ef59a7d9dfce95d9fadf59" Oct 29 23:08:04.876344 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:04.876267 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e58c63cf8d89bd25e95ba2b9afb495e0840adae3c8ef59a7d9dfce95d9fadf59"} err="failed to get container status \"e58c63cf8d89bd25e95ba2b9afb495e0840adae3c8ef59a7d9dfce95d9fadf59\": rpc error: code = NotFound desc = could not find container \"e58c63cf8d89bd25e95ba2b9afb495e0840adae3c8ef59a7d9dfce95d9fadf59\": container with ID starting with e58c63cf8d89bd25e95ba2b9afb495e0840adae3c8ef59a7d9dfce95d9fadf59 not found: ID does not exist" Oct 29 23:08:04.909028 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:04.908995 5057 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21dce295-63ce-4d61-8cfa-86669417b525-istio-data\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:08:04.909028 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:04.909028 5057 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21dce295-63ce-4d61-8cfa-86669417b525-credential-socket\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:08:04.909028 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:04.909037 5057 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21dce295-63ce-4d61-8cfa-86669417b525-istio-podinfo\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:08:04.909275 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:04.909047 5057 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21dce295-63ce-4d61-8cfa-86669417b525-workload-certs\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:08:04.909275 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:04.909057 5057 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21dce295-63ce-4d61-8cfa-86669417b525-istio-token\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:08:04.909275 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:04.909064 5057 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21dce295-63ce-4d61-8cfa-86669417b525-istio-envoy\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:08:04.909275 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:04.909073 5057 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21dce295-63ce-4d61-8cfa-86669417b525-istiod-ca-cert\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:08:04.909275 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:04.909083 5057 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21dce295-63ce-4d61-8cfa-86669417b525-workload-socket\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:08:04.909275 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:04.909095 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9pkxv\" (UniqueName: \"kubernetes.io/projected/21dce295-63ce-4d61-8cfa-86669417b525-kube-api-access-9pkxv\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:08:04.916807 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:04.916778 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-987xq"] Oct 29 23:08:04.923376 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:04.923348 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-987xq"] Oct 29 23:08:05.212924 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:05.212896 5057 patch_prober.go:29] interesting pod/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-z2jqh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body= Oct 29 23:08:05.213320 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:05.212965 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-z2jqh" podUID="7e2e4597-f643-42fb-a968-1dd7f39407fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" Oct 29 23:08:05.335836 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:05.335800 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21dce295-63ce-4d61-8cfa-86669417b525" path="/var/lib/kubelet/pods/21dce295-63ce-4d61-8cfa-86669417b525/volumes" Oct 29 23:08:05.479720 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:05.479688 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-z2jqh" Oct 29 23:08:05.621931 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:05.621900 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e2e4597-f643-42fb-a968-1dd7f39407fe-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e2e4597-f643-42fb-a968-1dd7f39407fe-workload-certs\") pod \"7e2e4597-f643-42fb-a968-1dd7f39407fe\" (UID: \"7e2e4597-f643-42fb-a968-1dd7f39407fe\") " Oct 29 23:08:05.622104 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:05.622052 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e2e4597-f643-42fb-a968-1dd7f39407fe-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e2e4597-f643-42fb-a968-1dd7f39407fe-istio-data\") pod \"7e2e4597-f643-42fb-a968-1dd7f39407fe\" (UID: \"7e2e4597-f643-42fb-a968-1dd7f39407fe\") " Oct 29 23:08:05.622182 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:05.622113 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e2e4597-f643-42fb-a968-1dd7f39407fe-kube-api-access-jq6vk\" (UniqueName: \"kubernetes.io/projected/7e2e4597-f643-42fb-a968-1dd7f39407fe-kube-api-access-jq6vk\") pod \"7e2e4597-f643-42fb-a968-1dd7f39407fe\" (UID: \"7e2e4597-f643-42fb-a968-1dd7f39407fe\") " Oct 29 23:08:05.622182 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:05.622167 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e2e4597-f643-42fb-a968-1dd7f39407fe-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e2e4597-f643-42fb-a968-1dd7f39407fe-credential-socket\") pod \"7e2e4597-f643-42fb-a968-1dd7f39407fe\" (UID: \"7e2e4597-f643-42fb-a968-1dd7f39407fe\") " Oct 29 23:08:05.622291 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:05.622215 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e2e4597-f643-42fb-a968-1dd7f39407fe-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e2e4597-f643-42fb-a968-1dd7f39407fe-workload-socket\") pod \"7e2e4597-f643-42fb-a968-1dd7f39407fe\" (UID: \"7e2e4597-f643-42fb-a968-1dd7f39407fe\") " Oct 29 23:08:05.622400 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:05.622377 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e2e4597-f643-42fb-a968-1dd7f39407fe-istio-token\" (UniqueName: \"kubernetes.io/projected/7e2e4597-f643-42fb-a968-1dd7f39407fe-istio-token\") pod \"7e2e4597-f643-42fb-a968-1dd7f39407fe\" (UID: \"7e2e4597-f643-42fb-a968-1dd7f39407fe\") " Oct 29 23:08:05.623017 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:05.622480 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e2e4597-f643-42fb-a968-1dd7f39407fe-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e2e4597-f643-42fb-a968-1dd7f39407fe-istiod-ca-cert\") pod \"7e2e4597-f643-42fb-a968-1dd7f39407fe\" (UID: \"7e2e4597-f643-42fb-a968-1dd7f39407fe\") " Oct 29 23:08:05.623017 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:05.622535 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e2e4597-f643-42fb-a968-1dd7f39407fe-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e2e4597-f643-42fb-a968-1dd7f39407fe-istio-envoy\") pod \"7e2e4597-f643-42fb-a968-1dd7f39407fe\" (UID: \"7e2e4597-f643-42fb-a968-1dd7f39407fe\") " Oct 29 23:08:05.623017 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:05.622581 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7e2e4597-f643-42fb-a968-1dd7f39407fe-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e2e4597-f643-42fb-a968-1dd7f39407fe-istio-podinfo\") pod \"7e2e4597-f643-42fb-a968-1dd7f39407fe\" (UID: \"7e2e4597-f643-42fb-a968-1dd7f39407fe\") " Oct 29 23:08:05.623017 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:05.622701 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e2e4597-f643-42fb-a968-1dd7f39407fe-workload-certs" pod "7e2e4597-f643-42fb-a968-1dd7f39407fe" (UID: "7e2e4597-f643-42fb-a968-1dd7f39407fe"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:08:05.623492 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:05.623415 5057 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e2e4597-f643-42fb-a968-1dd7f39407fe-workload-certs\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:08:05.623695 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:05.623633 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e2e4597-f643-42fb-a968-1dd7f39407fe-istio-data" pod "7e2e4597-f643-42fb-a968-1dd7f39407fe" (UID: "7e2e4597-f643-42fb-a968-1dd7f39407fe"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:08:05.624109 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:05.624088 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e2e4597-f643-42fb-a968-1dd7f39407fe-istiod-ca-cert" pod "7e2e4597-f643-42fb-a968-1dd7f39407fe" (UID: "7e2e4597-f643-42fb-a968-1dd7f39407fe"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:08:05.624715 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:05.624652 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e2e4597-f643-42fb-a968-1dd7f39407fe-workload-socket" pod "7e2e4597-f643-42fb-a968-1dd7f39407fe" (UID: "7e2e4597-f643-42fb-a968-1dd7f39407fe"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:08:05.624880 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:05.624857 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e2e4597-f643-42fb-a968-1dd7f39407fe-credential-socket" pod "7e2e4597-f643-42fb-a968-1dd7f39407fe" (UID: "7e2e4597-f643-42fb-a968-1dd7f39407fe"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:08:05.625794 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:05.625767 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e2e4597-f643-42fb-a968-1dd7f39407fe-kube-api-access-jq6vk" pod "7e2e4597-f643-42fb-a968-1dd7f39407fe" (UID: "7e2e4597-f643-42fb-a968-1dd7f39407fe"). InnerVolumeSpecName "kube-api-access-jq6vk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:08:05.628704 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:05.628668 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e2e4597-f643-42fb-a968-1dd7f39407fe-istio-envoy" pod "7e2e4597-f643-42fb-a968-1dd7f39407fe" (UID: "7e2e4597-f643-42fb-a968-1dd7f39407fe"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:08:05.628800 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:05.628742 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7e2e4597-f643-42fb-a968-1dd7f39407fe-istio-podinfo" pod "7e2e4597-f643-42fb-a968-1dd7f39407fe" (UID: "7e2e4597-f643-42fb-a968-1dd7f39407fe"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:08:05.629060 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:05.629042 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e2e4597-f643-42fb-a968-1dd7f39407fe-istio-token" pod "7e2e4597-f643-42fb-a968-1dd7f39407fe" (UID: "7e2e4597-f643-42fb-a968-1dd7f39407fe"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:08:05.712989 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:05.712955 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:08:05.724717 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:05.724692 5057 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e2e4597-f643-42fb-a968-1dd7f39407fe-istio-data\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:08:05.724825 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:05.724719 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jq6vk\" (UniqueName: \"kubernetes.io/projected/7e2e4597-f643-42fb-a968-1dd7f39407fe-kube-api-access-jq6vk\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:08:05.724825 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:05.724730 5057 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e2e4597-f643-42fb-a968-1dd7f39407fe-credential-socket\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:08:05.724825 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:05.724739 5057 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e2e4597-f643-42fb-a968-1dd7f39407fe-workload-socket\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:08:05.724825 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:05.724747 5057 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e2e4597-f643-42fb-a968-1dd7f39407fe-istio-token\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:08:05.724825 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:05.724756 5057 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e2e4597-f643-42fb-a968-1dd7f39407fe-istiod-ca-cert\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:08:05.724825 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:05.724763 5057 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e2e4597-f643-42fb-a968-1dd7f39407fe-istio-envoy\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:08:05.724825 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:05.724770 5057 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e2e4597-f643-42fb-a968-1dd7f39407fe-istio-podinfo\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:08:05.872993 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:05.872960 5057 generic.go:356] "Generic (PLEG): container finished" podID="7e2e4597-f643-42fb-a968-1dd7f39407fe" containerID="c6589eeaa1d8ca92893c11400a4be947aa88b506dc64ace9bf3c6b7aa841ddfb" exitCode=0 Oct 29 23:08:05.873146 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:05.873013 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-z2jqh" Oct 29 23:08:05.873146 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:05.873045 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-z2jqh" event={"ID":"7e2e4597-f643-42fb-a968-1dd7f39407fe","Type":"ContainerDied","Data":"c6589eeaa1d8ca92893c11400a4be947aa88b506dc64ace9bf3c6b7aa841ddfb"} Oct 29 23:08:05.873146 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:05.873104 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-z2jqh" event={"ID":"7e2e4597-f643-42fb-a968-1dd7f39407fe","Type":"ContainerDied","Data":"f20a73ace8a30370ad1fb811d7ac69189b680bfe2502ce3808cb85251be41ef4"} Oct 29 23:08:05.873146 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:05.873127 5057 scope.go:122] "RemoveContainer" containerID="c6589eeaa1d8ca92893c11400a4be947aa88b506dc64ace9bf3c6b7aa841ddfb" Oct 29 23:08:05.886732 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:05.886707 5057 scope.go:122] "RemoveContainer" containerID="c6589eeaa1d8ca92893c11400a4be947aa88b506dc64ace9bf3c6b7aa841ddfb" Oct 29 23:08:05.887045 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:08:05.887020 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6589eeaa1d8ca92893c11400a4be947aa88b506dc64ace9bf3c6b7aa841ddfb\": container with ID starting with c6589eeaa1d8ca92893c11400a4be947aa88b506dc64ace9bf3c6b7aa841ddfb not found: ID does not exist" containerID="c6589eeaa1d8ca92893c11400a4be947aa88b506dc64ace9bf3c6b7aa841ddfb" Oct 29 23:08:05.887162 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:05.887062 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6589eeaa1d8ca92893c11400a4be947aa88b506dc64ace9bf3c6b7aa841ddfb"} err="failed to get container status \"c6589eeaa1d8ca92893c11400a4be947aa88b506dc64ace9bf3c6b7aa841ddfb\": rpc error: code = NotFound desc = could not find container \"c6589eeaa1d8ca92893c11400a4be947aa88b506dc64ace9bf3c6b7aa841ddfb\": container with ID starting with c6589eeaa1d8ca92893c11400a4be947aa88b506dc64ace9bf3c6b7aa841ddfb not found: ID does not exist" Oct 29 23:08:05.924707 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:05.924666 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-z2jqh"] Oct 29 23:08:05.931754 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:05.931732 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-z2jqh"] Oct 29 23:08:07.330758 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:07.330681 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e2e4597-f643-42fb-a968-1dd7f39407fe" path="/var/lib/kubelet/pods/7e2e4597-f643-42fb-a968-1dd7f39407fe/volumes" Oct 29 23:08:07.424648 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:07.424606 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:08:09.718607 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:09.718558 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:08:11.417231 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:11.417191 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:08:12.767252 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:12.767165 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:08:12.870763 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:12.870722 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c88955-5c45cdbd7f-sdtmr"] Oct 29 23:08:12.870995 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:12.870963 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c88955-5c45cdbd7f-sdtmr" podUID="8c9bed15-70b8-401f-820b-b793e47becd6" containerName="payload-pre-processing" containerID="cri-o://f5de29f403607277e8f853ff5db1666d72e872f617b33eaeef650c38407ece88" gracePeriod=30 Oct 29 23:08:12.933722 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:12.933639 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:08:13.044751 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:13.044668 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:08:13.193571 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:13.193536 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c88955-5c45cdbd7f-sdtmr" Oct 29 23:08:13.254358 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:13.253764 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c9bed15-70b8-401f-820b-b793e47becd6-kube-api-access-srzqh\" (UniqueName: \"kubernetes.io/projected/8c9bed15-70b8-401f-820b-b793e47becd6-kube-api-access-srzqh\") pod \"8c9bed15-70b8-401f-820b-b793e47becd6\" (UID: \"8c9bed15-70b8-401f-820b-b793e47becd6\") " Oct 29 23:08:13.254358 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:13.253892 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c9bed15-70b8-401f-820b-b793e47becd6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c9bed15-70b8-401f-820b-b793e47becd6-plugins-config-volume\") pod \"8c9bed15-70b8-401f-820b-b793e47becd6\" (UID: \"8c9bed15-70b8-401f-820b-b793e47becd6\") " Oct 29 23:08:13.268374 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:13.263609 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c9bed15-70b8-401f-820b-b793e47becd6-plugins-config-volume" pod "8c9bed15-70b8-401f-820b-b793e47becd6" (UID: "8c9bed15-70b8-401f-820b-b793e47becd6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:08:13.289372 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:13.271393 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c9bed15-70b8-401f-820b-b793e47becd6-kube-api-access-srzqh" pod "8c9bed15-70b8-401f-820b-b793e47becd6" (UID: "8c9bed15-70b8-401f-820b-b793e47becd6"). InnerVolumeSpecName "kube-api-access-srzqh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:08:13.289372 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:13.275619 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:08:13.445843 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:13.445811 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-srzqh\" (UniqueName: \"kubernetes.io/projected/8c9bed15-70b8-401f-820b-b793e47becd6-kube-api-access-srzqh\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:08:13.446436 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:13.446412 5057 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c9bed15-70b8-401f-820b-b793e47becd6-plugins-config-volume\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:08:13.485261 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:13.485216 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:08:13.618035 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:13.617946 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:08:13.972508 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:13.966583 5057 generic.go:356] "Generic (PLEG): container finished" podID="8c9bed15-70b8-401f-820b-b793e47becd6" containerID="f5de29f403607277e8f853ff5db1666d72e872f617b33eaeef650c38407ece88" exitCode=0 Oct 29 23:08:13.972508 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:13.966855 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c88955-5c45cdbd7f-sdtmr" event={"ID":"8c9bed15-70b8-401f-820b-b793e47becd6","Type":"ContainerDied","Data":"f5de29f403607277e8f853ff5db1666d72e872f617b33eaeef650c38407ece88"} Oct 29 23:08:13.972508 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:13.966906 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c88955-5c45cdbd7f-sdtmr" event={"ID":"8c9bed15-70b8-401f-820b-b793e47becd6","Type":"ContainerDied","Data":"95517aed5e508ad1a9649c872e8b0c01ae4e13687fece244d72268d345141d5f"} Oct 29 23:08:13.972508 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:13.966939 5057 scope.go:122] "RemoveContainer" containerID="f5de29f403607277e8f853ff5db1666d72e872f617b33eaeef650c38407ece88" Oct 29 23:08:13.972508 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:13.967080 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c88955-5c45cdbd7f-sdtmr" Oct 29 23:08:13.992849 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:13.992821 5057 scope.go:122] "RemoveContainer" containerID="f5de29f403607277e8f853ff5db1666d72e872f617b33eaeef650c38407ece88" Oct 29 23:08:13.993249 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:08:13.993213 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5de29f403607277e8f853ff5db1666d72e872f617b33eaeef650c38407ece88\": container with ID starting with f5de29f403607277e8f853ff5db1666d72e872f617b33eaeef650c38407ece88 not found: ID does not exist" containerID="f5de29f403607277e8f853ff5db1666d72e872f617b33eaeef650c38407ece88" Oct 29 23:08:13.993362 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:13.993268 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5de29f403607277e8f853ff5db1666d72e872f617b33eaeef650c38407ece88"} err="failed to get container status \"f5de29f403607277e8f853ff5db1666d72e872f617b33eaeef650c38407ece88\": rpc error: code = NotFound desc = could not find container \"f5de29f403607277e8f853ff5db1666d72e872f617b33eaeef650c38407ece88\": container with ID starting with f5de29f403607277e8f853ff5db1666d72e872f617b33eaeef650c38407ece88 not found: ID does not exist" Oct 29 23:08:14.023759 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:14.023726 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c88955-5c45cdbd7f-sdtmr"] Oct 29 23:08:14.031944 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:14.031921 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c88955-5c45cdbd7f-sdtmr"] Oct 29 23:08:14.057769 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:14.057740 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:08:15.343351 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:15.337414 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c9bed15-70b8-401f-820b-b793e47becd6" path="/var/lib/kubelet/pods/8c9bed15-70b8-401f-820b-b793e47becd6/volumes" Oct 29 23:08:15.709950 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:15.709910 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:08:17.430175 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:17.430134 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:08:18.476474 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:18.476431 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:08:18.597374 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:18.597276 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:08:19.110036 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:19.110000 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:08:19.115037 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:08:19.114998 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d2dnn" containerName="extractor" Oct 29 23:08:20.617906 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:20.617867 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:08:20.730318 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:20.730277 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:08:20.927221 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:20.926552 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:08:21.040307 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:21.040271 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:08:22.585229 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:22.585187 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:08:24.218782 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:24.218740 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:08:25.572265 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:25.572217 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:08:25.909576 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:25.909494 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:08:25.998264 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:25.998231 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-77238f-5c9c8b447d-ksqsl"] Oct 29 23:08:25.998520 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:25.998492 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-77238f-5c9c8b447d-ksqsl" podUID="27e3d9c6-89a4-42fd-9a98-c030f6a22990" containerName="payload-processing" containerID="cri-o://a2c731e87b3a2c106692ea57c197e10939529d0a4967ee09ae45faa632e50518" gracePeriod=30 Oct 29 23:08:26.034559 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:26.034516 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-77238f-5db9746b9f-7cvgq"] Oct 29 23:08:26.034802 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:26.034775 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-77238f-5db9746b9f-7cvgq" podUID="96d53b6e-441d-411e-aa0c-efcb79e89983" containerName="payload-pre-processing" containerID="cri-o://f325daa6f7d43b2c19c6b89804ce93ded4d488befe209d70e81b8a31671f885f" gracePeriod=30 Oct 29 23:08:26.056520 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:26.056457 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:08:26.172905 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:26.172818 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:08:26.199893 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:26.199856 5057 generic.go:356] "Generic (PLEG): container finished" podID="96d53b6e-441d-411e-aa0c-efcb79e89983" containerID="f325daa6f7d43b2c19c6b89804ce93ded4d488befe209d70e81b8a31671f885f" exitCode=0 Oct 29 23:08:26.200136 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:26.200062 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-77238f-5db9746b9f-7cvgq" event={"ID":"96d53b6e-441d-411e-aa0c-efcb79e89983","Type":"ContainerDied","Data":"f325daa6f7d43b2c19c6b89804ce93ded4d488befe209d70e81b8a31671f885f"} Oct 29 23:08:26.202664 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:26.202534 5057 generic.go:356] "Generic (PLEG): container finished" podID="27e3d9c6-89a4-42fd-9a98-c030f6a22990" containerID="a2c731e87b3a2c106692ea57c197e10939529d0a4967ee09ae45faa632e50518" exitCode=0 Oct 29 23:08:26.202664 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:26.202649 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-77238f-5c9c8b447d-ksqsl" event={"ID":"27e3d9c6-89a4-42fd-9a98-c030f6a22990","Type":"ContainerDied","Data":"a2c731e87b3a2c106692ea57c197e10939529d0a4967ee09ae45faa632e50518"} Oct 29 23:08:26.604927 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:26.604833 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:08:26.654355 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:26.641899 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-77238f-5c9c8b447d-ksqsl" Oct 29 23:08:26.677356 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:26.672896 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-77238f-5db9746b9f-7cvgq" Oct 29 23:08:26.722353 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:26.721092 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:08:26.830791 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:26.830716 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27e3d9c6-89a4-42fd-9a98-c030f6a22990-kube-api-access-cq2gv\" (UniqueName: \"kubernetes.io/projected/27e3d9c6-89a4-42fd-9a98-c030f6a22990-kube-api-access-cq2gv\") pod \"27e3d9c6-89a4-42fd-9a98-c030f6a22990\" (UID: \"27e3d9c6-89a4-42fd-9a98-c030f6a22990\") " Oct 29 23:08:26.830983 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:26.830813 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/96d53b6e-441d-411e-aa0c-efcb79e89983-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96d53b6e-441d-411e-aa0c-efcb79e89983-plugins-config-volume\") pod \"96d53b6e-441d-411e-aa0c-efcb79e89983\" (UID: \"96d53b6e-441d-411e-aa0c-efcb79e89983\") " Oct 29 23:08:26.830983 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:26.830970 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96d53b6e-441d-411e-aa0c-efcb79e89983-kube-api-access-qtgsz\" (UniqueName: \"kubernetes.io/projected/96d53b6e-441d-411e-aa0c-efcb79e89983-kube-api-access-qtgsz\") pod \"96d53b6e-441d-411e-aa0c-efcb79e89983\" (UID: \"96d53b6e-441d-411e-aa0c-efcb79e89983\") " Oct 29 23:08:26.831143 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:26.831129 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27e3d9c6-89a4-42fd-9a98-c030f6a22990-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27e3d9c6-89a4-42fd-9a98-c030f6a22990-plugins-config-volume\") pod \"27e3d9c6-89a4-42fd-9a98-c030f6a22990\" (UID: \"27e3d9c6-89a4-42fd-9a98-c030f6a22990\") " Oct 29 23:08:26.831267 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:26.831236 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96d53b6e-441d-411e-aa0c-efcb79e89983-plugins-config-volume" pod "96d53b6e-441d-411e-aa0c-efcb79e89983" (UID: "96d53b6e-441d-411e-aa0c-efcb79e89983"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:08:26.831608 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:26.831582 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27e3d9c6-89a4-42fd-9a98-c030f6a22990-plugins-config-volume" pod "27e3d9c6-89a4-42fd-9a98-c030f6a22990" (UID: "27e3d9c6-89a4-42fd-9a98-c030f6a22990"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:08:26.831740 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:26.831721 5057 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27e3d9c6-89a4-42fd-9a98-c030f6a22990-plugins-config-volume\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:08:26.831816 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:26.831742 5057 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96d53b6e-441d-411e-aa0c-efcb79e89983-plugins-config-volume\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:08:26.833456 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:26.833434 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27e3d9c6-89a4-42fd-9a98-c030f6a22990-kube-api-access-cq2gv" pod "27e3d9c6-89a4-42fd-9a98-c030f6a22990" (UID: "27e3d9c6-89a4-42fd-9a98-c030f6a22990"). InnerVolumeSpecName "kube-api-access-cq2gv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:08:26.833625 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:26.833605 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96d53b6e-441d-411e-aa0c-efcb79e89983-kube-api-access-qtgsz" pod "96d53b6e-441d-411e-aa0c-efcb79e89983" (UID: "96d53b6e-441d-411e-aa0c-efcb79e89983"). InnerVolumeSpecName "kube-api-access-qtgsz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:08:26.838653 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:26.838625 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:08:26.933691 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:26.933655 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qtgsz\" (UniqueName: \"kubernetes.io/projected/96d53b6e-441d-411e-aa0c-efcb79e89983-kube-api-access-qtgsz\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:08:26.933691 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:26.933686 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cq2gv\" (UniqueName: \"kubernetes.io/projected/27e3d9c6-89a4-42fd-9a98-c030f6a22990-kube-api-access-cq2gv\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:08:27.031964 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:27.031930 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:08:27.214001 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:27.213973 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-77238f-5db9746b9f-7cvgq" Oct 29 23:08:27.214190 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:27.213976 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-77238f-5db9746b9f-7cvgq" event={"ID":"96d53b6e-441d-411e-aa0c-efcb79e89983","Type":"ContainerDied","Data":"3cc7bfc93ffe0084eed517575e4a94bc2441aa34fee6d3479026fc2fd03d0896"} Oct 29 23:08:27.214190 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:27.214137 5057 scope.go:122] "RemoveContainer" containerID="f325daa6f7d43b2c19c6b89804ce93ded4d488befe209d70e81b8a31671f885f" Oct 29 23:08:27.216995 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:27.216957 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-77238f-5c9c8b447d-ksqsl" Oct 29 23:08:27.217139 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:27.216996 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-77238f-5c9c8b447d-ksqsl" event={"ID":"27e3d9c6-89a4-42fd-9a98-c030f6a22990","Type":"ContainerDied","Data":"8a234877436ec56d87b6ecbafd69ab7b984179a5f955c809c0f2c4ab6e699d53"} Oct 29 23:08:27.227587 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:27.227568 5057 scope.go:122] "RemoveContainer" containerID="a2c731e87b3a2c106692ea57c197e10939529d0a4967ee09ae45faa632e50518" Oct 29 23:08:27.273462 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:27.273436 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-77238f-5c9c8b447d-ksqsl"] Oct 29 23:08:27.285116 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:27.285085 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-77238f-5c9c8b447d-ksqsl"] Oct 29 23:08:27.312570 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:27.312543 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-77238f-5db9746b9f-7cvgq"] Oct 29 23:08:27.317229 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:27.317205 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-77238f-5db9746b9f-7cvgq"] Oct 29 23:08:27.332941 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:27.332915 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27e3d9c6-89a4-42fd-9a98-c030f6a22990" path="/var/lib/kubelet/pods/27e3d9c6-89a4-42fd-9a98-c030f6a22990/volumes" Oct 29 23:08:27.333483 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:27.333467 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96d53b6e-441d-411e-aa0c-efcb79e89983" path="/var/lib/kubelet/pods/96d53b6e-441d-411e-aa0c-efcb79e89983/volumes" Oct 29 23:08:27.373638 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:27.373608 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:08:28.939661 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:28.939618 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:08:30.717531 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:30.717491 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:08:31.787363 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:31.787202 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:08:32.187726 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:32.187636 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:08:32.509837 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:32.509789 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:08:33.375852 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:33.375788 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:08:33.488571 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:33.488531 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:08:33.603459 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:33.603427 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:08:33.722070 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:33.722024 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:08:34.186988 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:34.186901 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:08:34.299357 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:34.299310 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:08:34.422606 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:34.422569 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:08:34.515804 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:34.515762 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-2mcsr"] Oct 29 23:08:34.516070 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:34.516046 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-2mcsr" podUID="e61a681b-48a0-4b15-bea3-013044a07714" containerName="istio-proxy" containerID="cri-o://13c8406382cc22f056158c48a1139c2274b7aef44fe6a7b5d36231960ba8f688" gracePeriod=30 Oct 29 23:08:34.539226 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:34.539191 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-k5t2b"] Oct 29 23:08:34.539456 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:34.539407 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-k5t2b" podUID="f8dc3f31-0a4c-44e6-a3e9-4c45e238d1bc" containerName="istio-proxy" containerID="cri-o://4ed085418f0efde7366ddf939dc16698831a8396b9388731e4e729ab98cc681c" gracePeriod=30 Oct 29 23:08:34.547105 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:34.547057 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:08:36.121700 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:36.121664 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:08:37.773709 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:37.773671 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:08:37.908154 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:37.908111 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:08:38.028084 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:38.028003 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:08:38.262522 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:38.262479 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:08:39.699385 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:39.699306 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-vj6md_aa6a190e-3283-458a-9dfd-2de7cb193fc5/console-operator/2.log" Oct 29 23:08:39.726139 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:39.726106 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-vj6md_aa6a190e-3283-458a-9dfd-2de7cb193fc5/console-operator/2.log" Oct 29 23:08:39.762564 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:39.762375 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:08:39.902282 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:39.902248 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-k5t2b" Oct 29 23:08:39.919917 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:39.913054 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8dc3f31-0a4c-44e6-a3e9-4c45e238d1bc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8dc3f31-0a4c-44e6-a3e9-4c45e238d1bc-istio-envoy\") pod \"f8dc3f31-0a4c-44e6-a3e9-4c45e238d1bc\" (UID: \"f8dc3f31-0a4c-44e6-a3e9-4c45e238d1bc\") " Oct 29 23:08:39.919917 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:39.913154 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8dc3f31-0a4c-44e6-a3e9-4c45e238d1bc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8dc3f31-0a4c-44e6-a3e9-4c45e238d1bc-workload-socket\") pod \"f8dc3f31-0a4c-44e6-a3e9-4c45e238d1bc\" (UID: \"f8dc3f31-0a4c-44e6-a3e9-4c45e238d1bc\") " Oct 29 23:08:39.919917 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:39.913232 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f8dc3f31-0a4c-44e6-a3e9-4c45e238d1bc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8dc3f31-0a4c-44e6-a3e9-4c45e238d1bc-istio-podinfo\") pod \"f8dc3f31-0a4c-44e6-a3e9-4c45e238d1bc\" (UID: \"f8dc3f31-0a4c-44e6-a3e9-4c45e238d1bc\") " Oct 29 23:08:39.919917 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:39.913355 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f8dc3f31-0a4c-44e6-a3e9-4c45e238d1bc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8dc3f31-0a4c-44e6-a3e9-4c45e238d1bc-istiod-ca-cert\") pod \"f8dc3f31-0a4c-44e6-a3e9-4c45e238d1bc\" (UID: \"f8dc3f31-0a4c-44e6-a3e9-4c45e238d1bc\") " Oct 29 23:08:39.919917 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:39.913411 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8dc3f31-0a4c-44e6-a3e9-4c45e238d1bc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8dc3f31-0a4c-44e6-a3e9-4c45e238d1bc-credential-socket\") pod \"f8dc3f31-0a4c-44e6-a3e9-4c45e238d1bc\" (UID: \"f8dc3f31-0a4c-44e6-a3e9-4c45e238d1bc\") " Oct 29 23:08:39.919917 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:39.913449 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8dc3f31-0a4c-44e6-a3e9-4c45e238d1bc-istio-token\" (UniqueName: \"kubernetes.io/projected/f8dc3f31-0a4c-44e6-a3e9-4c45e238d1bc-istio-token\") pod \"f8dc3f31-0a4c-44e6-a3e9-4c45e238d1bc\" (UID: \"f8dc3f31-0a4c-44e6-a3e9-4c45e238d1bc\") " Oct 29 23:08:39.919917 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:39.913527 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8dc3f31-0a4c-44e6-a3e9-4c45e238d1bc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8dc3f31-0a4c-44e6-a3e9-4c45e238d1bc-istio-data\") pod \"f8dc3f31-0a4c-44e6-a3e9-4c45e238d1bc\" (UID: \"f8dc3f31-0a4c-44e6-a3e9-4c45e238d1bc\") " Oct 29 23:08:39.919917 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:39.913654 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8dc3f31-0a4c-44e6-a3e9-4c45e238d1bc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8dc3f31-0a4c-44e6-a3e9-4c45e238d1bc-workload-certs\") pod \"f8dc3f31-0a4c-44e6-a3e9-4c45e238d1bc\" (UID: \"f8dc3f31-0a4c-44e6-a3e9-4c45e238d1bc\") " Oct 29 23:08:39.919917 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:39.913789 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8dc3f31-0a4c-44e6-a3e9-4c45e238d1bc-kube-api-access-9xmfv\" (UniqueName: \"kubernetes.io/projected/f8dc3f31-0a4c-44e6-a3e9-4c45e238d1bc-kube-api-access-9xmfv\") pod \"f8dc3f31-0a4c-44e6-a3e9-4c45e238d1bc\" (UID: \"f8dc3f31-0a4c-44e6-a3e9-4c45e238d1bc\") " Oct 29 23:08:39.919917 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:39.914630 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8dc3f31-0a4c-44e6-a3e9-4c45e238d1bc-workload-socket" pod "f8dc3f31-0a4c-44e6-a3e9-4c45e238d1bc" (UID: "f8dc3f31-0a4c-44e6-a3e9-4c45e238d1bc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:08:39.919917 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:39.915480 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8dc3f31-0a4c-44e6-a3e9-4c45e238d1bc-credential-socket" pod "f8dc3f31-0a4c-44e6-a3e9-4c45e238d1bc" (UID: "f8dc3f31-0a4c-44e6-a3e9-4c45e238d1bc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:08:39.919917 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:39.915872 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8dc3f31-0a4c-44e6-a3e9-4c45e238d1bc-workload-certs" pod "f8dc3f31-0a4c-44e6-a3e9-4c45e238d1bc" (UID: "f8dc3f31-0a4c-44e6-a3e9-4c45e238d1bc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:08:39.919917 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:39.916138 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8dc3f31-0a4c-44e6-a3e9-4c45e238d1bc-istio-data" pod "f8dc3f31-0a4c-44e6-a3e9-4c45e238d1bc" (UID: "f8dc3f31-0a4c-44e6-a3e9-4c45e238d1bc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:08:39.919917 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:39.917361 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8dc3f31-0a4c-44e6-a3e9-4c45e238d1bc-istiod-ca-cert" pod "f8dc3f31-0a4c-44e6-a3e9-4c45e238d1bc" (UID: "f8dc3f31-0a4c-44e6-a3e9-4c45e238d1bc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:08:39.920867 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:39.920717 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8dc3f31-0a4c-44e6-a3e9-4c45e238d1bc-istio-envoy" pod "f8dc3f31-0a4c-44e6-a3e9-4c45e238d1bc" (UID: "f8dc3f31-0a4c-44e6-a3e9-4c45e238d1bc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:08:39.920867 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:39.920756 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8dc3f31-0a4c-44e6-a3e9-4c45e238d1bc-kube-api-access-9xmfv" pod "f8dc3f31-0a4c-44e6-a3e9-4c45e238d1bc" (UID: "f8dc3f31-0a4c-44e6-a3e9-4c45e238d1bc"). InnerVolumeSpecName "kube-api-access-9xmfv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:08:39.924459 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:39.922926 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f8dc3f31-0a4c-44e6-a3e9-4c45e238d1bc-istio-podinfo" pod "f8dc3f31-0a4c-44e6-a3e9-4c45e238d1bc" (UID: "f8dc3f31-0a4c-44e6-a3e9-4c45e238d1bc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:08:39.924459 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:39.923020 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8dc3f31-0a4c-44e6-a3e9-4c45e238d1bc-istio-token" pod "f8dc3f31-0a4c-44e6-a3e9-4c45e238d1bc" (UID: "f8dc3f31-0a4c-44e6-a3e9-4c45e238d1bc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:08:39.939239 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:39.939214 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-2mcsr" Oct 29 23:08:40.015720 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:40.015683 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e61a681b-48a0-4b15-bea3-013044a07714-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e61a681b-48a0-4b15-bea3-013044a07714-workload-certs\") pod \"e61a681b-48a0-4b15-bea3-013044a07714\" (UID: \"e61a681b-48a0-4b15-bea3-013044a07714\") " Oct 29 23:08:40.015918 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:40.015804 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e61a681b-48a0-4b15-bea3-013044a07714-kube-api-access-dntns\" (UniqueName: \"kubernetes.io/projected/e61a681b-48a0-4b15-bea3-013044a07714-kube-api-access-dntns\") pod \"e61a681b-48a0-4b15-bea3-013044a07714\" (UID: \"e61a681b-48a0-4b15-bea3-013044a07714\") " Oct 29 23:08:40.015986 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:40.015929 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e61a681b-48a0-4b15-bea3-013044a07714-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e61a681b-48a0-4b15-bea3-013044a07714-credential-socket\") pod \"e61a681b-48a0-4b15-bea3-013044a07714\" (UID: \"e61a681b-48a0-4b15-bea3-013044a07714\") " Oct 29 23:08:40.016045 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:40.016004 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e61a681b-48a0-4b15-bea3-013044a07714-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e61a681b-48a0-4b15-bea3-013044a07714-istio-envoy\") pod \"e61a681b-48a0-4b15-bea3-013044a07714\" (UID: \"e61a681b-48a0-4b15-bea3-013044a07714\") " Oct 29 23:08:40.016100 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:40.016047 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e61a681b-48a0-4b15-bea3-013044a07714-istio-token\" (UniqueName: \"kubernetes.io/projected/e61a681b-48a0-4b15-bea3-013044a07714-istio-token\") pod \"e61a681b-48a0-4b15-bea3-013044a07714\" (UID: \"e61a681b-48a0-4b15-bea3-013044a07714\") " Oct 29 23:08:40.016100 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:40.016075 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e61a681b-48a0-4b15-bea3-013044a07714-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e61a681b-48a0-4b15-bea3-013044a07714-istio-data\") pod \"e61a681b-48a0-4b15-bea3-013044a07714\" (UID: \"e61a681b-48a0-4b15-bea3-013044a07714\") " Oct 29 23:08:40.016246 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:40.016223 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e61a681b-48a0-4b15-bea3-013044a07714-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e61a681b-48a0-4b15-bea3-013044a07714-workload-socket\") pod \"e61a681b-48a0-4b15-bea3-013044a07714\" (UID: \"e61a681b-48a0-4b15-bea3-013044a07714\") " Oct 29 23:08:40.016318 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:40.016265 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e61a681b-48a0-4b15-bea3-013044a07714-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e61a681b-48a0-4b15-bea3-013044a07714-istiod-ca-cert\") pod \"e61a681b-48a0-4b15-bea3-013044a07714\" (UID: \"e61a681b-48a0-4b15-bea3-013044a07714\") " Oct 29 23:08:40.016318 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:40.016314 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e61a681b-48a0-4b15-bea3-013044a07714-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e61a681b-48a0-4b15-bea3-013044a07714-istio-podinfo\") pod \"e61a681b-48a0-4b15-bea3-013044a07714\" (UID: \"e61a681b-48a0-4b15-bea3-013044a07714\") " Oct 29 23:08:40.017032 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:40.017011 5057 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8dc3f31-0a4c-44e6-a3e9-4c45e238d1bc-istio-podinfo\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:08:40.017158 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:40.017037 5057 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8dc3f31-0a4c-44e6-a3e9-4c45e238d1bc-istiod-ca-cert\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:08:40.017158 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:40.017052 5057 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8dc3f31-0a4c-44e6-a3e9-4c45e238d1bc-credential-socket\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:08:40.017158 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:40.017067 5057 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8dc3f31-0a4c-44e6-a3e9-4c45e238d1bc-istio-token\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:08:40.017158 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:40.017083 5057 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8dc3f31-0a4c-44e6-a3e9-4c45e238d1bc-istio-data\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:08:40.017158 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:40.017094 5057 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8dc3f31-0a4c-44e6-a3e9-4c45e238d1bc-workload-certs\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:08:40.017158 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:40.017108 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9xmfv\" (UniqueName: \"kubernetes.io/projected/f8dc3f31-0a4c-44e6-a3e9-4c45e238d1bc-kube-api-access-9xmfv\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:08:40.017158 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:40.017123 5057 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8dc3f31-0a4c-44e6-a3e9-4c45e238d1bc-istio-envoy\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:08:40.017158 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:40.017136 5057 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8dc3f31-0a4c-44e6-a3e9-4c45e238d1bc-workload-socket\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:08:40.018773 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:40.018732 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e61a681b-48a0-4b15-bea3-013044a07714-istiod-ca-cert" pod "e61a681b-48a0-4b15-bea3-013044a07714" (UID: "e61a681b-48a0-4b15-bea3-013044a07714"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:08:40.019000 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:40.018927 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e61a681b-48a0-4b15-bea3-013044a07714-credential-socket" pod "e61a681b-48a0-4b15-bea3-013044a07714" (UID: "e61a681b-48a0-4b15-bea3-013044a07714"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:08:40.019086 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:40.018995 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e61a681b-48a0-4b15-bea3-013044a07714-workload-certs" pod "e61a681b-48a0-4b15-bea3-013044a07714" (UID: "e61a681b-48a0-4b15-bea3-013044a07714"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:08:40.019086 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:40.019042 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e61a681b-48a0-4b15-bea3-013044a07714-workload-socket" pod "e61a681b-48a0-4b15-bea3-013044a07714" (UID: "e61a681b-48a0-4b15-bea3-013044a07714"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:08:40.019460 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:40.019436 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e61a681b-48a0-4b15-bea3-013044a07714-istio-data" pod "e61a681b-48a0-4b15-bea3-013044a07714" (UID: "e61a681b-48a0-4b15-bea3-013044a07714"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:08:40.021460 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:40.021432 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e61a681b-48a0-4b15-bea3-013044a07714-istio-podinfo" pod "e61a681b-48a0-4b15-bea3-013044a07714" (UID: "e61a681b-48a0-4b15-bea3-013044a07714"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:08:40.021460 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:40.021450 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e61a681b-48a0-4b15-bea3-013044a07714-istio-envoy" pod "e61a681b-48a0-4b15-bea3-013044a07714" (UID: "e61a681b-48a0-4b15-bea3-013044a07714"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:08:40.021636 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:40.021504 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e61a681b-48a0-4b15-bea3-013044a07714-kube-api-access-dntns" pod "e61a681b-48a0-4b15-bea3-013044a07714" (UID: "e61a681b-48a0-4b15-bea3-013044a07714"). InnerVolumeSpecName "kube-api-access-dntns". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:08:40.022781 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:40.022756 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e61a681b-48a0-4b15-bea3-013044a07714-istio-token" pod "e61a681b-48a0-4b15-bea3-013044a07714" (UID: "e61a681b-48a0-4b15-bea3-013044a07714"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:08:40.119216 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:40.119132 5057 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e61a681b-48a0-4b15-bea3-013044a07714-workload-certs\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:08:40.119216 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:40.119165 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dntns\" (UniqueName: \"kubernetes.io/projected/e61a681b-48a0-4b15-bea3-013044a07714-kube-api-access-dntns\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:08:40.119216 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:40.119175 5057 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e61a681b-48a0-4b15-bea3-013044a07714-credential-socket\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:08:40.119216 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:40.119185 5057 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e61a681b-48a0-4b15-bea3-013044a07714-istio-envoy\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:08:40.119216 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:40.119194 5057 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e61a681b-48a0-4b15-bea3-013044a07714-istio-token\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:08:40.119216 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:40.119201 5057 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e61a681b-48a0-4b15-bea3-013044a07714-istio-data\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:08:40.119216 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:40.119209 5057 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e61a681b-48a0-4b15-bea3-013044a07714-workload-socket\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:08:40.119216 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:40.119217 5057 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e61a681b-48a0-4b15-bea3-013044a07714-istiod-ca-cert\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:08:40.119216 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:40.119225 5057 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e61a681b-48a0-4b15-bea3-013044a07714-istio-podinfo\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:08:40.336736 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:40.336702 5057 generic.go:356] "Generic (PLEG): container finished" podID="e61a681b-48a0-4b15-bea3-013044a07714" containerID="13c8406382cc22f056158c48a1139c2274b7aef44fe6a7b5d36231960ba8f688" exitCode=0 Oct 29 23:08:40.336939 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:40.336755 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-2mcsr" Oct 29 23:08:40.336939 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:40.336792 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-2mcsr" event={"ID":"e61a681b-48a0-4b15-bea3-013044a07714","Type":"ContainerDied","Data":"13c8406382cc22f056158c48a1139c2274b7aef44fe6a7b5d36231960ba8f688"} Oct 29 23:08:40.336939 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:40.336848 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-2mcsr" event={"ID":"e61a681b-48a0-4b15-bea3-013044a07714","Type":"ContainerDied","Data":"139b73a73512a5b10f5f8f997ee4fe307a47f8054f4696c73210bb0105ae83d2"} Oct 29 23:08:40.336939 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:40.336877 5057 scope.go:122] "RemoveContainer" containerID="13c8406382cc22f056158c48a1139c2274b7aef44fe6a7b5d36231960ba8f688" Oct 29 23:08:40.339117 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:40.339085 5057 generic.go:356] "Generic (PLEG): container finished" podID="f8dc3f31-0a4c-44e6-a3e9-4c45e238d1bc" containerID="4ed085418f0efde7366ddf939dc16698831a8396b9388731e4e729ab98cc681c" exitCode=0 Oct 29 23:08:40.339252 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:40.339122 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-k5t2b" event={"ID":"f8dc3f31-0a4c-44e6-a3e9-4c45e238d1bc","Type":"ContainerDied","Data":"4ed085418f0efde7366ddf939dc16698831a8396b9388731e4e729ab98cc681c"} Oct 29 23:08:40.339252 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:40.339156 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-k5t2b" event={"ID":"f8dc3f31-0a4c-44e6-a3e9-4c45e238d1bc","Type":"ContainerDied","Data":"942b81c58f2a95c946999a14e9d3948ce75d5b61d2ed3c0b0cb7246d53f707c2"} Oct 29 23:08:40.339252 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:40.339160 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-k5t2b" Oct 29 23:08:40.351617 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:40.351601 5057 scope.go:122] "RemoveContainer" containerID="13c8406382cc22f056158c48a1139c2274b7aef44fe6a7b5d36231960ba8f688" Oct 29 23:08:40.351962 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:08:40.351929 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13c8406382cc22f056158c48a1139c2274b7aef44fe6a7b5d36231960ba8f688\": container with ID starting with 13c8406382cc22f056158c48a1139c2274b7aef44fe6a7b5d36231960ba8f688 not found: ID does not exist" containerID="13c8406382cc22f056158c48a1139c2274b7aef44fe6a7b5d36231960ba8f688" Oct 29 23:08:40.352106 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:40.351965 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13c8406382cc22f056158c48a1139c2274b7aef44fe6a7b5d36231960ba8f688"} err="failed to get container status \"13c8406382cc22f056158c48a1139c2274b7aef44fe6a7b5d36231960ba8f688\": rpc error: code = NotFound desc = could not find container \"13c8406382cc22f056158c48a1139c2274b7aef44fe6a7b5d36231960ba8f688\": container with ID starting with 13c8406382cc22f056158c48a1139c2274b7aef44fe6a7b5d36231960ba8f688 not found: ID does not exist" Oct 29 23:08:40.352106 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:40.351990 5057 scope.go:122] "RemoveContainer" containerID="4ed085418f0efde7366ddf939dc16698831a8396b9388731e4e729ab98cc681c" Oct 29 23:08:40.362469 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:40.362449 5057 scope.go:122] "RemoveContainer" containerID="4ed085418f0efde7366ddf939dc16698831a8396b9388731e4e729ab98cc681c" Oct 29 23:08:40.362761 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:08:40.362732 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ed085418f0efde7366ddf939dc16698831a8396b9388731e4e729ab98cc681c\": container with ID starting with 4ed085418f0efde7366ddf939dc16698831a8396b9388731e4e729ab98cc681c not found: ID does not exist" containerID="4ed085418f0efde7366ddf939dc16698831a8396b9388731e4e729ab98cc681c" Oct 29 23:08:40.362831 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:40.362763 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ed085418f0efde7366ddf939dc16698831a8396b9388731e4e729ab98cc681c"} err="failed to get container status \"4ed085418f0efde7366ddf939dc16698831a8396b9388731e4e729ab98cc681c\": rpc error: code = NotFound desc = could not find container \"4ed085418f0efde7366ddf939dc16698831a8396b9388731e4e729ab98cc681c\": container with ID starting with 4ed085418f0efde7366ddf939dc16698831a8396b9388731e4e729ab98cc681c not found: ID does not exist" Oct 29 23:08:40.402593 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:40.402531 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-k5t2b"] Oct 29 23:08:40.411901 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:40.411878 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-k5t2b"] Oct 29 23:08:40.435712 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:40.435687 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-2mcsr"] Oct 29 23:08:40.442547 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:40.442525 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-2mcsr"] Oct 29 23:08:41.333150 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:41.333115 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e61a681b-48a0-4b15-bea3-013044a07714" path="/var/lib/kubelet/pods/e61a681b-48a0-4b15-bea3-013044a07714/volumes" Oct 29 23:08:41.333832 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:41.333813 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8dc3f31-0a4c-44e6-a3e9-4c45e238d1bc" path="/var/lib/kubelet/pods/f8dc3f31-0a4c-44e6-a3e9-4c45e238d1bc/volumes" Oct 29 23:08:42.090183 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:42.090145 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:08:44.071903 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:44.071853 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:08:45.986091 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:45.986050 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:08:47.700099 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:47.700059 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:08:49.029370 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:08:49.029303 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d2dnn" containerName="extractor" Oct 29 23:08:49.375479 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:49.375244 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:08:49.507065 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:49.507025 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:08:49.605761 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:49.605726 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v2tfm"] Oct 29 23:08:52.311587 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:52.311492 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-31226c/distinct-w6-31226c-kserve-858d95fd76-bkf7c" podUID="f3fc11c6-03c9-4314-99c3-f857b149415c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 29 23:08:52.479295 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:52.479258 5057 patch_prober.go:29] interesting pod/e2e-worker-w4-063b46-openshift-default-799d8d64cf-kk86b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 23:08:52.479615 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:52.479583 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-kk86b" podUID="79d262bd-3d59-44ec-b375-28e6c7a57c1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 23:08:52.981272 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:52.981235 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:08:53.246497 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:53.246405 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/distinct-2-w2-575d59-kserve-d4565b76c-hdgbf" podUID="06bb975e-d825-426f-b030-17bd1f99bb65" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:08:53.315482 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:53.315420 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-31226c/distinct-w6-31226c-kserve-858d95fd76-bkf7c" podUID="f3fc11c6-03c9-4314-99c3-f857b149415c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:08:53.316914 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:53.316890 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-31226c/distinct-w6-31226c-kserve-858d95fd76-bkf7c" Oct 29 23:08:53.933003 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:53.932440 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-08e15d/distinct-w0-08e15d-kserve-85695c78bb-p67gk" podUID="e664f012-3b80-4a8a-ac19-d767d92c0a33" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": context deadline exceeded" Oct 29 23:08:54.522276 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:54.522237 5057 patch_prober.go:29] interesting pod/prometheus-operator-admission-webhook-6489d8bf89-hsxj9 container/prometheus-operator-admission-webhook namespace/openshift-monitoring: Liveness probe status=failure output="Get \"https://10.132.0.28:8443/healthz\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 23:08:54.579657 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:54.579168 5057 patch_prober.go:29] interesting pod/e2e-worker-w1-e131af-openshift-default-c476c699c-d6kwq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 23:08:54.580154 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:54.580116 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-d6kwq" podUID="73a11a50-d1bd-4eb5-8685-13e724462b24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 23:08:54.584732 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:54.584667 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-jqfzk" podUID="cee3c70c-dc5a-4fae-9d28-820fc4dea236" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:08:54.663684 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:54.663631 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17394a/distinct-2-w3-17394a-kserve-6545c766c4-h4chf" podUID="f29e8b65-b7dd-45d4-86b0-f13a29ef8e66" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": context deadline exceeded" Oct 29 23:08:54.665157 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:54.665111 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-31226c/distinct-w6-31226c-kserve-858d95fd76-bkf7c" podUID="f3fc11c6-03c9-4314-99c3-f857b149415c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded" Oct 29 23:08:54.666051 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:54.666025 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/distinct-2-w2-575d59-kserve-d4565b76c-hdgbf" podUID="06bb975e-d825-426f-b030-17bd1f99bb65" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": context deadline exceeded" Oct 29 23:08:54.713396 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:54.689258 5057 patch_prober.go:29] interesting pod/downloads-6dd789c8b7-9s8rp container/download-server namespace/openshift-console: Liveness probe status=failure output="Get \"http://10.132.0.29:8080/\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 23:08:54.767300 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:54.766688 5057 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-hsxj9" podUID="b83c5586-450b-46e6-9759-c3d94b50da13" containerName="prometheus-operator-admission-webhook" probeResult="failure" output="Get \"https://10.132.0.28:8443/healthz\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:08:54.785510 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:08:54.777192 5057 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.441s" Oct 29 23:08:54.785850 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:54.785830 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-575d59/distinct-2-w2-575d59-kserve-d4565b76c-hdgbf" Oct 29 23:08:54.812917 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:54.689325 5057 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-console/downloads-6dd789c8b7-9s8rp" podUID="612c3068-fbdf-4cd3-bf23-4099b112c655" containerName="download-server" probeResult="failure" output="Get \"http://10.132.0.29:8080/\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 23:08:54.827358 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:54.826110 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-575d59/distinct-2-w2-575d59-kserve-d4565b76c-hdgbf" Oct 29 23:08:54.859633 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:54.859582 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-31226c/distinct-w6-31226c-kserve-858d95fd76-bkf7c" Oct 29 23:08:54.866038 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:54.866016 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:08:55.480016 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:55.479984 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:08:55.601714 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:55.601680 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:08:55.718209 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:55.718172 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:08:56.132661 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:56.132619 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:08:57.420049 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:57.420012 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:08:59.118527 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:08:59.118484 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:09:00.747679 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:00.747632 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:09:00.951494 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:00.951456 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:09:01.275683 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:01.275636 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:09:01.472184 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:01.472148 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:09:01.646462 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:01.646316 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:09:01.919505 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:01.919407 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:09:02.085054 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:02.085018 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:09:02.982351 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:02.982301 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:09:03.118781 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:03.118737 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:09:03.241410 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:03.240993 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:09:03.355273 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:03.355233 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:09:03.476198 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:03.476155 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:09:04.846394 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:04.846363 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:09:05.774073 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:05.774038 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:09:05.901988 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:05.901947 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:09:06.026011 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:06.025925 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:09:06.138090 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:06.138045 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:09:06.804014 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:06.803977 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:09:07.194554 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:07.194470 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:09:07.675022 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:07.674978 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:09:07.808100 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:07.808056 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:09:08.713613 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:08.713565 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:09:08.997437 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:08.997346 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:09:09.117952 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:09.117906 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:09:09.281193 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:09.281057 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:09:09.927457 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:09.926732 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:09:10.039678 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:10.039629 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:09:10.924102 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:10.924063 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:09:11.891771 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:11.891732 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:09:12.007899 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:12.007862 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:09:12.559180 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:12.559145 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:09:12.913522 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:12.913436 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:09:13.180260 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:13.180132 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:09:13.295946 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:13.295906 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:09:13.761493 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:13.761451 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:09:13.887670 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:13.887626 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:09:15.322007 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:15.321913 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:09:16.959699 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:16.959664 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:09:17.081581 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:17.081545 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:09:17.233711 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:17.233633 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:09:18.751226 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:18.751188 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:09:18.870214 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:18.870176 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:09:19.059431 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:09:19.059281 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d2dnn" containerName="extractor" Oct 29 23:09:19.502289 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:19.502253 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:09:21.300991 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:21.300953 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:09:21.422294 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:21.422234 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:09:21.535054 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:21.534933 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:09:21.643672 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:21.643549 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:09:22.029461 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:22.029356 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:09:22.153545 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:22.153514 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:09:23.201374 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:23.201291 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:09:23.323983 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:23.323949 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:09:23.524392 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:23.524303 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:09:25.739356 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:25.738228 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-hchzr"] Oct 29 23:09:25.755603 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:25.755529 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:09:25.765216 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:25.765188 5057 cpu_manager.go:414] "containerMap: removing container" podUID="e61a681b-48a0-4b15-bea3-013044a07714" containerName="istio-proxy" Oct 29 23:09:25.765463 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:25.765446 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e61a681b-48a0-4b15-bea3-013044a07714" containerName="istio-proxy" Oct 29 23:09:25.765591 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:25.765580 5057 cpu_manager.go:414] "containerMap: removing container" podUID="f8dc3f31-0a4c-44e6-a3e9-4c45e238d1bc" containerName="istio-proxy" Oct 29 23:09:25.765689 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:25.765677 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8dc3f31-0a4c-44e6-a3e9-4c45e238d1bc" containerName="istio-proxy" Oct 29 23:09:25.765814 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:25.765802 5057 cpu_manager.go:414] "containerMap: removing container" podUID="8c9bed15-70b8-401f-820b-b793e47becd6" containerName="payload-pre-processing" Oct 29 23:09:25.765915 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:25.765903 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c9bed15-70b8-401f-820b-b793e47becd6" containerName="payload-pre-processing" Oct 29 23:09:25.766016 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:25.766005 5057 cpu_manager.go:414] "containerMap: removing container" podUID="21dce295-63ce-4d61-8cfa-86669417b525" containerName="istio-proxy" Oct 29 23:09:25.766113 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:25.766102 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21dce295-63ce-4d61-8cfa-86669417b525" containerName="istio-proxy" Oct 29 23:09:25.766211 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:25.766201 5057 cpu_manager.go:414] "containerMap: removing container" podUID="03a26d04-61f6-4e70-b52d-f6263220a95d" containerName="istio-proxy" Oct 29 23:09:25.766299 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:25.766289 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03a26d04-61f6-4e70-b52d-f6263220a95d" containerName="istio-proxy" Oct 29 23:09:25.766422 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:25.766410 5057 cpu_manager.go:414] "containerMap: removing container" podUID="8d30e7e8-5647-43a3-b15f-074537fa73ba" containerName="payload-pre-processing" Oct 29 23:09:25.766527 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:25.766515 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d30e7e8-5647-43a3-b15f-074537fa73ba" containerName="payload-pre-processing" Oct 29 23:09:25.766630 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:25.766620 5057 cpu_manager.go:414] "containerMap: removing container" podUID="cd71b31d-7725-420c-94ab-f94eb006b35f" containerName="payload-pre-processing" Oct 29 23:09:25.766729 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:25.766717 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd71b31d-7725-420c-94ab-f94eb006b35f" containerName="payload-pre-processing" Oct 29 23:09:25.766832 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:25.766822 5057 cpu_manager.go:414] "containerMap: removing container" podUID="96d53b6e-441d-411e-aa0c-efcb79e89983" containerName="payload-pre-processing" Oct 29 23:09:25.766925 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:25.766914 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96d53b6e-441d-411e-aa0c-efcb79e89983" containerName="payload-pre-processing" Oct 29 23:09:25.767028 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:25.767017 5057 cpu_manager.go:414] "containerMap: removing container" podUID="b4535855-9770-43ca-abb3-3ee2a56d03d3" containerName="payload-processing" Oct 29 23:09:25.767130 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:25.767119 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4535855-9770-43ca-abb3-3ee2a56d03d3" containerName="payload-processing" Oct 29 23:09:25.767267 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:25.767256 5057 cpu_manager.go:414] "containerMap: removing container" podUID="27e3d9c6-89a4-42fd-9a98-c030f6a22990" containerName="payload-processing" Oct 29 23:09:25.767383 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:25.767371 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27e3d9c6-89a4-42fd-9a98-c030f6a22990" containerName="payload-processing" Oct 29 23:09:25.767566 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:25.767551 5057 cpu_manager.go:414] "containerMap: removing container" podUID="7e2e4597-f643-42fb-a968-1dd7f39407fe" containerName="istio-proxy" Oct 29 23:09:25.767685 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:25.767672 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e2e4597-f643-42fb-a968-1dd7f39407fe" containerName="istio-proxy" Oct 29 23:09:25.768031 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:25.768014 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="27e3d9c6-89a4-42fd-9a98-c030f6a22990" containerName="payload-processing" Oct 29 23:09:25.768165 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:25.768154 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e2e4597-f643-42fb-a968-1dd7f39407fe" containerName="istio-proxy" Oct 29 23:09:25.768273 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:25.768263 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="e61a681b-48a0-4b15-bea3-013044a07714" containerName="istio-proxy" Oct 29 23:09:25.768465 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:25.768450 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8dc3f31-0a4c-44e6-a3e9-4c45e238d1bc" containerName="istio-proxy" Oct 29 23:09:25.768583 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:25.768572 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c9bed15-70b8-401f-820b-b793e47becd6" containerName="payload-pre-processing" Oct 29 23:09:25.768685 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:25.768675 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="21dce295-63ce-4d61-8cfa-86669417b525" containerName="istio-proxy" Oct 29 23:09:25.768785 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:25.768774 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="03a26d04-61f6-4e70-b52d-f6263220a95d" containerName="istio-proxy" Oct 29 23:09:25.768877 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:25.768867 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d30e7e8-5647-43a3-b15f-074537fa73ba" containerName="payload-pre-processing" Oct 29 23:09:25.768978 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:25.768967 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd71b31d-7725-420c-94ab-f94eb006b35f" containerName="payload-pre-processing" Oct 29 23:09:25.769063 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:25.769053 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="96d53b6e-441d-411e-aa0c-efcb79e89983" containerName="payload-pre-processing" Oct 29 23:09:25.771025 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:25.771009 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4535855-9770-43ca-abb3-3ee2a56d03d3" containerName="payload-processing" Oct 29 23:09:25.867356 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:25.844762 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-hchzr"] Oct 29 23:09:25.922353 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:25.868550 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-hchzr" Oct 29 23:09:25.922353 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:25.878386 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x56mg\" (UniqueName: \"kubernetes.io/projected/c0fe6ba5-aa29-48d1-955d-e8e063d8e77f-kube-api-access-x56mg\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-hchzr\" (UID: \"c0fe6ba5-aa29-48d1-955d-e8e063d8e77f\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-hchzr" Oct 29 23:09:25.922353 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:25.878435 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0fe6ba5-aa29-48d1-955d-e8e063d8e77f-credential-socket\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-hchzr\" (UID: \"c0fe6ba5-aa29-48d1-955d-e8e063d8e77f\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-hchzr" Oct 29 23:09:25.922353 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:25.878497 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0fe6ba5-aa29-48d1-955d-e8e063d8e77f-istio-envoy\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-hchzr\" (UID: \"c0fe6ba5-aa29-48d1-955d-e8e063d8e77f\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-hchzr" Oct 29 23:09:25.922353 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:25.878543 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0fe6ba5-aa29-48d1-955d-e8e063d8e77f-istio-token\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-hchzr\" (UID: \"c0fe6ba5-aa29-48d1-955d-e8e063d8e77f\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-hchzr" Oct 29 23:09:25.922353 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:25.878625 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0fe6ba5-aa29-48d1-955d-e8e063d8e77f-istio-data\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-hchzr\" (UID: \"c0fe6ba5-aa29-48d1-955d-e8e063d8e77f\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-hchzr" Oct 29 23:09:25.922353 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:25.878702 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0fe6ba5-aa29-48d1-955d-e8e063d8e77f-workload-certs\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-hchzr\" (UID: \"c0fe6ba5-aa29-48d1-955d-e8e063d8e77f\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-hchzr" Oct 29 23:09:25.922353 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:25.883706 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-08e15d/distinct-w0-08e15d-kserve-85695c78bb-p67gk" podUID="e664f012-3b80-4a8a-ac19-d767d92c0a33" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:09:25.971355 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:25.968257 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0fe6ba5-aa29-48d1-955d-e8e063d8e77f-istio-podinfo\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-hchzr\" (UID: \"c0fe6ba5-aa29-48d1-955d-e8e063d8e77f\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-hchzr" Oct 29 23:09:25.971355 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:25.968375 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0fe6ba5-aa29-48d1-955d-e8e063d8e77f-workload-socket\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-hchzr\" (UID: \"c0fe6ba5-aa29-48d1-955d-e8e063d8e77f\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-hchzr" Oct 29 23:09:25.971355 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:25.968480 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0fe6ba5-aa29-48d1-955d-e8e063d8e77f-istiod-ca-cert\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-hchzr\" (UID: \"c0fe6ba5-aa29-48d1-955d-e8e063d8e77f\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-hchzr" Oct 29 23:09:26.080409 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:26.079467 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0fe6ba5-aa29-48d1-955d-e8e063d8e77f-istio-podinfo\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-hchzr\" (UID: \"c0fe6ba5-aa29-48d1-955d-e8e063d8e77f\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-hchzr" Oct 29 23:09:26.080409 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:26.079534 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0fe6ba5-aa29-48d1-955d-e8e063d8e77f-workload-socket\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-hchzr\" (UID: \"c0fe6ba5-aa29-48d1-955d-e8e063d8e77f\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-hchzr" Oct 29 23:09:26.080409 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:26.079593 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0fe6ba5-aa29-48d1-955d-e8e063d8e77f-istiod-ca-cert\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-hchzr\" (UID: \"c0fe6ba5-aa29-48d1-955d-e8e063d8e77f\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-hchzr" Oct 29 23:09:26.080409 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:26.079693 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x56mg\" (UniqueName: \"kubernetes.io/projected/c0fe6ba5-aa29-48d1-955d-e8e063d8e77f-kube-api-access-x56mg\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-hchzr\" (UID: \"c0fe6ba5-aa29-48d1-955d-e8e063d8e77f\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-hchzr" Oct 29 23:09:26.080409 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:26.079733 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0fe6ba5-aa29-48d1-955d-e8e063d8e77f-credential-socket\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-hchzr\" (UID: \"c0fe6ba5-aa29-48d1-955d-e8e063d8e77f\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-hchzr" Oct 29 23:09:26.080409 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:26.079804 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0fe6ba5-aa29-48d1-955d-e8e063d8e77f-istio-envoy\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-hchzr\" (UID: \"c0fe6ba5-aa29-48d1-955d-e8e063d8e77f\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-hchzr" Oct 29 23:09:26.080409 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:26.079857 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0fe6ba5-aa29-48d1-955d-e8e063d8e77f-istio-token\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-hchzr\" (UID: \"c0fe6ba5-aa29-48d1-955d-e8e063d8e77f\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-hchzr" Oct 29 23:09:26.080409 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:26.079959 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0fe6ba5-aa29-48d1-955d-e8e063d8e77f-istio-data\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-hchzr\" (UID: \"c0fe6ba5-aa29-48d1-955d-e8e063d8e77f\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-hchzr" Oct 29 23:09:26.080409 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:26.080049 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0fe6ba5-aa29-48d1-955d-e8e063d8e77f-workload-certs\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-hchzr\" (UID: \"c0fe6ba5-aa29-48d1-955d-e8e063d8e77f\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-hchzr" Oct 29 23:09:26.081013 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:26.080543 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0fe6ba5-aa29-48d1-955d-e8e063d8e77f-workload-certs\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-hchzr\" (UID: \"c0fe6ba5-aa29-48d1-955d-e8e063d8e77f\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-hchzr" Oct 29 23:09:26.119361 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:26.105766 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0fe6ba5-aa29-48d1-955d-e8e063d8e77f-credential-socket\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-hchzr\" (UID: \"c0fe6ba5-aa29-48d1-955d-e8e063d8e77f\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-hchzr" Oct 29 23:09:26.119361 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:26.106047 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0fe6ba5-aa29-48d1-955d-e8e063d8e77f-workload-socket\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-hchzr\" (UID: \"c0fe6ba5-aa29-48d1-955d-e8e063d8e77f\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-hchzr" Oct 29 23:09:26.119361 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:26.106729 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0fe6ba5-aa29-48d1-955d-e8e063d8e77f-istiod-ca-cert\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-hchzr\" (UID: \"c0fe6ba5-aa29-48d1-955d-e8e063d8e77f\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-hchzr" Oct 29 23:09:26.135354 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:26.124747 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0fe6ba5-aa29-48d1-955d-e8e063d8e77f-istio-data\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-hchzr\" (UID: \"c0fe6ba5-aa29-48d1-955d-e8e063d8e77f\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-hchzr" Oct 29 23:09:26.165414 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:26.157996 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0fe6ba5-aa29-48d1-955d-e8e063d8e77f-istio-podinfo\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-hchzr\" (UID: \"c0fe6ba5-aa29-48d1-955d-e8e063d8e77f\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-hchzr" Oct 29 23:09:26.171354 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:26.168979 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0fe6ba5-aa29-48d1-955d-e8e063d8e77f-istio-envoy\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-hchzr\" (UID: \"c0fe6ba5-aa29-48d1-955d-e8e063d8e77f\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-hchzr" Oct 29 23:09:26.198386 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:26.191394 5057 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-17394a/distinct-2-w3-17394a-kserve-6545c766c4-h4chf" podUID="f29e8b65-b7dd-45d4-86b0-f13a29ef8e66" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/health\": context deadline exceeded" Oct 29 23:09:26.246355 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:26.246058 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x56mg\" (UniqueName: \"kubernetes.io/projected/c0fe6ba5-aa29-48d1-955d-e8e063d8e77f-kube-api-access-x56mg\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-hchzr\" (UID: \"c0fe6ba5-aa29-48d1-955d-e8e063d8e77f\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-hchzr" Oct 29 23:09:26.281744 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:26.259920 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0fe6ba5-aa29-48d1-955d-e8e063d8e77f-istio-token\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-hchzr\" (UID: \"c0fe6ba5-aa29-48d1-955d-e8e063d8e77f\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-hchzr" Oct 29 23:09:26.375222 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:26.365555 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-hchzr" Oct 29 23:09:26.795362 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:26.738648 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:09:26.861651 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:26.861602 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/distinct-2-w2-575d59-kserve-d4565b76c-hdgbf" podUID="06bb975e-d825-426f-b030-17bd1f99bb65" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": context deadline exceeded" Oct 29 23:09:26.912427 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:26.912386 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-08e15d/distinct-w0-08e15d-kserve-85695c78bb-p67gk" podUID="e664f012-3b80-4a8a-ac19-d767d92c0a33" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": context deadline exceeded" Oct 29 23:09:26.913047 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:26.913022 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-08e15d/distinct-w0-08e15d-kserve-85695c78bb-p67gk" Oct 29 23:09:27.731690 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:27.731636 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:09:27.910002 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:27.909929 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-575d59/distinct-2-w2-575d59-kserve-d4565b76c-hdgbf" podUID="06bb975e-d825-426f-b030-17bd1f99bb65" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:09:28.057365 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:28.057258 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17394a/distinct-2-w3-17394a-kserve-6545c766c4-h4chf" podUID="f29e8b65-b7dd-45d4-86b0-f13a29ef8e66" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": context deadline exceeded" Oct 29 23:09:28.107322 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:28.105054 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-08e15d/distinct-w0-08e15d-kserve-85695c78bb-p67gk" Oct 29 23:09:28.107322 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:28.105121 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-575d59/distinct-2-w2-575d59-kserve-d4565b76c-hdgbf" Oct 29 23:09:28.107322 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:28.105151 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-575d59/distinct-2-w2-575d59-kserve-d4565b76c-hdgbf" Oct 29 23:09:28.361035 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:28.361001 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-hchzr"] Oct 29 23:09:28.366501 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:28.366459 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 23:09:28.366711 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:28.366684 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 23:09:28.366850 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:28.366828 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 23:09:28.702883 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:28.702849 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:09:28.829018 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:28.828966 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:09:28.901994 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:28.901953 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b0c3c3-75d548567-9267h"] Oct 29 23:09:28.908773 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:28.908748 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b0c3c3-75d548567-9267h" Oct 29 23:09:28.921416 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:28.921386 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6fls\" (UniqueName: \"kubernetes.io/projected/34be4000-c954-4c58-a304-1944cace0093-kube-api-access-l6fls\") pod \"payload-pre-processing-e2e-shared-a-w2-b0c3c3-75d548567-9267h\" (UID: \"34be4000-c954-4c58-a304-1944cace0093\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b0c3c3-75d548567-9267h" Oct 29 23:09:28.921918 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:28.921882 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34be4000-c954-4c58-a304-1944cace0093-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-b0c3c3-75d548567-9267h\" (UID: \"34be4000-c954-4c58-a304-1944cace0093\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b0c3c3-75d548567-9267h" Oct 29 23:09:28.922037 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:28.921694 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-b0c3c3-dockercfg-6gwvr\"" Oct 29 23:09:28.922373 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:28.922355 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-b0c3c3\"" Oct 29 23:09:28.937759 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:28.937634 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b0c3c3-75d548567-9267h"] Oct 29 23:09:28.970160 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:28.970082 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-b0c3c3-5867dcd5b4-xs5ll"] Oct 29 23:09:28.976494 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:28.976462 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b0c3c3-5867dcd5b4-xs5ll" Oct 29 23:09:28.995631 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:28.995601 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-b0c3c3-5867dcd5b4-xs5ll"] Oct 29 23:09:29.023799 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:29.023769 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8pcch\" (UniqueName: \"kubernetes.io/projected/93b0246c-b176-4daa-8090-6bb557b4fa84-kube-api-access-8pcch\") pod \"payload-processing-e2e-shared-a-w2-b0c3c3-5867dcd5b4-xs5ll\" (UID: \"93b0246c-b176-4daa-8090-6bb557b4fa84\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b0c3c3-5867dcd5b4-xs5ll" Oct 29 23:09:29.023954 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:29.023881 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6fls\" (UniqueName: \"kubernetes.io/projected/34be4000-c954-4c58-a304-1944cace0093-kube-api-access-l6fls\") pod \"payload-pre-processing-e2e-shared-a-w2-b0c3c3-75d548567-9267h\" (UID: \"34be4000-c954-4c58-a304-1944cace0093\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b0c3c3-75d548567-9267h" Oct 29 23:09:29.023954 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:29.023913 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34be4000-c954-4c58-a304-1944cace0093-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-b0c3c3-75d548567-9267h\" (UID: \"34be4000-c954-4c58-a304-1944cace0093\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b0c3c3-75d548567-9267h" Oct 29 23:09:29.023954 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:29.023946 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93b0246c-b176-4daa-8090-6bb557b4fa84-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-b0c3c3-5867dcd5b4-xs5ll\" (UID: \"93b0246c-b176-4daa-8090-6bb557b4fa84\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b0c3c3-5867dcd5b4-xs5ll" Oct 29 23:09:29.024832 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:29.024787 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34be4000-c954-4c58-a304-1944cace0093-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-b0c3c3-75d548567-9267h\" (UID: \"34be4000-c954-4c58-a304-1944cace0093\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b0c3c3-75d548567-9267h" Oct 29 23:09:29.047512 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:29.047484 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6fls\" (UniqueName: \"kubernetes.io/projected/34be4000-c954-4c58-a304-1944cace0093-kube-api-access-l6fls\") pod \"payload-pre-processing-e2e-shared-a-w2-b0c3c3-75d548567-9267h\" (UID: \"34be4000-c954-4c58-a304-1944cace0093\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b0c3c3-75d548567-9267h" Oct 29 23:09:29.112924 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:29.112881 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-hchzr" event={"ID":"c0fe6ba5-aa29-48d1-955d-e8e063d8e77f","Type":"ContainerStarted","Data":"448ae9294e937a80008216ac8aea75ebed0a6da9d3caa4292cdc432ee1bf3060"} Oct 29 23:09:29.113129 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:29.112949 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-hchzr" event={"ID":"c0fe6ba5-aa29-48d1-955d-e8e063d8e77f","Type":"ContainerStarted","Data":"8695b21af758fc7ea39c8bc5f1efe2e54ababa2d0791867b02422b6d15775888"} Oct 29 23:09:29.134060 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:29.134020 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8pcch\" (UniqueName: \"kubernetes.io/projected/93b0246c-b176-4daa-8090-6bb557b4fa84-kube-api-access-8pcch\") pod \"payload-processing-e2e-shared-a-w2-b0c3c3-5867dcd5b4-xs5ll\" (UID: \"93b0246c-b176-4daa-8090-6bb557b4fa84\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b0c3c3-5867dcd5b4-xs5ll" Oct 29 23:09:29.134257 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:29.134134 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93b0246c-b176-4daa-8090-6bb557b4fa84-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-b0c3c3-5867dcd5b4-xs5ll\" (UID: \"93b0246c-b176-4daa-8090-6bb557b4fa84\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b0c3c3-5867dcd5b4-xs5ll" Oct 29 23:09:29.135460 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:29.135409 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93b0246c-b176-4daa-8090-6bb557b4fa84-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-b0c3c3-5867dcd5b4-xs5ll\" (UID: \"93b0246c-b176-4daa-8090-6bb557b4fa84\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b0c3c3-5867dcd5b4-xs5ll" Oct 29 23:09:29.158647 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:29.158617 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8pcch\" (UniqueName: \"kubernetes.io/projected/93b0246c-b176-4daa-8090-6bb557b4fa84-kube-api-access-8pcch\") pod \"payload-processing-e2e-shared-a-w2-b0c3c3-5867dcd5b4-xs5ll\" (UID: \"93b0246c-b176-4daa-8090-6bb557b4fa84\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b0c3c3-5867dcd5b4-xs5ll" Oct 29 23:09:29.166050 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:29.166003 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-hchzr" podStartSLOduration=4.165988819 podStartE2EDuration="4.165988819s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:09:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:09:29.162576115 +0000 UTC m=+2149.945789541" watchObservedRunningTime="2026-10-29 23:09:29.165988819 +0000 UTC m=+2149.949202244" Oct 29 23:09:29.203374 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:29.201388 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:09:29.225521 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:29.225438 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b0c3c3-75d548567-9267h" Oct 29 23:09:29.315354 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:29.314193 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b0c3c3-5867dcd5b4-xs5ll" Oct 29 23:09:29.371394 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:29.367768 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-hchzr" Oct 29 23:09:29.371394 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:29.369027 5057 patch_prober.go:29] interesting pod/e2e-worker-w5-b2921b-openshift-default-5547c88fff-hchzr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" start-of-body= Oct 29 23:09:29.371394 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:29.369075 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-hchzr" podUID="c0fe6ba5-aa29-48d1-955d-e8e063d8e77f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" Oct 29 23:09:29.759198 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:29.723025 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b0c3c3-75d548567-9267h"] Oct 29 23:09:29.966929 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:29.966889 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-b0c3c3-5867dcd5b4-xs5ll"] Oct 29 23:09:30.127214 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:30.127177 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b0c3c3-75d548567-9267h" event={"ID":"34be4000-c954-4c58-a304-1944cace0093","Type":"ContainerStarted","Data":"6794564be13c7916c1a2d4e55916bade68a5ef4d697eed35e8e76c4545b2b8cc"} Oct 29 23:09:30.127421 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:30.127253 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b0c3c3-75d548567-9267h" event={"ID":"34be4000-c954-4c58-a304-1944cace0093","Type":"ContainerStarted","Data":"193fafe5a3daf3825a3bbe1670f6c73600763fa990f199fe7c9b36fcfe1918bb"} Oct 29 23:09:30.127421 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:30.127286 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b0c3c3-75d548567-9267h" Oct 29 23:09:30.129018 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:30.128990 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b0c3c3-5867dcd5b4-xs5ll" event={"ID":"93b0246c-b176-4daa-8090-6bb557b4fa84","Type":"ContainerStarted","Data":"695f4fa4c9287bcb836c7f4d1ad500efb1806738ed52eecfdd492b7f2b4f1416"} Oct 29 23:09:30.167020 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:30.166980 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b0c3c3-75d548567-9267h" podStartSLOduration=2.166966899 podStartE2EDuration="2.166966899s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:09:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:09:30.162972173 +0000 UTC m=+2150.946185601" watchObservedRunningTime="2026-10-29 23:09:30.166966899 +0000 UTC m=+2150.950180325" Oct 29 23:09:30.367010 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:30.366925 5057 patch_prober.go:29] interesting pod/e2e-worker-w5-b2921b-openshift-default-5547c88fff-hchzr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" start-of-body= Oct 29 23:09:30.367010 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:30.366988 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-hchzr" podUID="c0fe6ba5-aa29-48d1-955d-e8e063d8e77f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" Oct 29 23:09:31.139885 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:31.139855 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b0c3c3-5867dcd5b4-xs5ll" event={"ID":"93b0246c-b176-4daa-8090-6bb557b4fa84","Type":"ContainerStarted","Data":"c4d8ab74870b6d6166812a9f91d9a3a42f00578efa4d9806c00f59c3a09fcbc5"} Oct 29 23:09:31.140463 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:31.140437 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b0c3c3-5867dcd5b4-xs5ll" Oct 29 23:09:31.210356 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:31.209858 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b0c3c3-5867dcd5b4-xs5ll" podStartSLOduration=3.209838301 podStartE2EDuration="3.209838301s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:09:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:09:31.175680497 +0000 UTC m=+2151.958893923" watchObservedRunningTime="2026-10-29 23:09:31.209838301 +0000 UTC m=+2151.993051728" Oct 29 23:09:31.370296 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:31.366751 5057 patch_prober.go:29] interesting pod/e2e-worker-w5-b2921b-openshift-default-5547c88fff-hchzr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" start-of-body= Oct 29 23:09:31.370296 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:31.366806 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-hchzr" podUID="c0fe6ba5-aa29-48d1-955d-e8e063d8e77f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" Oct 29 23:09:31.725372 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:31.724699 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:09:32.367070 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:32.367039 5057 patch_prober.go:29] interesting pod/e2e-worker-w5-b2921b-openshift-default-5547c88fff-hchzr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" start-of-body= Oct 29 23:09:32.367584 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:32.367087 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-hchzr" podUID="c0fe6ba5-aa29-48d1-955d-e8e063d8e77f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" Oct 29 23:09:33.366737 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:33.366706 5057 patch_prober.go:29] interesting pod/e2e-worker-w5-b2921b-openshift-default-5547c88fff-hchzr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" start-of-body= Oct 29 23:09:33.366916 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:33.366766 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-hchzr" podUID="c0fe6ba5-aa29-48d1-955d-e8e063d8e77f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" Oct 29 23:09:33.637157 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:33.637075 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:09:34.027104 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:34.027071 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:09:34.139955 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:34.139920 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:09:34.309546 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:34.309444 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:09:34.366317 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:34.366286 5057 patch_prober.go:29] interesting pod/e2e-worker-w5-b2921b-openshift-default-5547c88fff-hchzr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" start-of-body= Oct 29 23:09:34.366536 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:34.366355 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-hchzr" podUID="c0fe6ba5-aa29-48d1-955d-e8e063d8e77f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" Oct 29 23:09:35.366243 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:35.366212 5057 patch_prober.go:29] interesting pod/e2e-worker-w5-b2921b-openshift-default-5547c88fff-hchzr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" start-of-body= Oct 29 23:09:35.366659 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:35.366256 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-hchzr" podUID="c0fe6ba5-aa29-48d1-955d-e8e063d8e77f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" Oct 29 23:09:36.447943 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:36.447911 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-hchzr" Oct 29 23:09:36.448683 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:36.448656 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-hchzr" Oct 29 23:09:36.449658 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:36.449635 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-hchzr" Oct 29 23:09:36.717628 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:36.717529 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:09:38.746059 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:38.746019 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:09:40.415637 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:40.415604 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:09:41.143398 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:41.143368 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b0c3c3-75d548567-9267h" Oct 29 23:09:42.068236 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:42.068191 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:09:42.151555 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:42.151528 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b0c3c3-5867dcd5b4-xs5ll" Oct 29 23:09:42.184029 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:42.183995 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:09:43.885497 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:43.885458 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:09:45.691987 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:45.691945 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:09:47.338667 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:47.338629 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:09:49.028004 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:09:49.027967 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d2dnn" containerName="extractor" Oct 29 23:09:49.176252 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:49.176220 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:09:51.086892 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:51.086834 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:09:51.216689 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:51.216656 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:09:51.407408 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:51.407309 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:09:51.613213 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:51.613179 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:09:51.649761 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:51.649726 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7614-d8e390"] Oct 29 23:09:51.670885 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:51.670785 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7614-d8e390"] Oct 29 23:09:51.670885 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:51.670884 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7614-d8e390" Oct 29 23:09:51.686098 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:51.686065 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-fvr2l\"" Oct 29 23:09:51.728615 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:51.728578 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:09:51.827994 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:51.827951 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8q2lv\" (UniqueName: \"kubernetes.io/projected/5d38cbfa-f5b9-4349-b380-a430e551e6f7-kube-api-access-8q2lv\") pod \"test-curl-7614-d8e390\" (UID: \"5d38cbfa-f5b9-4349-b380-a430e551e6f7\") " pod="opendatahub/test-curl-7614-d8e390" Oct 29 23:09:51.929930 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:51.929847 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8q2lv\" (UniqueName: \"kubernetes.io/projected/5d38cbfa-f5b9-4349-b380-a430e551e6f7-kube-api-access-8q2lv\") pod \"test-curl-7614-d8e390\" (UID: \"5d38cbfa-f5b9-4349-b380-a430e551e6f7\") " pod="opendatahub/test-curl-7614-d8e390" Oct 29 23:09:51.951474 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:51.951441 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8q2lv\" (UniqueName: \"kubernetes.io/projected/5d38cbfa-f5b9-4349-b380-a430e551e6f7-kube-api-access-8q2lv\") pod \"test-curl-7614-d8e390\" (UID: \"5d38cbfa-f5b9-4349-b380-a430e551e6f7\") " pod="opendatahub/test-curl-7614-d8e390" Oct 29 23:09:51.988627 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:51.988595 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7614-d8e390" Oct 29 23:09:52.021342 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:52.021286 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:09:52.162481 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:52.162448 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:09:52.230174 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:52.230141 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-ksjql"] Oct 29 23:09:52.236523 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:52.236499 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-ksjql" Oct 29 23:09:52.338441 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:52.336713 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88b81626-da3c-4b1d-b9d8-3dbece193e80-istio-token\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-ksjql\" (UID: \"88b81626-da3c-4b1d-b9d8-3dbece193e80\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-ksjql" Oct 29 23:09:52.339487 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:52.339453 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qcmjw\" (UniqueName: \"kubernetes.io/projected/88b81626-da3c-4b1d-b9d8-3dbece193e80-kube-api-access-qcmjw\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-ksjql\" (UID: \"88b81626-da3c-4b1d-b9d8-3dbece193e80\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-ksjql" Oct 29 23:09:52.339735 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:52.339714 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88b81626-da3c-4b1d-b9d8-3dbece193e80-istio-podinfo\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-ksjql\" (UID: \"88b81626-da3c-4b1d-b9d8-3dbece193e80\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-ksjql" Oct 29 23:09:52.339872 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:52.339857 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88b81626-da3c-4b1d-b9d8-3dbece193e80-istio-data\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-ksjql\" (UID: \"88b81626-da3c-4b1d-b9d8-3dbece193e80\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-ksjql" Oct 29 23:09:52.340097 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:52.340079 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88b81626-da3c-4b1d-b9d8-3dbece193e80-istiod-ca-cert\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-ksjql\" (UID: \"88b81626-da3c-4b1d-b9d8-3dbece193e80\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-ksjql" Oct 29 23:09:52.340244 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:52.340227 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88b81626-da3c-4b1d-b9d8-3dbece193e80-credential-socket\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-ksjql\" (UID: \"88b81626-da3c-4b1d-b9d8-3dbece193e80\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-ksjql" Oct 29 23:09:52.340370 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:52.340353 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88b81626-da3c-4b1d-b9d8-3dbece193e80-workload-certs\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-ksjql\" (UID: \"88b81626-da3c-4b1d-b9d8-3dbece193e80\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-ksjql" Oct 29 23:09:52.340544 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:52.340528 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88b81626-da3c-4b1d-b9d8-3dbece193e80-workload-socket\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-ksjql\" (UID: \"88b81626-da3c-4b1d-b9d8-3dbece193e80\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-ksjql" Oct 29 23:09:52.340634 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:52.340620 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88b81626-da3c-4b1d-b9d8-3dbece193e80-istio-envoy\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-ksjql\" (UID: \"88b81626-da3c-4b1d-b9d8-3dbece193e80\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-ksjql" Oct 29 23:09:52.350700 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:52.348504 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-ksjql"] Oct 29 23:09:52.354453 ip-10-0-140-74 kubenswrapper[5057]: W1029 23:09:52.354417 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5d38cbfa_f5b9_4349_b380_a430e551e6f7.slice/crio-b34740a64bc1b8cc019e15e8be04bfafdd447d931e24635ac8eb2d16a7bb8a90 WatchSource:0}: Error finding container b34740a64bc1b8cc019e15e8be04bfafdd447d931e24635ac8eb2d16a7bb8a90: Status 404 returned error can't find the container with id b34740a64bc1b8cc019e15e8be04bfafdd447d931e24635ac8eb2d16a7bb8a90 Oct 29 23:09:52.357377 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:52.357090 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7614-d8e390"] Oct 29 23:09:52.357853 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:52.357808 5057 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 29 23:09:52.442553 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:52.442520 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88b81626-da3c-4b1d-b9d8-3dbece193e80-istio-podinfo\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-ksjql\" (UID: \"88b81626-da3c-4b1d-b9d8-3dbece193e80\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-ksjql" Oct 29 23:09:52.442755 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:52.442573 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88b81626-da3c-4b1d-b9d8-3dbece193e80-istio-data\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-ksjql\" (UID: \"88b81626-da3c-4b1d-b9d8-3dbece193e80\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-ksjql" Oct 29 23:09:52.442755 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:52.442734 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88b81626-da3c-4b1d-b9d8-3dbece193e80-istiod-ca-cert\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-ksjql\" (UID: \"88b81626-da3c-4b1d-b9d8-3dbece193e80\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-ksjql" Oct 29 23:09:52.443118 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:52.442971 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88b81626-da3c-4b1d-b9d8-3dbece193e80-credential-socket\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-ksjql\" (UID: \"88b81626-da3c-4b1d-b9d8-3dbece193e80\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-ksjql" Oct 29 23:09:52.443118 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:52.443023 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88b81626-da3c-4b1d-b9d8-3dbece193e80-workload-certs\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-ksjql\" (UID: \"88b81626-da3c-4b1d-b9d8-3dbece193e80\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-ksjql" Oct 29 23:09:52.443118 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:52.443075 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88b81626-da3c-4b1d-b9d8-3dbece193e80-istio-data\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-ksjql\" (UID: \"88b81626-da3c-4b1d-b9d8-3dbece193e80\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-ksjql" Oct 29 23:09:52.443371 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:52.443147 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88b81626-da3c-4b1d-b9d8-3dbece193e80-workload-socket\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-ksjql\" (UID: \"88b81626-da3c-4b1d-b9d8-3dbece193e80\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-ksjql" Oct 29 23:09:52.443371 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:52.443174 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88b81626-da3c-4b1d-b9d8-3dbece193e80-istio-envoy\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-ksjql\" (UID: \"88b81626-da3c-4b1d-b9d8-3dbece193e80\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-ksjql" Oct 29 23:09:52.443371 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:52.443274 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88b81626-da3c-4b1d-b9d8-3dbece193e80-credential-socket\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-ksjql\" (UID: \"88b81626-da3c-4b1d-b9d8-3dbece193e80\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-ksjql" Oct 29 23:09:52.443371 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:52.443290 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88b81626-da3c-4b1d-b9d8-3dbece193e80-workload-certs\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-ksjql\" (UID: \"88b81626-da3c-4b1d-b9d8-3dbece193e80\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-ksjql" Oct 29 23:09:52.443589 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:52.443443 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88b81626-da3c-4b1d-b9d8-3dbece193e80-istio-token\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-ksjql\" (UID: \"88b81626-da3c-4b1d-b9d8-3dbece193e80\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-ksjql" Oct 29 23:09:52.443589 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:52.443558 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88b81626-da3c-4b1d-b9d8-3dbece193e80-istiod-ca-cert\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-ksjql\" (UID: \"88b81626-da3c-4b1d-b9d8-3dbece193e80\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-ksjql" Oct 29 23:09:52.443699 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:52.443572 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88b81626-da3c-4b1d-b9d8-3dbece193e80-workload-socket\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-ksjql\" (UID: \"88b81626-da3c-4b1d-b9d8-3dbece193e80\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-ksjql" Oct 29 23:09:52.443699 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:52.443630 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qcmjw\" (UniqueName: \"kubernetes.io/projected/88b81626-da3c-4b1d-b9d8-3dbece193e80-kube-api-access-qcmjw\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-ksjql\" (UID: \"88b81626-da3c-4b1d-b9d8-3dbece193e80\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-ksjql" Oct 29 23:09:52.445755 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:52.445722 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88b81626-da3c-4b1d-b9d8-3dbece193e80-istio-podinfo\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-ksjql\" (UID: \"88b81626-da3c-4b1d-b9d8-3dbece193e80\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-ksjql" Oct 29 23:09:52.445875 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:52.445755 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88b81626-da3c-4b1d-b9d8-3dbece193e80-istio-envoy\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-ksjql\" (UID: \"88b81626-da3c-4b1d-b9d8-3dbece193e80\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-ksjql" Oct 29 23:09:52.509403 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:52.505222 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88b81626-da3c-4b1d-b9d8-3dbece193e80-istio-token\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-ksjql\" (UID: \"88b81626-da3c-4b1d-b9d8-3dbece193e80\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-ksjql" Oct 29 23:09:52.509403 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:52.506178 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qcmjw\" (UniqueName: \"kubernetes.io/projected/88b81626-da3c-4b1d-b9d8-3dbece193e80-kube-api-access-qcmjw\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-ksjql\" (UID: \"88b81626-da3c-4b1d-b9d8-3dbece193e80\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-ksjql" Oct 29 23:09:52.666355 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:52.665653 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-ksjql" Oct 29 23:09:52.886404 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:52.885924 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:09:53.143018 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:53.138542 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-ksjql"] Oct 29 23:09:53.152654 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:53.151102 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 23:09:53.152654 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:53.151198 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 23:09:53.152654 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:53.151246 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 23:09:53.233048 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:53.231956 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:09:53.373732 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:53.373693 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7614-d8e390" event={"ID":"5d38cbfa-f5b9-4349-b380-a430e551e6f7","Type":"ContainerStarted","Data":"b34740a64bc1b8cc019e15e8be04bfafdd447d931e24635ac8eb2d16a7bb8a90"} Oct 29 23:09:53.387090 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:53.381767 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-ksjql" event={"ID":"88b81626-da3c-4b1d-b9d8-3dbece193e80","Type":"ContainerStarted","Data":"7ca2261d53f6a83819eec2b97eca9011fbbe73946a9253be4bb99c6c1f5652a1"} Oct 29 23:09:53.387090 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:53.381827 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-ksjql" event={"ID":"88b81626-da3c-4b1d-b9d8-3dbece193e80","Type":"ContainerStarted","Data":"43721d5d5af6fe8fdaa916dd51d681c7dd42aaaf405bd62c9f95f8bf4be90e05"} Oct 29 23:09:53.486267 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:53.486211 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-ksjql" podStartSLOduration=1.486195525 podStartE2EDuration="1.486195525s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:09:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:09:53.480657335 +0000 UTC m=+2174.263870764" watchObservedRunningTime="2026-10-29 23:09:53.486195525 +0000 UTC m=+2174.269409022" Oct 29 23:09:53.603358 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:53.603122 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:09:53.666509 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:53.666427 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-ksjql" Oct 29 23:09:53.792564 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:53.792521 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:09:53.912589 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:53.912558 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:09:54.439182 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:54.436953 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7614-d8e390" event={"ID":"5d38cbfa-f5b9-4349-b380-a430e551e6f7","Type":"ContainerStarted","Data":"d3c4c53d768566bc67696e7a4bb45f6651095700b527045ac8d7c8976b306828"} Oct 29 23:09:54.497356 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:54.485497 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7614-d8e390" podStartSLOduration=2.574848045 podStartE2EDuration="3.485475599s" totalImagesPullingTime="910.627554ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:09:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:09:54.481494076 +0000 UTC m=+2175.264707502" watchObservedRunningTime="2026-10-29 23:09:54.485475599 +0000 UTC m=+2175.268689038" Oct 29 23:09:54.667612 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:54.667571 5057 patch_prober.go:29] interesting pod/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-ksjql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body= Oct 29 23:09:54.667812 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:54.667633 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-ksjql" podUID="88b81626-da3c-4b1d-b9d8-3dbece193e80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" Oct 29 23:09:55.430227 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:55.427071 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7614-d8e390"] Oct 29 23:09:55.430227 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:55.427141 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7614-d8e390"] Oct 29 23:09:55.558997 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:55.558943 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7614-a395a4"] Oct 29 23:09:55.560239 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:55.560193 5057 cpu_manager.go:414] "containerMap: removing container" podUID="5d38cbfa-f5b9-4349-b380-a430e551e6f7" containerName="test-curl-7614-d8e390" Oct 29 23:09:55.560239 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:55.560216 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d38cbfa-f5b9-4349-b380-a430e551e6f7" containerName="test-curl-7614-d8e390" Oct 29 23:09:55.560459 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:55.560378 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d38cbfa-f5b9-4349-b380-a430e551e6f7" containerName="test-curl-7614-d8e390" Oct 29 23:09:55.573455 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:55.572854 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7614-a395a4" Oct 29 23:09:55.645714 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:55.645638 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4k5s\" (UniqueName: \"kubernetes.io/projected/13255730-1e05-48aa-ba2d-d9f184364158-kube-api-access-l4k5s\") pod \"test-curl-7614-a395a4\" (UID: \"13255730-1e05-48aa-ba2d-d9f184364158\") " pod="opendatahub/test-curl-7614-a395a4" Oct 29 23:09:55.652885 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:55.652833 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7614-a395a4"] Oct 29 23:09:55.666989 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:55.666424 5057 patch_prober.go:29] interesting pod/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-ksjql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body= Oct 29 23:09:55.666989 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:55.666464 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-ksjql" podUID="88b81626-da3c-4b1d-b9d8-3dbece193e80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" Oct 29 23:09:55.748065 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:55.748031 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4k5s\" (UniqueName: \"kubernetes.io/projected/13255730-1e05-48aa-ba2d-d9f184364158-kube-api-access-l4k5s\") pod \"test-curl-7614-a395a4\" (UID: \"13255730-1e05-48aa-ba2d-d9f184364158\") " pod="opendatahub/test-curl-7614-a395a4" Oct 29 23:09:55.768748 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:55.768722 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4k5s\" (UniqueName: \"kubernetes.io/projected/13255730-1e05-48aa-ba2d-d9f184364158-kube-api-access-l4k5s\") pod \"test-curl-7614-a395a4\" (UID: \"13255730-1e05-48aa-ba2d-d9f184364158\") " pod="opendatahub/test-curl-7614-a395a4" Oct 29 23:09:55.906356 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:55.902780 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:09:55.962268 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:55.960645 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7614-a395a4" Oct 29 23:09:56.430446 ip-10-0-140-74 kubenswrapper[5057]: W1029 23:09:56.429426 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod13255730_1e05_48aa_ba2d_d9f184364158.slice/crio-0bb6d7d66aefa9a90c38d42ae8a91f36267fae939136357e92051418b2bc05a6 WatchSource:0}: Error finding container 0bb6d7d66aefa9a90c38d42ae8a91f36267fae939136357e92051418b2bc05a6: Status 404 returned error can't find the container with id 0bb6d7d66aefa9a90c38d42ae8a91f36267fae939136357e92051418b2bc05a6 Oct 29 23:09:56.430636 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:56.430543 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7614-a395a4"] Oct 29 23:09:56.464293 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:56.464258 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7614-d8e390" podUID="5d38cbfa-f5b9-4349-b380-a430e551e6f7" containerName="test-curl-7614-d8e390" containerID="cri-o://d3c4c53d768566bc67696e7a4bb45f6651095700b527045ac8d7c8976b306828" gracePeriod=2 Oct 29 23:09:56.464588 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:56.464566 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7614-a395a4" event={"ID":"13255730-1e05-48aa-ba2d-d9f184364158","Type":"ContainerStarted","Data":"0bb6d7d66aefa9a90c38d42ae8a91f36267fae939136357e92051418b2bc05a6"} Oct 29 23:09:56.476176 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:09:56.476147 5057 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7614-d8e390\" is forbidden: User \"system:node:ip-10-0-140-74.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-74.ec2.internal' and this object" podUID="5d38cbfa-f5b9-4349-b380-a430e551e6f7" pod="opendatahub/test-curl-7614-d8e390" Oct 29 23:09:56.666006 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:56.665967 5057 patch_prober.go:29] interesting pod/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-ksjql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body= Oct 29 23:09:56.666432 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:56.666023 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-ksjql" podUID="88b81626-da3c-4b1d-b9d8-3dbece193e80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" Oct 29 23:09:56.712996 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:56.712958 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:09:56.822112 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:56.822074 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:09:57.472359 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:57.472291 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7614-a395a4" event={"ID":"13255730-1e05-48aa-ba2d-d9f184364158","Type":"ContainerStarted","Data":"113e1e9d278a945ccdf747085a9688ccd30139a22f4762507314f03f31503594"} Oct 29 23:09:57.517856 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:57.517796 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7614-a395a4" podStartSLOduration=2.517778583 podStartE2EDuration="2.517778583s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:09:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:09:57.517351205 +0000 UTC m=+2178.300564632" watchObservedRunningTime="2026-10-29 23:09:57.517778583 +0000 UTC m=+2178.300992009" Oct 29 23:09:57.616633 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:57.616594 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:09:57.666102 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:57.666066 5057 patch_prober.go:29] interesting pod/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-ksjql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body= Oct 29 23:09:57.666544 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:57.666127 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-ksjql" podUID="88b81626-da3c-4b1d-b9d8-3dbece193e80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" Oct 29 23:09:57.748063 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:57.747984 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:09:58.128734 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:58.128648 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:09:58.540422 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:58.531093 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7614-a395a4"] Oct 29 23:09:58.540422 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:58.537543 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7614-a395a4"] Oct 29 23:09:58.545344 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:58.544834 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:09:58.666369 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:58.665942 5057 patch_prober.go:29] interesting pod/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-ksjql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body= Oct 29 23:09:58.666369 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:58.665997 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-ksjql" podUID="88b81626-da3c-4b1d-b9d8-3dbece193e80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" Oct 29 23:09:58.668691 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:58.668224 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:09:58.730697 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:58.730656 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-65e2d3-5f9cb8568c-qqh8t"] Oct 29 23:09:58.731987 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:58.731962 5057 cpu_manager.go:414] "containerMap: removing container" podUID="13255730-1e05-48aa-ba2d-d9f184364158" containerName="test-curl-7614-a395a4" Oct 29 23:09:58.732149 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:58.731990 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13255730-1e05-48aa-ba2d-d9f184364158" containerName="test-curl-7614-a395a4" Oct 29 23:09:58.732221 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:58.732190 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="13255730-1e05-48aa-ba2d-d9f184364158" containerName="test-curl-7614-a395a4" Oct 29 23:09:58.770365 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:58.766863 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-65e2d3-5f9cb8568c-qqh8t"] Oct 29 23:09:58.770365 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:58.766978 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-65e2d3-5f9cb8568c-qqh8t" Oct 29 23:09:58.796356 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:58.794627 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w4-65e2d3-dockercfg-fgbmj\"" Oct 29 23:09:58.796356 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:58.794711 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w4-65e2d3\"" Oct 29 23:09:58.850821 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:58.850406 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6v6pp\" (UniqueName: \"kubernetes.io/projected/862fe3ac-f842-48d3-a722-ab1689671efd-kube-api-access-6v6pp\") pod \"payload-pre-processing-e2e-shared-b-w4-65e2d3-5f9cb8568c-qqh8t\" (UID: \"862fe3ac-f842-48d3-a722-ab1689671efd\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-65e2d3-5f9cb8568c-qqh8t" Oct 29 23:09:58.850821 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:58.850691 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/862fe3ac-f842-48d3-a722-ab1689671efd-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-65e2d3-5f9cb8568c-qqh8t\" (UID: \"862fe3ac-f842-48d3-a722-ab1689671efd\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-65e2d3-5f9cb8568c-qqh8t" Oct 29 23:09:58.972360 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:58.965454 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6v6pp\" (UniqueName: \"kubernetes.io/projected/862fe3ac-f842-48d3-a722-ab1689671efd-kube-api-access-6v6pp\") pod \"payload-pre-processing-e2e-shared-b-w4-65e2d3-5f9cb8568c-qqh8t\" (UID: \"862fe3ac-f842-48d3-a722-ab1689671efd\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-65e2d3-5f9cb8568c-qqh8t" Oct 29 23:09:58.972360 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:58.965814 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/862fe3ac-f842-48d3-a722-ab1689671efd-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-65e2d3-5f9cb8568c-qqh8t\" (UID: \"862fe3ac-f842-48d3-a722-ab1689671efd\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-65e2d3-5f9cb8568c-qqh8t" Oct 29 23:09:58.972360 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:58.966839 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/862fe3ac-f842-48d3-a722-ab1689671efd-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-65e2d3-5f9cb8568c-qqh8t\" (UID: \"862fe3ac-f842-48d3-a722-ab1689671efd\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-65e2d3-5f9cb8568c-qqh8t" Oct 29 23:09:59.156515 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:59.153688 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-65e2d3-b8f79d5d4-5f2dp"] Oct 29 23:09:59.160878 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:59.157830 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6v6pp\" (UniqueName: \"kubernetes.io/projected/862fe3ac-f842-48d3-a722-ab1689671efd-kube-api-access-6v6pp\") pod \"payload-pre-processing-e2e-shared-b-w4-65e2d3-5f9cb8568c-qqh8t\" (UID: \"862fe3ac-f842-48d3-a722-ab1689671efd\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-65e2d3-5f9cb8568c-qqh8t" Oct 29 23:09:59.160878 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:59.158889 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:09:59.160878 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:59.160534 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-65e2d3-b8f79d5d4-5f2dp" Oct 29 23:09:59.169757 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:59.169732 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7614-d8e390" Oct 29 23:09:59.169915 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:59.169894 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-65e2d3-b8f79d5d4-5f2dp"] Oct 29 23:09:59.249506 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:59.249470 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-65e2d3-5f9cb8568c-qqh8t" Oct 29 23:09:59.312990 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:59.298676 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d38cbfa-f5b9-4349-b380-a430e551e6f7-kube-api-access-8q2lv\" (UniqueName: \"kubernetes.io/projected/5d38cbfa-f5b9-4349-b380-a430e551e6f7-kube-api-access-8q2lv\") pod \"5d38cbfa-f5b9-4349-b380-a430e551e6f7\" (UID: \"5d38cbfa-f5b9-4349-b380-a430e551e6f7\") " Oct 29 23:09:59.312990 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:59.299445 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05cd9ea1-8a8c-4434-a32d-540ccf6bfd4c-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-65e2d3-b8f79d5d4-5f2dp\" (UID: \"05cd9ea1-8a8c-4434-a32d-540ccf6bfd4c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-65e2d3-b8f79d5d4-5f2dp" Oct 29 23:09:59.312990 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:59.299580 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52bc9\" (UniqueName: \"kubernetes.io/projected/05cd9ea1-8a8c-4434-a32d-540ccf6bfd4c-kube-api-access-52bc9\") pod \"payload-processing-e2e-shared-b-w4-65e2d3-b8f79d5d4-5f2dp\" (UID: \"05cd9ea1-8a8c-4434-a32d-540ccf6bfd4c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-65e2d3-b8f79d5d4-5f2dp" Oct 29 23:09:59.340845 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:59.340795 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d38cbfa-f5b9-4349-b380-a430e551e6f7-kube-api-access-8q2lv" pod "5d38cbfa-f5b9-4349-b380-a430e551e6f7" (UID: "5d38cbfa-f5b9-4349-b380-a430e551e6f7"). InnerVolumeSpecName "kube-api-access-8q2lv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:09:59.422354 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:59.412663 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05cd9ea1-8a8c-4434-a32d-540ccf6bfd4c-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-65e2d3-b8f79d5d4-5f2dp\" (UID: \"05cd9ea1-8a8c-4434-a32d-540ccf6bfd4c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-65e2d3-b8f79d5d4-5f2dp" Oct 29 23:09:59.422354 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:59.412822 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52bc9\" (UniqueName: \"kubernetes.io/projected/05cd9ea1-8a8c-4434-a32d-540ccf6bfd4c-kube-api-access-52bc9\") pod \"payload-processing-e2e-shared-b-w4-65e2d3-b8f79d5d4-5f2dp\" (UID: \"05cd9ea1-8a8c-4434-a32d-540ccf6bfd4c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-65e2d3-b8f79d5d4-5f2dp" Oct 29 23:09:59.422354 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:59.413154 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8q2lv\" (UniqueName: \"kubernetes.io/projected/5d38cbfa-f5b9-4349-b380-a430e551e6f7-kube-api-access-8q2lv\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:09:59.422354 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:59.414526 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05cd9ea1-8a8c-4434-a32d-540ccf6bfd4c-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-65e2d3-b8f79d5d4-5f2dp\" (UID: \"05cd9ea1-8a8c-4434-a32d-540ccf6bfd4c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-65e2d3-b8f79d5d4-5f2dp" Oct 29 23:09:59.440502 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:59.440279 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-52bc9\" (UniqueName: \"kubernetes.io/projected/05cd9ea1-8a8c-4434-a32d-540ccf6bfd4c-kube-api-access-52bc9\") pod \"payload-processing-e2e-shared-b-w4-65e2d3-b8f79d5d4-5f2dp\" (UID: \"05cd9ea1-8a8c-4434-a32d-540ccf6bfd4c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-65e2d3-b8f79d5d4-5f2dp" Oct 29 23:09:59.532821 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:59.532718 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-65e2d3-b8f79d5d4-5f2dp" Oct 29 23:09:59.545251 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:59.545216 5057 generic.go:356] "Generic (PLEG): container finished" podID="5d38cbfa-f5b9-4349-b380-a430e551e6f7" containerID="d3c4c53d768566bc67696e7a4bb45f6651095700b527045ac8d7c8976b306828" exitCode=137 Oct 29 23:09:59.545467 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:59.545443 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7614-a395a4" podUID="13255730-1e05-48aa-ba2d-d9f184364158" containerName="test-curl-7614-a395a4" containerID="cri-o://113e1e9d278a945ccdf747085a9688ccd30139a22f4762507314f03f31503594" gracePeriod=2 Oct 29 23:09:59.545821 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:59.545800 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7614-d8e390" Oct 29 23:09:59.555158 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:59.555137 5057 scope.go:122] "RemoveContainer" containerID="d3c4c53d768566bc67696e7a4bb45f6651095700b527045ac8d7c8976b306828" Oct 29 23:09:59.562844 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:59.562712 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:09:59.639912 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:59.621489 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-65e2d3-5f9cb8568c-qqh8t"] Oct 29 23:09:59.672767 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:59.672415 5057 patch_prober.go:29] interesting pod/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-ksjql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body= Oct 29 23:09:59.672767 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:59.672459 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-ksjql" podUID="88b81626-da3c-4b1d-b9d8-3dbece193e80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" Oct 29 23:09:59.827972 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:59.805698 5057 scope.go:122] "RemoveContainer" containerID="d3c4c53d768566bc67696e7a4bb45f6651095700b527045ac8d7c8976b306828" Oct 29 23:09:59.845096 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:09:59.828959 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3c4c53d768566bc67696e7a4bb45f6651095700b527045ac8d7c8976b306828\": container with ID starting with d3c4c53d768566bc67696e7a4bb45f6651095700b527045ac8d7c8976b306828 not found: ID does not exist" containerID="d3c4c53d768566bc67696e7a4bb45f6651095700b527045ac8d7c8976b306828" Oct 29 23:09:59.845096 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:59.829004 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3c4c53d768566bc67696e7a4bb45f6651095700b527045ac8d7c8976b306828"} err="failed to get container status \"d3c4c53d768566bc67696e7a4bb45f6651095700b527045ac8d7c8976b306828\": rpc error: code = NotFound desc = could not find container \"d3c4c53d768566bc67696e7a4bb45f6651095700b527045ac8d7c8976b306828\": container with ID starting with d3c4c53d768566bc67696e7a4bb45f6651095700b527045ac8d7c8976b306828 not found: ID does not exist" Oct 29 23:09:59.918530 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:09:59.914576 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v2tfm"] Oct 29 23:10:00.204883 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:00.204821 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-65e2d3-b8f79d5d4-5f2dp"] Oct 29 23:10:00.556478 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:00.556393 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-65e2d3-5f9cb8568c-qqh8t" event={"ID":"862fe3ac-f842-48d3-a722-ab1689671efd","Type":"ContainerStarted","Data":"9f448b248d837ec5c3071d5f5417d480c6a30bf44e29c77b3a8df4d5c28ae83d"} Oct 29 23:10:00.556478 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:00.556455 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-65e2d3-5f9cb8568c-qqh8t" event={"ID":"862fe3ac-f842-48d3-a722-ab1689671efd","Type":"ContainerStarted","Data":"0fd6930ee21e4670446c526403a4a59f4d24c7d06f434cd7ac7dc57ae4f2d897"} Oct 29 23:10:00.556717 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:00.556615 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-65e2d3-5f9cb8568c-qqh8t" Oct 29 23:10:00.559417 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:00.559361 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-65e2d3-b8f79d5d4-5f2dp" event={"ID":"05cd9ea1-8a8c-4434-a32d-540ccf6bfd4c","Type":"ContainerStarted","Data":"f8bc163c52a340bd35c7edafa3089678729d3c00c4ef0bbeaaa8595222e730be"} Oct 29 23:10:00.559417 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:00.559415 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-65e2d3-b8f79d5d4-5f2dp" event={"ID":"05cd9ea1-8a8c-4434-a32d-540ccf6bfd4c","Type":"ContainerStarted","Data":"7ad93ccee587571bbc568b875a5c35a1b7bfc62248b5cc832859f84dfc94170b"} Oct 29 23:10:00.559606 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:00.559518 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-65e2d3-b8f79d5d4-5f2dp" Oct 29 23:10:00.599959 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:00.599901 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-65e2d3-5f9cb8568c-qqh8t" podStartSLOduration=2.599886955 podStartE2EDuration="2.599886955s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:09:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:10:00.595969068 +0000 UTC m=+2181.379182494" watchObservedRunningTime="2026-10-29 23:10:00.599886955 +0000 UTC m=+2181.383100380" Oct 29 23:10:00.639219 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:00.639170 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-65e2d3-b8f79d5d4-5f2dp" podStartSLOduration=2.639152749 podStartE2EDuration="2.639152749s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:09:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:10:00.635400194 +0000 UTC m=+2181.418613634" watchObservedRunningTime="2026-10-29 23:10:00.639152749 +0000 UTC m=+2181.422366190" Oct 29 23:10:00.666416 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:00.666385 5057 patch_prober.go:29] interesting pod/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-ksjql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body= Oct 29 23:10:00.666577 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:00.666437 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-ksjql" podUID="88b81626-da3c-4b1d-b9d8-3dbece193e80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" Oct 29 23:10:01.070876 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:01.070836 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-67363b-745447b99f-jsvb4"] Oct 29 23:10:01.077254 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:01.077227 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-67363b-745447b99f-jsvb4" Oct 29 23:10:01.090199 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:01.090166 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-67363b-745447b99f-jsvb4"] Oct 29 23:10:01.104368 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:01.103125 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w0-67363b-dockercfg-z26ts\"" Oct 29 23:10:01.108057 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:01.108019 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w0-67363b\"" Oct 29 23:10:01.115529 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:01.115497 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-67363b-5d8768c84d-sgxgm"] Oct 29 23:10:01.121932 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:01.121911 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-67363b-5d8768c84d-sgxgm" Oct 29 23:10:01.151695 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:01.149852 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-67363b-5d8768c84d-sgxgm"] Oct 29 23:10:01.173858 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:01.173815 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r69mw\" (UniqueName: \"kubernetes.io/projected/59602ea3-b37d-479e-96f3-9f72411920b4-kube-api-access-r69mw\") pod \"payload-pre-processing-e2e-shared-b-w0-67363b-745447b99f-jsvb4\" (UID: \"59602ea3-b37d-479e-96f3-9f72411920b4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-67363b-745447b99f-jsvb4" Oct 29 23:10:01.174146 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:01.174127 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59602ea3-b37d-479e-96f3-9f72411920b4-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-67363b-745447b99f-jsvb4\" (UID: \"59602ea3-b37d-479e-96f3-9f72411920b4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-67363b-745447b99f-jsvb4" Oct 29 23:10:01.338232 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:01.331738 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9gj4\" (UniqueName: \"kubernetes.io/projected/f15821b0-cb92-4759-aba2-a4fe6f8e62f9-kube-api-access-z9gj4\") pod \"payload-processing-e2e-shared-b-w0-67363b-5d8768c84d-sgxgm\" (UID: \"f15821b0-cb92-4759-aba2-a4fe6f8e62f9\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-67363b-5d8768c84d-sgxgm" Oct 29 23:10:01.342696 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:01.342506 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r69mw\" (UniqueName: \"kubernetes.io/projected/59602ea3-b37d-479e-96f3-9f72411920b4-kube-api-access-r69mw\") pod \"payload-pre-processing-e2e-shared-b-w0-67363b-745447b99f-jsvb4\" (UID: \"59602ea3-b37d-479e-96f3-9f72411920b4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-67363b-745447b99f-jsvb4" Oct 29 23:10:01.351356 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:01.343236 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59602ea3-b37d-479e-96f3-9f72411920b4-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-67363b-745447b99f-jsvb4\" (UID: \"59602ea3-b37d-479e-96f3-9f72411920b4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-67363b-745447b99f-jsvb4" Oct 29 23:10:01.580836 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:01.575600 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f15821b0-cb92-4759-aba2-a4fe6f8e62f9-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-67363b-5d8768c84d-sgxgm\" (UID: \"f15821b0-cb92-4759-aba2-a4fe6f8e62f9\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-67363b-5d8768c84d-sgxgm" Oct 29 23:10:01.580836 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:01.473458 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:10:01.580836 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:01.415909 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59602ea3-b37d-479e-96f3-9f72411920b4-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-67363b-745447b99f-jsvb4\" (UID: \"59602ea3-b37d-479e-96f3-9f72411920b4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-67363b-745447b99f-jsvb4" Oct 29 23:10:01.690577 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:01.690543 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r69mw\" (UniqueName: \"kubernetes.io/projected/59602ea3-b37d-479e-96f3-9f72411920b4-kube-api-access-r69mw\") pod \"payload-pre-processing-e2e-shared-b-w0-67363b-745447b99f-jsvb4\" (UID: \"59602ea3-b37d-479e-96f3-9f72411920b4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-67363b-745447b99f-jsvb4" Oct 29 23:10:01.725651 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:01.697847 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d38cbfa-f5b9-4349-b380-a430e551e6f7" path="/var/lib/kubelet/pods/5d38cbfa-f5b9-4349-b380-a430e551e6f7/volumes" Oct 29 23:10:01.725651 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:01.721613 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-67363b-745447b99f-jsvb4" Oct 29 23:10:01.801399 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:01.785833 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f15821b0-cb92-4759-aba2-a4fe6f8e62f9-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-67363b-5d8768c84d-sgxgm\" (UID: \"f15821b0-cb92-4759-aba2-a4fe6f8e62f9\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-67363b-5d8768c84d-sgxgm" Oct 29 23:10:01.801399 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:01.785922 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9gj4\" (UniqueName: \"kubernetes.io/projected/f15821b0-cb92-4759-aba2-a4fe6f8e62f9-kube-api-access-z9gj4\") pod \"payload-processing-e2e-shared-b-w0-67363b-5d8768c84d-sgxgm\" (UID: \"f15821b0-cb92-4759-aba2-a4fe6f8e62f9\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-67363b-5d8768c84d-sgxgm" Oct 29 23:10:01.801399 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:01.787164 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f15821b0-cb92-4759-aba2-a4fe6f8e62f9-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-67363b-5d8768c84d-sgxgm\" (UID: \"f15821b0-cb92-4759-aba2-a4fe6f8e62f9\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-67363b-5d8768c84d-sgxgm" Oct 29 23:10:01.838389 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:01.838145 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9gj4\" (UniqueName: \"kubernetes.io/projected/f15821b0-cb92-4759-aba2-a4fe6f8e62f9-kube-api-access-z9gj4\") pod \"payload-processing-e2e-shared-b-w0-67363b-5d8768c84d-sgxgm\" (UID: \"f15821b0-cb92-4759-aba2-a4fe6f8e62f9\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-67363b-5d8768c84d-sgxgm" Oct 29 23:10:02.115411 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:02.115379 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-67363b-5d8768c84d-sgxgm" Oct 29 23:10:02.485726 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:02.485697 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-ksjql" Oct 29 23:10:02.491751 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:02.490005 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-ksjql" Oct 29 23:10:02.492937 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:02.492907 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-ksjql" Oct 29 23:10:02.536801 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:02.527206 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-67363b-745447b99f-jsvb4"] Oct 29 23:10:02.859358 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:02.833574 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7614-a395a4" Oct 29 23:10:02.956365 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:02.942999 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13255730-1e05-48aa-ba2d-d9f184364158-kube-api-access-l4k5s\" (UniqueName: \"kubernetes.io/projected/13255730-1e05-48aa-ba2d-d9f184364158-kube-api-access-l4k5s\") pod \"13255730-1e05-48aa-ba2d-d9f184364158\" (UID: \"13255730-1e05-48aa-ba2d-d9f184364158\") " Oct 29 23:10:03.247526 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:03.239147 5057 generic.go:356] "Generic (PLEG): container finished" podID="13255730-1e05-48aa-ba2d-d9f184364158" containerID="113e1e9d278a945ccdf747085a9688ccd30139a22f4762507314f03f31503594" exitCode=137 Oct 29 23:10:03.247526 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:03.239367 5057 scope.go:122] "RemoveContainer" containerID="113e1e9d278a945ccdf747085a9688ccd30139a22f4762507314f03f31503594" Oct 29 23:10:03.247526 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:03.239488 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7614-a395a4" Oct 29 23:10:03.317346 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:03.314463 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-67363b-745447b99f-jsvb4" event={"ID":"59602ea3-b37d-479e-96f3-9f72411920b4","Type":"ContainerStarted","Data":"73c3f36e7e8c5454693b045da0b65f798aced1ac46537126506718e4f00dac9c"} Oct 29 23:10:03.492381 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:03.489955 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13255730-1e05-48aa-ba2d-d9f184364158-kube-api-access-l4k5s" pod "13255730-1e05-48aa-ba2d-d9f184364158" (UID: "13255730-1e05-48aa-ba2d-d9f184364158"). InnerVolumeSpecName "kube-api-access-l4k5s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:10:03.572518 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:03.532806 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l4k5s\" (UniqueName: \"kubernetes.io/projected/13255730-1e05-48aa-ba2d-d9f184364158-kube-api-access-l4k5s\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:10:03.769178 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:03.756012 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="13255730-1e05-48aa-ba2d-d9f184364158" path="/var/lib/kubelet/pods/13255730-1e05-48aa-ba2d-d9f184364158/volumes" Oct 29 23:10:03.856135 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:03.856099 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-67363b-5d8768c84d-sgxgm"] Oct 29 23:10:04.013389 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:04.010798 5057 scope.go:122] "RemoveContainer" containerID="113e1e9d278a945ccdf747085a9688ccd30139a22f4762507314f03f31503594" Oct 29 23:10:04.016611 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:10:04.016528 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"113e1e9d278a945ccdf747085a9688ccd30139a22f4762507314f03f31503594\": container with ID starting with 113e1e9d278a945ccdf747085a9688ccd30139a22f4762507314f03f31503594 not found: ID does not exist" containerID="113e1e9d278a945ccdf747085a9688ccd30139a22f4762507314f03f31503594" Oct 29 23:10:04.016611 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:04.016568 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"113e1e9d278a945ccdf747085a9688ccd30139a22f4762507314f03f31503594"} err="failed to get container status \"113e1e9d278a945ccdf747085a9688ccd30139a22f4762507314f03f31503594\": rpc error: code = NotFound desc = could not find container \"113e1e9d278a945ccdf747085a9688ccd30139a22f4762507314f03f31503594\": container with ID starting with 113e1e9d278a945ccdf747085a9688ccd30139a22f4762507314f03f31503594 not found: ID does not exist" Oct 29 23:10:04.572559 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:04.572523 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-67363b-745447b99f-jsvb4" event={"ID":"59602ea3-b37d-479e-96f3-9f72411920b4","Type":"ContainerStarted","Data":"b4700e6140c401d453d38a5d21981223b3514db88f89cc8d87eeaf127e8f19b0"} Oct 29 23:10:04.574298 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:04.574252 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-67363b-745447b99f-jsvb4" Oct 29 23:10:04.585497 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:04.585468 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-67363b-5d8768c84d-sgxgm" event={"ID":"f15821b0-cb92-4759-aba2-a4fe6f8e62f9","Type":"ContainerStarted","Data":"5f1308eb98a7b994261e7d61d774565cbda997c5e70dfe4f455ec298930d3d8e"} Oct 29 23:10:04.650352 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:04.635035 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-67363b-745447b99f-jsvb4" podStartSLOduration=3.635016875 podStartE2EDuration="3.635016875s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:10:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:10:04.62477099 +0000 UTC m=+2185.407984417" watchObservedRunningTime="2026-10-29 23:10:04.635016875 +0000 UTC m=+2185.418230302" Oct 29 23:10:05.596786 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:05.596748 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-67363b-5d8768c84d-sgxgm" event={"ID":"f15821b0-cb92-4759-aba2-a4fe6f8e62f9","Type":"ContainerStarted","Data":"a8626ca70853027be2afbd044099cce1be94a720312d395c38ab234d25fa908d"} Oct 29 23:10:05.597271 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:05.596859 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-67363b-5d8768c84d-sgxgm" Oct 29 23:10:05.641972 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:05.641919 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-67363b-5d8768c84d-sgxgm" podStartSLOduration=4.64190461 podStartE2EDuration="4.64190461s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:10:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:10:05.637620983 +0000 UTC m=+2186.420834410" watchObservedRunningTime="2026-10-29 23:10:05.64190461 +0000 UTC m=+2186.425118036" Oct 29 23:10:05.775700 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:05.775658 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:10:05.910262 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:05.910165 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:10:06.195616 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:06.195482 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:10:07.630261 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:07.630224 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:10:07.744443 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:07.744407 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:10:07.790847 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:07.790809 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-3e5b21-d7b5c69c5-ds82z"] Oct 29 23:10:07.797146 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:07.797121 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-3e5b21-d7b5c69c5-ds82z" Oct 29 23:10:07.815650 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:07.815612 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-3e5b21-d7b5c69c5-ds82z"] Oct 29 23:10:07.819504 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:07.819479 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w6-3e5b21\"" Oct 29 23:10:07.819770 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:07.819748 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w6-3e5b21-dockercfg-w2t47\"" Oct 29 23:10:07.885975 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:07.885883 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e05f5f87-b9fc-4dfa-b61a-ce1f776e146c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w6-3e5b21-d7b5c69c5-ds82z\" (UID: \"e05f5f87-b9fc-4dfa-b61a-ce1f776e146c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-3e5b21-d7b5c69c5-ds82z" Oct 29 23:10:07.886428 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:07.886398 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2c7l\" (UniqueName: \"kubernetes.io/projected/e05f5f87-b9fc-4dfa-b61a-ce1f776e146c-kube-api-access-d2c7l\") pod \"payload-pre-processing-e2e-shared-a-w6-3e5b21-d7b5c69c5-ds82z\" (UID: \"e05f5f87-b9fc-4dfa-b61a-ce1f776e146c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-3e5b21-d7b5c69c5-ds82z" Oct 29 23:10:07.914356 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:07.909403 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-3e5b21-f69958676-g78g2"] Oct 29 23:10:07.919359 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:07.917874 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-3e5b21-f69958676-g78g2" Oct 29 23:10:07.947352 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:07.944224 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:10:07.947352 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:07.944912 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-3e5b21-f69958676-g78g2"] Oct 29 23:10:07.996363 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:07.988356 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdtwn\" (UniqueName: \"kubernetes.io/projected/0157b7d8-b442-4302-9fc2-4e87e55ad7aa-kube-api-access-xdtwn\") pod \"payload-processing-e2e-shared-a-w6-3e5b21-f69958676-g78g2\" (UID: \"0157b7d8-b442-4302-9fc2-4e87e55ad7aa\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-3e5b21-f69958676-g78g2" Oct 29 23:10:07.996363 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:07.988412 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0157b7d8-b442-4302-9fc2-4e87e55ad7aa-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w6-3e5b21-f69958676-g78g2\" (UID: \"0157b7d8-b442-4302-9fc2-4e87e55ad7aa\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-3e5b21-f69958676-g78g2" Oct 29 23:10:07.996363 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:07.988472 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2c7l\" (UniqueName: \"kubernetes.io/projected/e05f5f87-b9fc-4dfa-b61a-ce1f776e146c-kube-api-access-d2c7l\") pod \"payload-pre-processing-e2e-shared-a-w6-3e5b21-d7b5c69c5-ds82z\" (UID: \"e05f5f87-b9fc-4dfa-b61a-ce1f776e146c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-3e5b21-d7b5c69c5-ds82z" Oct 29 23:10:07.996363 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:07.988623 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e05f5f87-b9fc-4dfa-b61a-ce1f776e146c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w6-3e5b21-d7b5c69c5-ds82z\" (UID: \"e05f5f87-b9fc-4dfa-b61a-ce1f776e146c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-3e5b21-d7b5c69c5-ds82z" Oct 29 23:10:07.996363 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:07.991077 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e05f5f87-b9fc-4dfa-b61a-ce1f776e146c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w6-3e5b21-d7b5c69c5-ds82z\" (UID: \"e05f5f87-b9fc-4dfa-b61a-ce1f776e146c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-3e5b21-d7b5c69c5-ds82z" Oct 29 23:10:08.024163 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:08.024126 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2c7l\" (UniqueName: \"kubernetes.io/projected/e05f5f87-b9fc-4dfa-b61a-ce1f776e146c-kube-api-access-d2c7l\") pod \"payload-pre-processing-e2e-shared-a-w6-3e5b21-d7b5c69c5-ds82z\" (UID: \"e05f5f87-b9fc-4dfa-b61a-ce1f776e146c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-3e5b21-d7b5c69c5-ds82z" Oct 29 23:10:08.091129 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:08.091094 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdtwn\" (UniqueName: \"kubernetes.io/projected/0157b7d8-b442-4302-9fc2-4e87e55ad7aa-kube-api-access-xdtwn\") pod \"payload-processing-e2e-shared-a-w6-3e5b21-f69958676-g78g2\" (UID: \"0157b7d8-b442-4302-9fc2-4e87e55ad7aa\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-3e5b21-f69958676-g78g2" Oct 29 23:10:08.091129 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:08.091135 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0157b7d8-b442-4302-9fc2-4e87e55ad7aa-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w6-3e5b21-f69958676-g78g2\" (UID: \"0157b7d8-b442-4302-9fc2-4e87e55ad7aa\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-3e5b21-f69958676-g78g2" Oct 29 23:10:08.091950 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:08.091926 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0157b7d8-b442-4302-9fc2-4e87e55ad7aa-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w6-3e5b21-f69958676-g78g2\" (UID: \"0157b7d8-b442-4302-9fc2-4e87e55ad7aa\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-3e5b21-f69958676-g78g2" Oct 29 23:10:08.114579 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:08.114548 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdtwn\" (UniqueName: \"kubernetes.io/projected/0157b7d8-b442-4302-9fc2-4e87e55ad7aa-kube-api-access-xdtwn\") pod \"payload-processing-e2e-shared-a-w6-3e5b21-f69958676-g78g2\" (UID: \"0157b7d8-b442-4302-9fc2-4e87e55ad7aa\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-3e5b21-f69958676-g78g2" Oct 29 23:10:08.115387 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:08.115356 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-3e5b21-d7b5c69c5-ds82z" Oct 29 23:10:08.262249 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:08.262211 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-3e5b21-f69958676-g78g2" Oct 29 23:10:08.401351 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:08.395957 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:10:08.444175 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:08.444098 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-3e5b21-d7b5c69c5-ds82z"] Oct 29 23:10:08.627416 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:08.627373 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-3e5b21-d7b5c69c5-ds82z" event={"ID":"e05f5f87-b9fc-4dfa-b61a-ce1f776e146c","Type":"ContainerStarted","Data":"91078cc6e3795c4ac92f4e63a8fc9158442f9ae23309168a63e15a221a4d9edd"} Oct 29 23:10:08.627567 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:08.627433 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-3e5b21-d7b5c69c5-ds82z" event={"ID":"e05f5f87-b9fc-4dfa-b61a-ce1f776e146c","Type":"ContainerStarted","Data":"d1f91f973ed09cc5b0829fb1b29a7e8f36ecbb581d417c392f32529f2ec7bb54"} Oct 29 23:10:08.627567 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:08.627493 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-3e5b21-d7b5c69c5-ds82z" Oct 29 23:10:08.655444 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:08.655417 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-3e5b21-f69958676-g78g2"] Oct 29 23:10:08.656434 ip-10-0-140-74 kubenswrapper[5057]: W1029 23:10:08.656410 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0157b7d8_b442_4302_9fc2_4e87e55ad7aa.slice/crio-86642ea1ec030e84cb937fbdbacf3cb366591e581de8ece19dbfca0a3c7d155b WatchSource:0}: Error finding container 86642ea1ec030e84cb937fbdbacf3cb366591e581de8ece19dbfca0a3c7d155b: Status 404 returned error can't find the container with id 86642ea1ec030e84cb937fbdbacf3cb366591e581de8ece19dbfca0a3c7d155b Oct 29 23:10:08.671456 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:08.671408 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-3e5b21-d7b5c69c5-ds82z" podStartSLOduration=1.6713956589999999 podStartE2EDuration="1.671395659s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:10:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:10:08.666819921 +0000 UTC m=+2189.450033365" watchObservedRunningTime="2026-10-29 23:10:08.671395659 +0000 UTC m=+2189.454609085" Oct 29 23:10:09.289577 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:09.289542 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:10:09.488736 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:09.488696 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:10:09.644539 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:09.644453 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-3e5b21-f69958676-g78g2" event={"ID":"0157b7d8-b442-4302-9fc2-4e87e55ad7aa","Type":"ContainerStarted","Data":"8aaf466ab895e32406715310466b10620eeaa03863836a83b5f1fc50ad9c378b"} Oct 29 23:10:09.644919 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:09.644894 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-3e5b21-f69958676-g78g2" Oct 29 23:10:09.645023 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:09.644937 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-3e5b21-f69958676-g78g2" event={"ID":"0157b7d8-b442-4302-9fc2-4e87e55ad7aa","Type":"ContainerStarted","Data":"86642ea1ec030e84cb937fbdbacf3cb366591e581de8ece19dbfca0a3c7d155b"} Oct 29 23:10:09.699562 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:09.699496 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-3e5b21-f69958676-g78g2" podStartSLOduration=2.699472811 podStartE2EDuration="2.699472811s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:10:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:10:09.687821287 +0000 UTC m=+2190.471034726" watchObservedRunningTime="2026-10-29 23:10:09.699472811 +0000 UTC m=+2190.482686237" Oct 29 23:10:10.579742 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:10.579706 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:10:11.837541 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:11.837513 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-65e2d3-b8f79d5d4-5f2dp" Oct 29 23:10:11.838064 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:11.838039 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-65e2d3-5f9cb8568c-qqh8t" Oct 29 23:10:12.106204 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:12.106109 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-9nnj5"] Oct 29 23:10:12.106384 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:12.106305 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-9nnj5" podUID="c237fe02-4087-448d-b7b1-c7e03f1872c3" containerName="istio-proxy" containerID="cri-o://a365358bd2744c25f8081adcaeaf4f38c39c9b3d8fda8100cd1a6497beec85e5" gracePeriod=30 Oct 29 23:10:13.235846 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:13.232383 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-9j6rn"] Oct 29 23:10:13.271279 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:13.271246 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-9j6rn"] Oct 29 23:10:13.271485 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:13.271371 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-9j6rn" Oct 29 23:10:13.333987 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:13.333961 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-a9ced3-openshift-default-dockercfg-mhf8n\"" Oct 29 23:10:13.389214 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:13.389177 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:10:13.407275 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:13.407244 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f521c155-eff0-4b0b-b4cd-e55abde4ed41-istio-envoy\") pod \"e2e-autores-a9ced3-openshift-default-7d8898bf57-9j6rn\" (UID: \"f521c155-eff0-4b0b-b4cd-e55abde4ed41\") " pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-9j6rn" Oct 29 23:10:13.407582 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:13.407558 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f521c155-eff0-4b0b-b4cd-e55abde4ed41-workload-certs\") pod \"e2e-autores-a9ced3-openshift-default-7d8898bf57-9j6rn\" (UID: \"f521c155-eff0-4b0b-b4cd-e55abde4ed41\") " pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-9j6rn" Oct 29 23:10:13.407685 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:13.407593 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gcwbw\" (UniqueName: \"kubernetes.io/projected/f521c155-eff0-4b0b-b4cd-e55abde4ed41-kube-api-access-gcwbw\") pod \"e2e-autores-a9ced3-openshift-default-7d8898bf57-9j6rn\" (UID: \"f521c155-eff0-4b0b-b4cd-e55abde4ed41\") " pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-9j6rn" Oct 29 23:10:13.407811 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:13.407771 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f521c155-eff0-4b0b-b4cd-e55abde4ed41-istio-data\") pod \"e2e-autores-a9ced3-openshift-default-7d8898bf57-9j6rn\" (UID: \"f521c155-eff0-4b0b-b4cd-e55abde4ed41\") " pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-9j6rn" Oct 29 23:10:13.407885 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:13.407828 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f521c155-eff0-4b0b-b4cd-e55abde4ed41-credential-socket\") pod \"e2e-autores-a9ced3-openshift-default-7d8898bf57-9j6rn\" (UID: \"f521c155-eff0-4b0b-b4cd-e55abde4ed41\") " pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-9j6rn" Oct 29 23:10:13.407885 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:13.407872 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f521c155-eff0-4b0b-b4cd-e55abde4ed41-istio-token\") pod \"e2e-autores-a9ced3-openshift-default-7d8898bf57-9j6rn\" (UID: \"f521c155-eff0-4b0b-b4cd-e55abde4ed41\") " pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-9j6rn" Oct 29 23:10:13.408011 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:13.407995 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f521c155-eff0-4b0b-b4cd-e55abde4ed41-workload-socket\") pod \"e2e-autores-a9ced3-openshift-default-7d8898bf57-9j6rn\" (UID: \"f521c155-eff0-4b0b-b4cd-e55abde4ed41\") " pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-9j6rn" Oct 29 23:10:13.408077 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:13.408061 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f521c155-eff0-4b0b-b4cd-e55abde4ed41-istiod-ca-cert\") pod \"e2e-autores-a9ced3-openshift-default-7d8898bf57-9j6rn\" (UID: \"f521c155-eff0-4b0b-b4cd-e55abde4ed41\") " pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-9j6rn" Oct 29 23:10:13.408137 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:13.408123 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f521c155-eff0-4b0b-b4cd-e55abde4ed41-istio-podinfo\") pod \"e2e-autores-a9ced3-openshift-default-7d8898bf57-9j6rn\" (UID: \"f521c155-eff0-4b0b-b4cd-e55abde4ed41\") " pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-9j6rn" Oct 29 23:10:13.509750 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:13.509652 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f521c155-eff0-4b0b-b4cd-e55abde4ed41-istio-envoy\") pod \"e2e-autores-a9ced3-openshift-default-7d8898bf57-9j6rn\" (UID: \"f521c155-eff0-4b0b-b4cd-e55abde4ed41\") " pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-9j6rn" Oct 29 23:10:13.509750 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:13.509732 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f521c155-eff0-4b0b-b4cd-e55abde4ed41-workload-certs\") pod \"e2e-autores-a9ced3-openshift-default-7d8898bf57-9j6rn\" (UID: \"f521c155-eff0-4b0b-b4cd-e55abde4ed41\") " pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-9j6rn" Oct 29 23:10:13.509981 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:13.509758 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gcwbw\" (UniqueName: \"kubernetes.io/projected/f521c155-eff0-4b0b-b4cd-e55abde4ed41-kube-api-access-gcwbw\") pod \"e2e-autores-a9ced3-openshift-default-7d8898bf57-9j6rn\" (UID: \"f521c155-eff0-4b0b-b4cd-e55abde4ed41\") " pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-9j6rn" Oct 29 23:10:13.509981 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:13.509859 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f521c155-eff0-4b0b-b4cd-e55abde4ed41-istio-data\") pod \"e2e-autores-a9ced3-openshift-default-7d8898bf57-9j6rn\" (UID: \"f521c155-eff0-4b0b-b4cd-e55abde4ed41\") " pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-9j6rn" Oct 29 23:10:13.509981 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:13.509903 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f521c155-eff0-4b0b-b4cd-e55abde4ed41-credential-socket\") pod \"e2e-autores-a9ced3-openshift-default-7d8898bf57-9j6rn\" (UID: \"f521c155-eff0-4b0b-b4cd-e55abde4ed41\") " pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-9j6rn" Oct 29 23:10:13.509981 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:13.509943 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f521c155-eff0-4b0b-b4cd-e55abde4ed41-istio-token\") pod \"e2e-autores-a9ced3-openshift-default-7d8898bf57-9j6rn\" (UID: \"f521c155-eff0-4b0b-b4cd-e55abde4ed41\") " pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-9j6rn" Oct 29 23:10:13.510187 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:13.510077 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f521c155-eff0-4b0b-b4cd-e55abde4ed41-workload-socket\") pod \"e2e-autores-a9ced3-openshift-default-7d8898bf57-9j6rn\" (UID: \"f521c155-eff0-4b0b-b4cd-e55abde4ed41\") " pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-9j6rn" Oct 29 23:10:13.510939 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:13.510761 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f521c155-eff0-4b0b-b4cd-e55abde4ed41-credential-socket\") pod \"e2e-autores-a9ced3-openshift-default-7d8898bf57-9j6rn\" (UID: \"f521c155-eff0-4b0b-b4cd-e55abde4ed41\") " pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-9j6rn" Oct 29 23:10:13.511080 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:13.510813 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f521c155-eff0-4b0b-b4cd-e55abde4ed41-workload-socket\") pod \"e2e-autores-a9ced3-openshift-default-7d8898bf57-9j6rn\" (UID: \"f521c155-eff0-4b0b-b4cd-e55abde4ed41\") " pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-9j6rn" Oct 29 23:10:13.511142 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:13.511094 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f521c155-eff0-4b0b-b4cd-e55abde4ed41-istiod-ca-cert\") pod \"e2e-autores-a9ced3-openshift-default-7d8898bf57-9j6rn\" (UID: \"f521c155-eff0-4b0b-b4cd-e55abde4ed41\") " pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-9j6rn" Oct 29 23:10:13.511252 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:13.511228 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f521c155-eff0-4b0b-b4cd-e55abde4ed41-workload-certs\") pod \"e2e-autores-a9ced3-openshift-default-7d8898bf57-9j6rn\" (UID: \"f521c155-eff0-4b0b-b4cd-e55abde4ed41\") " pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-9j6rn" Oct 29 23:10:13.511380 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:13.511362 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f521c155-eff0-4b0b-b4cd-e55abde4ed41-istio-podinfo\") pod \"e2e-autores-a9ced3-openshift-default-7d8898bf57-9j6rn\" (UID: \"f521c155-eff0-4b0b-b4cd-e55abde4ed41\") " pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-9j6rn" Oct 29 23:10:13.511887 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:13.511832 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f521c155-eff0-4b0b-b4cd-e55abde4ed41-istiod-ca-cert\") pod \"e2e-autores-a9ced3-openshift-default-7d8898bf57-9j6rn\" (UID: \"f521c155-eff0-4b0b-b4cd-e55abde4ed41\") " pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-9j6rn" Oct 29 23:10:13.512220 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:13.512198 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f521c155-eff0-4b0b-b4cd-e55abde4ed41-istio-data\") pod \"e2e-autores-a9ced3-openshift-default-7d8898bf57-9j6rn\" (UID: \"f521c155-eff0-4b0b-b4cd-e55abde4ed41\") " pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-9j6rn" Oct 29 23:10:13.513366 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:13.513344 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f521c155-eff0-4b0b-b4cd-e55abde4ed41-istio-envoy\") pod \"e2e-autores-a9ced3-openshift-default-7d8898bf57-9j6rn\" (UID: \"f521c155-eff0-4b0b-b4cd-e55abde4ed41\") " pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-9j6rn" Oct 29 23:10:13.513807 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:13.513789 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f521c155-eff0-4b0b-b4cd-e55abde4ed41-istio-podinfo\") pod \"e2e-autores-a9ced3-openshift-default-7d8898bf57-9j6rn\" (UID: \"f521c155-eff0-4b0b-b4cd-e55abde4ed41\") " pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-9j6rn" Oct 29 23:10:13.533688 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:13.533657 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f521c155-eff0-4b0b-b4cd-e55abde4ed41-istio-token\") pod \"e2e-autores-a9ced3-openshift-default-7d8898bf57-9j6rn\" (UID: \"f521c155-eff0-4b0b-b4cd-e55abde4ed41\") " pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-9j6rn" Oct 29 23:10:13.533939 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:13.533919 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gcwbw\" (UniqueName: \"kubernetes.io/projected/f521c155-eff0-4b0b-b4cd-e55abde4ed41-kube-api-access-gcwbw\") pod \"e2e-autores-a9ced3-openshift-default-7d8898bf57-9j6rn\" (UID: \"f521c155-eff0-4b0b-b4cd-e55abde4ed41\") " pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-9j6rn" Oct 29 23:10:13.597858 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:13.597818 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-9j6rn" Oct 29 23:10:13.952473 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:13.952445 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-9j6rn"] Oct 29 23:10:13.952921 ip-10-0-140-74 kubenswrapper[5057]: W1029 23:10:13.952892 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf521c155_eff0_4b0b_b4cd_e55abde4ed41.slice/crio-4bfb4774928800727c634a9ae215db57ae3f072633424f4ae11b5c5abfd25ddd WatchSource:0}: Error finding container 4bfb4774928800727c634a9ae215db57ae3f072633424f4ae11b5c5abfd25ddd: Status 404 returned error can't find the container with id 4bfb4774928800727c634a9ae215db57ae3f072633424f4ae11b5c5abfd25ddd Oct 29 23:10:13.956516 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:13.956480 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 23:10:13.956626 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:13.956566 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 23:10:13.956626 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:13.956612 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 23:10:14.724314 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:14.724279 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-9j6rn" event={"ID":"f521c155-eff0-4b0b-b4cd-e55abde4ed41","Type":"ContainerStarted","Data":"677b5bd71da6734023894e303e2563a729dc8609d8a2fd6a966cb4bc19e3c3db"} Oct 29 23:10:14.724851 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:14.724380 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-9j6rn" event={"ID":"f521c155-eff0-4b0b-b4cd-e55abde4ed41","Type":"ContainerStarted","Data":"4bfb4774928800727c634a9ae215db57ae3f072633424f4ae11b5c5abfd25ddd"} Oct 29 23:10:14.804352 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:14.804194 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-9j6rn" podStartSLOduration=1.804172976 podStartE2EDuration="1.804172976s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:10:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:10:14.792838369 +0000 UTC m=+2195.576051798" watchObservedRunningTime="2026-10-29 23:10:14.804172976 +0000 UTC m=+2195.587386403" Oct 29 23:10:15.241860 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:15.241826 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:10:15.607467 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:15.606515 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-9j6rn" Oct 29 23:10:15.609045 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:15.608994 5057 patch_prober.go:29] interesting pod/e2e-autores-a9ced3-openshift-default-7d8898bf57-9j6rn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" start-of-body= Oct 29 23:10:15.609209 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:15.609068 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-9j6rn" podUID="f521c155-eff0-4b0b-b4cd-e55abde4ed41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" Oct 29 23:10:16.607357 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:16.604993 5057 patch_prober.go:29] interesting pod/e2e-autores-a9ced3-openshift-default-7d8898bf57-9j6rn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" start-of-body= Oct 29 23:10:16.607357 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:16.605053 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-9j6rn" podUID="f521c155-eff0-4b0b-b4cd-e55abde4ed41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" Oct 29 23:10:16.612189 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:16.612162 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-67363b-745447b99f-jsvb4" Oct 29 23:10:16.612695 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:16.612675 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-67363b-5d8768c84d-sgxgm" Oct 29 23:10:16.741880 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:16.741326 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:10:16.747354 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:16.745267 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-62lhb"] Oct 29 23:10:16.755388 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:16.754209 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-62lhb" Oct 29 23:10:16.755388 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:16.754697 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-62lhb"] Oct 29 23:10:16.760408 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:16.759801 5057 patch_prober.go:29] interesting pod/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-9nnj5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 23:10:16.760408 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:16.759844 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-9nnj5" podUID="c237fe02-4087-448d-b7b1-c7e03f1872c3" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 23:10:16.816673 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:16.806286 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w4-af237d-openshift-default-dockercfg-9g5k4\"" Oct 29 23:10:16.856396 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:16.856361 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:10:16.928450 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:16.926407 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21f95b5c-4c31-4836-a8ba-41dc2949edea-credential-socket\") pod \"e2e-shared-a-w4-af237d-openshift-default-64f945b797-62lhb\" (UID: \"21f95b5c-4c31-4836-a8ba-41dc2949edea\") " pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-62lhb" Oct 29 23:10:16.928450 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:16.927464 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21f95b5c-4c31-4836-a8ba-41dc2949edea-istio-token\") pod \"e2e-shared-a-w4-af237d-openshift-default-64f945b797-62lhb\" (UID: \"21f95b5c-4c31-4836-a8ba-41dc2949edea\") " pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-62lhb" Oct 29 23:10:16.928450 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:16.927530 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21f95b5c-4c31-4836-a8ba-41dc2949edea-workload-certs\") pod \"e2e-shared-a-w4-af237d-openshift-default-64f945b797-62lhb\" (UID: \"21f95b5c-4c31-4836-a8ba-41dc2949edea\") " pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-62lhb" Oct 29 23:10:16.928450 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:16.927652 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrkfn\" (UniqueName: \"kubernetes.io/projected/21f95b5c-4c31-4836-a8ba-41dc2949edea-kube-api-access-wrkfn\") pod \"e2e-shared-a-w4-af237d-openshift-default-64f945b797-62lhb\" (UID: \"21f95b5c-4c31-4836-a8ba-41dc2949edea\") " pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-62lhb" Oct 29 23:10:16.928450 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:16.927718 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21f95b5c-4c31-4836-a8ba-41dc2949edea-istiod-ca-cert\") pod \"e2e-shared-a-w4-af237d-openshift-default-64f945b797-62lhb\" (UID: \"21f95b5c-4c31-4836-a8ba-41dc2949edea\") " pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-62lhb" Oct 29 23:10:16.928450 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:16.927747 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21f95b5c-4c31-4836-a8ba-41dc2949edea-istio-envoy\") pod \"e2e-shared-a-w4-af237d-openshift-default-64f945b797-62lhb\" (UID: \"21f95b5c-4c31-4836-a8ba-41dc2949edea\") " pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-62lhb" Oct 29 23:10:16.928450 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:16.927831 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21f95b5c-4c31-4836-a8ba-41dc2949edea-istio-podinfo\") pod \"e2e-shared-a-w4-af237d-openshift-default-64f945b797-62lhb\" (UID: \"21f95b5c-4c31-4836-a8ba-41dc2949edea\") " pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-62lhb" Oct 29 23:10:16.928450 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:16.927886 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21f95b5c-4c31-4836-a8ba-41dc2949edea-workload-socket\") pod \"e2e-shared-a-w4-af237d-openshift-default-64f945b797-62lhb\" (UID: \"21f95b5c-4c31-4836-a8ba-41dc2949edea\") " pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-62lhb" Oct 29 23:10:16.949350 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:16.936663 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21f95b5c-4c31-4836-a8ba-41dc2949edea-istio-data\") pod \"e2e-shared-a-w4-af237d-openshift-default-64f945b797-62lhb\" (UID: \"21f95b5c-4c31-4836-a8ba-41dc2949edea\") " pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-62lhb" Oct 29 23:10:17.038063 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:17.038020 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21f95b5c-4c31-4836-a8ba-41dc2949edea-istiod-ca-cert\") pod \"e2e-shared-a-w4-af237d-openshift-default-64f945b797-62lhb\" (UID: \"21f95b5c-4c31-4836-a8ba-41dc2949edea\") " pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-62lhb" Oct 29 23:10:17.038271 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:17.038075 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21f95b5c-4c31-4836-a8ba-41dc2949edea-istio-envoy\") pod \"e2e-shared-a-w4-af237d-openshift-default-64f945b797-62lhb\" (UID: \"21f95b5c-4c31-4836-a8ba-41dc2949edea\") " pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-62lhb" Oct 29 23:10:17.038271 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:17.038137 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21f95b5c-4c31-4836-a8ba-41dc2949edea-istio-podinfo\") pod \"e2e-shared-a-w4-af237d-openshift-default-64f945b797-62lhb\" (UID: \"21f95b5c-4c31-4836-a8ba-41dc2949edea\") " pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-62lhb" Oct 29 23:10:17.038271 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:17.038170 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21f95b5c-4c31-4836-a8ba-41dc2949edea-workload-socket\") pod \"e2e-shared-a-w4-af237d-openshift-default-64f945b797-62lhb\" (UID: \"21f95b5c-4c31-4836-a8ba-41dc2949edea\") " pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-62lhb" Oct 29 23:10:17.038495 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:17.038369 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21f95b5c-4c31-4836-a8ba-41dc2949edea-istio-data\") pod \"e2e-shared-a-w4-af237d-openshift-default-64f945b797-62lhb\" (UID: \"21f95b5c-4c31-4836-a8ba-41dc2949edea\") " pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-62lhb" Oct 29 23:10:17.038495 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:17.038446 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21f95b5c-4c31-4836-a8ba-41dc2949edea-credential-socket\") pod \"e2e-shared-a-w4-af237d-openshift-default-64f945b797-62lhb\" (UID: \"21f95b5c-4c31-4836-a8ba-41dc2949edea\") " pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-62lhb" Oct 29 23:10:17.038601 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:17.038499 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21f95b5c-4c31-4836-a8ba-41dc2949edea-istio-token\") pod \"e2e-shared-a-w4-af237d-openshift-default-64f945b797-62lhb\" (UID: \"21f95b5c-4c31-4836-a8ba-41dc2949edea\") " pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-62lhb" Oct 29 23:10:17.038601 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:17.038551 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21f95b5c-4c31-4836-a8ba-41dc2949edea-workload-certs\") pod \"e2e-shared-a-w4-af237d-openshift-default-64f945b797-62lhb\" (UID: \"21f95b5c-4c31-4836-a8ba-41dc2949edea\") " pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-62lhb" Oct 29 23:10:17.038720 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:17.038630 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrkfn\" (UniqueName: \"kubernetes.io/projected/21f95b5c-4c31-4836-a8ba-41dc2949edea-kube-api-access-wrkfn\") pod \"e2e-shared-a-w4-af237d-openshift-default-64f945b797-62lhb\" (UID: \"21f95b5c-4c31-4836-a8ba-41dc2949edea\") " pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-62lhb" Oct 29 23:10:17.039746 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:17.039717 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21f95b5c-4c31-4836-a8ba-41dc2949edea-istiod-ca-cert\") pod \"e2e-shared-a-w4-af237d-openshift-default-64f945b797-62lhb\" (UID: \"21f95b5c-4c31-4836-a8ba-41dc2949edea\") " pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-62lhb" Oct 29 23:10:17.040374 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:17.040350 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21f95b5c-4c31-4836-a8ba-41dc2949edea-istio-data\") pod \"e2e-shared-a-w4-af237d-openshift-default-64f945b797-62lhb\" (UID: \"21f95b5c-4c31-4836-a8ba-41dc2949edea\") " pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-62lhb" Oct 29 23:10:17.040957 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:17.040935 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21f95b5c-4c31-4836-a8ba-41dc2949edea-credential-socket\") pod \"e2e-shared-a-w4-af237d-openshift-default-64f945b797-62lhb\" (UID: \"21f95b5c-4c31-4836-a8ba-41dc2949edea\") " pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-62lhb" Oct 29 23:10:17.041434 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:17.041364 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21f95b5c-4c31-4836-a8ba-41dc2949edea-workload-certs\") pod \"e2e-shared-a-w4-af237d-openshift-default-64f945b797-62lhb\" (UID: \"21f95b5c-4c31-4836-a8ba-41dc2949edea\") " pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-62lhb" Oct 29 23:10:17.041766 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:17.041720 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21f95b5c-4c31-4836-a8ba-41dc2949edea-workload-socket\") pod \"e2e-shared-a-w4-af237d-openshift-default-64f945b797-62lhb\" (UID: \"21f95b5c-4c31-4836-a8ba-41dc2949edea\") " pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-62lhb" Oct 29 23:10:17.050506 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:17.050455 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21f95b5c-4c31-4836-a8ba-41dc2949edea-istio-podinfo\") pod \"e2e-shared-a-w4-af237d-openshift-default-64f945b797-62lhb\" (UID: \"21f95b5c-4c31-4836-a8ba-41dc2949edea\") " pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-62lhb" Oct 29 23:10:17.054317 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:17.054297 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21f95b5c-4c31-4836-a8ba-41dc2949edea-istio-envoy\") pod \"e2e-shared-a-w4-af237d-openshift-default-64f945b797-62lhb\" (UID: \"21f95b5c-4c31-4836-a8ba-41dc2949edea\") " pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-62lhb" Oct 29 23:10:17.065044 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:17.064974 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrkfn\" (UniqueName: \"kubernetes.io/projected/21f95b5c-4c31-4836-a8ba-41dc2949edea-kube-api-access-wrkfn\") pod \"e2e-shared-a-w4-af237d-openshift-default-64f945b797-62lhb\" (UID: \"21f95b5c-4c31-4836-a8ba-41dc2949edea\") " pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-62lhb" Oct 29 23:10:17.067861 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:17.067835 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21f95b5c-4c31-4836-a8ba-41dc2949edea-istio-token\") pod \"e2e-shared-a-w4-af237d-openshift-default-64f945b797-62lhb\" (UID: \"21f95b5c-4c31-4836-a8ba-41dc2949edea\") " pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-62lhb" Oct 29 23:10:17.125527 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:17.125259 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-62lhb" Oct 29 23:10:17.570680 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:17.570589 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:10:17.612355 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:17.610434 5057 patch_prober.go:29] interesting pod/e2e-autores-a9ced3-openshift-default-7d8898bf57-9j6rn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" start-of-body= Oct 29 23:10:17.612355 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:17.610495 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-9j6rn" podUID="f521c155-eff0-4b0b-b4cd-e55abde4ed41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" Oct 29 23:10:17.660095 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:17.660045 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-62lhb"] Oct 29 23:10:17.662051 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:17.661881 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 23:10:17.662051 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:17.661975 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 23:10:17.662051 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:17.662019 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 23:10:17.774166 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:17.774132 5057 generic.go:356] "Generic (PLEG): container finished" podID="c237fe02-4087-448d-b7b1-c7e03f1872c3" containerID="a365358bd2744c25f8081adcaeaf4f38c39c9b3d8fda8100cd1a6497beec85e5" exitCode=0 Oct 29 23:10:17.774267 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:17.774227 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-9nnj5" event={"ID":"c237fe02-4087-448d-b7b1-c7e03f1872c3","Type":"ContainerDied","Data":"a365358bd2744c25f8081adcaeaf4f38c39c9b3d8fda8100cd1a6497beec85e5"} Oct 29 23:10:17.776266 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:17.776237 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-62lhb" event={"ID":"21f95b5c-4c31-4836-a8ba-41dc2949edea","Type":"ContainerStarted","Data":"9455feefaf15fe63ece1884b8357e4854c72960e3fad9f5f1537c00896736afb"} Oct 29 23:10:17.827304 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:17.827211 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-9nnj5" Oct 29 23:10:18.020384 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:18.018638 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c237fe02-4087-448d-b7b1-c7e03f1872c3-kube-api-access-dfkbf\" (UniqueName: \"kubernetes.io/projected/c237fe02-4087-448d-b7b1-c7e03f1872c3-kube-api-access-dfkbf\") pod \"c237fe02-4087-448d-b7b1-c7e03f1872c3\" (UID: \"c237fe02-4087-448d-b7b1-c7e03f1872c3\") " Oct 29 23:10:18.020384 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:18.019568 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c237fe02-4087-448d-b7b1-c7e03f1872c3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c237fe02-4087-448d-b7b1-c7e03f1872c3-istio-podinfo\") pod \"c237fe02-4087-448d-b7b1-c7e03f1872c3\" (UID: \"c237fe02-4087-448d-b7b1-c7e03f1872c3\") " Oct 29 23:10:18.020384 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:18.019659 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c237fe02-4087-448d-b7b1-c7e03f1872c3-istio-token\" (UniqueName: \"kubernetes.io/projected/c237fe02-4087-448d-b7b1-c7e03f1872c3-istio-token\") pod \"c237fe02-4087-448d-b7b1-c7e03f1872c3\" (UID: \"c237fe02-4087-448d-b7b1-c7e03f1872c3\") " Oct 29 23:10:18.020384 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:18.019741 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c237fe02-4087-448d-b7b1-c7e03f1872c3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c237fe02-4087-448d-b7b1-c7e03f1872c3-workload-certs\") pod \"c237fe02-4087-448d-b7b1-c7e03f1872c3\" (UID: \"c237fe02-4087-448d-b7b1-c7e03f1872c3\") " Oct 29 23:10:18.020384 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:18.019793 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c237fe02-4087-448d-b7b1-c7e03f1872c3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c237fe02-4087-448d-b7b1-c7e03f1872c3-istio-envoy\") pod \"c237fe02-4087-448d-b7b1-c7e03f1872c3\" (UID: \"c237fe02-4087-448d-b7b1-c7e03f1872c3\") " Oct 29 23:10:18.020384 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:18.019837 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c237fe02-4087-448d-b7b1-c7e03f1872c3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c237fe02-4087-448d-b7b1-c7e03f1872c3-istiod-ca-cert\") pod \"c237fe02-4087-448d-b7b1-c7e03f1872c3\" (UID: \"c237fe02-4087-448d-b7b1-c7e03f1872c3\") " Oct 29 23:10:18.020384 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:18.019883 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c237fe02-4087-448d-b7b1-c7e03f1872c3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c237fe02-4087-448d-b7b1-c7e03f1872c3-istio-data\") pod \"c237fe02-4087-448d-b7b1-c7e03f1872c3\" (UID: \"c237fe02-4087-448d-b7b1-c7e03f1872c3\") " Oct 29 23:10:18.020384 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:18.019937 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c237fe02-4087-448d-b7b1-c7e03f1872c3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c237fe02-4087-448d-b7b1-c7e03f1872c3-workload-socket\") pod \"c237fe02-4087-448d-b7b1-c7e03f1872c3\" (UID: \"c237fe02-4087-448d-b7b1-c7e03f1872c3\") " Oct 29 23:10:18.020384 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:18.020037 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c237fe02-4087-448d-b7b1-c7e03f1872c3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c237fe02-4087-448d-b7b1-c7e03f1872c3-credential-socket\") pod \"c237fe02-4087-448d-b7b1-c7e03f1872c3\" (UID: \"c237fe02-4087-448d-b7b1-c7e03f1872c3\") " Oct 29 23:10:18.021024 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:18.020667 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c237fe02-4087-448d-b7b1-c7e03f1872c3-workload-socket" pod "c237fe02-4087-448d-b7b1-c7e03f1872c3" (UID: "c237fe02-4087-448d-b7b1-c7e03f1872c3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:10:18.021024 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:18.020885 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c237fe02-4087-448d-b7b1-c7e03f1872c3-istio-data" pod "c237fe02-4087-448d-b7b1-c7e03f1872c3" (UID: "c237fe02-4087-448d-b7b1-c7e03f1872c3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:10:18.022078 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:18.021225 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c237fe02-4087-448d-b7b1-c7e03f1872c3-istiod-ca-cert" pod "c237fe02-4087-448d-b7b1-c7e03f1872c3" (UID: "c237fe02-4087-448d-b7b1-c7e03f1872c3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:10:18.022078 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:18.021349 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c237fe02-4087-448d-b7b1-c7e03f1872c3-workload-certs" pod "c237fe02-4087-448d-b7b1-c7e03f1872c3" (UID: "c237fe02-4087-448d-b7b1-c7e03f1872c3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:10:18.022078 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:18.022012 5057 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c237fe02-4087-448d-b7b1-c7e03f1872c3-workload-certs\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:10:18.022078 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:18.022034 5057 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c237fe02-4087-448d-b7b1-c7e03f1872c3-istiod-ca-cert\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:10:18.022078 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:18.022045 5057 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c237fe02-4087-448d-b7b1-c7e03f1872c3-istio-data\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:10:18.022078 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:18.022056 5057 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c237fe02-4087-448d-b7b1-c7e03f1872c3-workload-socket\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:10:18.022654 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:18.022464 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c237fe02-4087-448d-b7b1-c7e03f1872c3-credential-socket" pod "c237fe02-4087-448d-b7b1-c7e03f1872c3" (UID: "c237fe02-4087-448d-b7b1-c7e03f1872c3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:10:18.022929 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:18.022895 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c237fe02-4087-448d-b7b1-c7e03f1872c3-kube-api-access-dfkbf" pod "c237fe02-4087-448d-b7b1-c7e03f1872c3" (UID: "c237fe02-4087-448d-b7b1-c7e03f1872c3"). InnerVolumeSpecName "kube-api-access-dfkbf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:10:18.034482 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:18.024760 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c237fe02-4087-448d-b7b1-c7e03f1872c3-istio-envoy" pod "c237fe02-4087-448d-b7b1-c7e03f1872c3" (UID: "c237fe02-4087-448d-b7b1-c7e03f1872c3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:10:18.041986 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:18.040883 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c237fe02-4087-448d-b7b1-c7e03f1872c3-istio-podinfo" pod "c237fe02-4087-448d-b7b1-c7e03f1872c3" (UID: "c237fe02-4087-448d-b7b1-c7e03f1872c3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:10:18.041986 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:18.041210 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c237fe02-4087-448d-b7b1-c7e03f1872c3-istio-token" pod "c237fe02-4087-448d-b7b1-c7e03f1872c3" (UID: "c237fe02-4087-448d-b7b1-c7e03f1872c3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:10:18.124649 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:18.124047 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dfkbf\" (UniqueName: \"kubernetes.io/projected/c237fe02-4087-448d-b7b1-c7e03f1872c3-kube-api-access-dfkbf\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:10:18.124649 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:18.124081 5057 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c237fe02-4087-448d-b7b1-c7e03f1872c3-istio-podinfo\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:10:18.124649 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:18.124095 5057 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c237fe02-4087-448d-b7b1-c7e03f1872c3-istio-token\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:10:18.124649 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:18.124107 5057 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c237fe02-4087-448d-b7b1-c7e03f1872c3-istio-envoy\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:10:18.124649 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:18.124119 5057 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c237fe02-4087-448d-b7b1-c7e03f1872c3-credential-socket\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:10:18.215396 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:18.214763 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:10:18.600355 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:18.599751 5057 patch_prober.go:29] interesting pod/e2e-autores-a9ced3-openshift-default-7d8898bf57-9j6rn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" start-of-body= Oct 29 23:10:18.600355 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:18.599814 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-9j6rn" podUID="f521c155-eff0-4b0b-b4cd-e55abde4ed41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" Oct 29 23:10:18.805357 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:18.805259 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-9nnj5" event={"ID":"c237fe02-4087-448d-b7b1-c7e03f1872c3","Type":"ContainerDied","Data":"ebc6668326b38613ad5c69a83cf5bc841a0f894a3f66d79c770822c3d53da637"} Oct 29 23:10:18.805357 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:18.805351 5057 scope.go:122] "RemoveContainer" containerID="a365358bd2744c25f8081adcaeaf4f38c39c9b3d8fda8100cd1a6497beec85e5" Oct 29 23:10:18.805930 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:18.805509 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-9nnj5" Oct 29 23:10:18.821457 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:18.817204 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-62lhb" event={"ID":"21f95b5c-4c31-4836-a8ba-41dc2949edea","Type":"ContainerStarted","Data":"b644e48c01c6ec554b2034fed0bc7a97e610c06b5895fe0e34337dc8d360c311"} Oct 29 23:10:18.940396 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:18.937973 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-62lhb" podStartSLOduration=2.937949279 podStartE2EDuration="2.937949279s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:10:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:10:18.88885033 +0000 UTC m=+2199.672063758" watchObservedRunningTime="2026-10-29 23:10:18.937949279 +0000 UTC m=+2199.721162705" Oct 29 23:10:18.951081 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:18.951041 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-9nnj5"] Oct 29 23:10:18.956761 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:18.956733 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-9nnj5"] Oct 29 23:10:19.075739 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:10:19.075703 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d2dnn" containerName="extractor" Oct 29 23:10:19.138536 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:19.128692 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-62lhb" Oct 29 23:10:19.141354 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:19.139517 5057 patch_prober.go:29] interesting pod/e2e-shared-a-w4-af237d-openshift-default-64f945b797-62lhb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body= Oct 29 23:10:19.141354 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:19.139582 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-62lhb" podUID="21f95b5c-4c31-4836-a8ba-41dc2949edea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" Oct 29 23:10:19.237160 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:19.236069 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:10:19.353379 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:19.353148 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c237fe02-4087-448d-b7b1-c7e03f1872c3" path="/var/lib/kubelet/pods/c237fe02-4087-448d-b7b1-c7e03f1872c3/volumes" Oct 29 23:10:19.354765 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:19.354680 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:10:19.459163 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:19.451833 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-755453-788f5c78c-svk2k"] Oct 29 23:10:19.459163 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:19.453695 5057 cpu_manager.go:414] "containerMap: removing container" podUID="c237fe02-4087-448d-b7b1-c7e03f1872c3" containerName="istio-proxy" Oct 29 23:10:19.459163 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:19.453718 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c237fe02-4087-448d-b7b1-c7e03f1872c3" containerName="istio-proxy" Oct 29 23:10:19.459163 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:19.455854 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="c237fe02-4087-448d-b7b1-c7e03f1872c3" containerName="istio-proxy" Oct 29 23:10:19.462593 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:19.462567 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-755453-788f5c78c-svk2k"] Oct 29 23:10:19.462740 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:19.462675 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-755453-788f5c78c-svk2k" Oct 29 23:10:19.495566 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:19.489745 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-755453\"" Oct 29 23:10:19.495566 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:19.489923 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-755453-dockercfg-kvhw4\"" Oct 29 23:10:19.601353 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:19.599887 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e08ea7b-6096-4253-a757-b092cf07bc7c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-755453-788f5c78c-svk2k\" (UID: \"0e08ea7b-6096-4253-a757-b092cf07bc7c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-755453-788f5c78c-svk2k" Oct 29 23:10:19.601353 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:19.600069 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6qfs\" (UniqueName: \"kubernetes.io/projected/0e08ea7b-6096-4253-a757-b092cf07bc7c-kube-api-access-d6qfs\") pod \"payload-pre-processing-e2e-shared-b-w3-755453-788f5c78c-svk2k\" (UID: \"0e08ea7b-6096-4253-a757-b092cf07bc7c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-755453-788f5c78c-svk2k" Oct 29 23:10:19.616386 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:19.607005 5057 patch_prober.go:29] interesting pod/e2e-autores-a9ced3-openshift-default-7d8898bf57-9j6rn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" start-of-body= Oct 29 23:10:19.616386 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:19.607064 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-9j6rn" podUID="f521c155-eff0-4b0b-b4cd-e55abde4ed41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" Oct 29 23:10:19.734413 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:19.723679 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e08ea7b-6096-4253-a757-b092cf07bc7c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-755453-788f5c78c-svk2k\" (UID: \"0e08ea7b-6096-4253-a757-b092cf07bc7c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-755453-788f5c78c-svk2k" Oct 29 23:10:19.867353 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:19.850016 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6qfs\" (UniqueName: \"kubernetes.io/projected/0e08ea7b-6096-4253-a757-b092cf07bc7c-kube-api-access-d6qfs\") pod \"payload-pre-processing-e2e-shared-b-w3-755453-788f5c78c-svk2k\" (UID: \"0e08ea7b-6096-4253-a757-b092cf07bc7c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-755453-788f5c78c-svk2k" Oct 29 23:10:19.867353 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:19.780460 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-3e5b21-d7b5c69c5-ds82z" Oct 29 23:10:19.867353 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:19.850323 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-755453-9bd867db-hdlwb"] Oct 29 23:10:19.867353 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:19.726575 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e08ea7b-6096-4253-a757-b092cf07bc7c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-755453-788f5c78c-svk2k\" (UID: \"0e08ea7b-6096-4253-a757-b092cf07bc7c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-755453-788f5c78c-svk2k" Oct 29 23:10:19.946592 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:19.946561 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-755453-9bd867db-hdlwb"] Oct 29 23:10:19.956436 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:19.947295 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-755453-9bd867db-hdlwb" Oct 29 23:10:20.083254 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:20.066648 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6qfs\" (UniqueName: \"kubernetes.io/projected/0e08ea7b-6096-4253-a757-b092cf07bc7c-kube-api-access-d6qfs\") pod \"payload-pre-processing-e2e-shared-b-w3-755453-788f5c78c-svk2k\" (UID: \"0e08ea7b-6096-4253-a757-b092cf07bc7c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-755453-788f5c78c-svk2k" Oct 29 23:10:20.105305 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:20.099782 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6760c386-e49a-4d99-b811-31a8d9df5b41-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-755453-9bd867db-hdlwb\" (UID: \"6760c386-e49a-4d99-b811-31a8d9df5b41\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-755453-9bd867db-hdlwb" Oct 29 23:10:20.105305 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:20.100009 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wkb6\" (UniqueName: \"kubernetes.io/projected/6760c386-e49a-4d99-b811-31a8d9df5b41-kube-api-access-8wkb6\") pod \"payload-processing-e2e-shared-b-w3-755453-9bd867db-hdlwb\" (UID: \"6760c386-e49a-4d99-b811-31a8d9df5b41\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-755453-9bd867db-hdlwb" Oct 29 23:10:20.143905 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:20.143869 5057 patch_prober.go:29] interesting pod/e2e-shared-a-w4-af237d-openshift-default-64f945b797-62lhb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body= Oct 29 23:10:20.144160 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:20.144119 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-62lhb" podUID="21f95b5c-4c31-4836-a8ba-41dc2949edea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" Oct 29 23:10:20.236539 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:20.221987 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wkb6\" (UniqueName: \"kubernetes.io/projected/6760c386-e49a-4d99-b811-31a8d9df5b41-kube-api-access-8wkb6\") pod \"payload-processing-e2e-shared-b-w3-755453-9bd867db-hdlwb\" (UID: \"6760c386-e49a-4d99-b811-31a8d9df5b41\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-755453-9bd867db-hdlwb" Oct 29 23:10:20.240353 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:20.240248 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6760c386-e49a-4d99-b811-31a8d9df5b41-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-755453-9bd867db-hdlwb\" (UID: \"6760c386-e49a-4d99-b811-31a8d9df5b41\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-755453-9bd867db-hdlwb" Oct 29 23:10:20.240514 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:20.240393 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6760c386-e49a-4d99-b811-31a8d9df5b41-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-755453-9bd867db-hdlwb\" (UID: \"6760c386-e49a-4d99-b811-31a8d9df5b41\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-755453-9bd867db-hdlwb" Oct 29 23:10:20.255286 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:20.253180 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wkb6\" (UniqueName: \"kubernetes.io/projected/6760c386-e49a-4d99-b811-31a8d9df5b41-kube-api-access-8wkb6\") pod \"payload-processing-e2e-shared-b-w3-755453-9bd867db-hdlwb\" (UID: \"6760c386-e49a-4d99-b811-31a8d9df5b41\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-755453-9bd867db-hdlwb" Oct 29 23:10:20.333402 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:20.332941 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-755453-788f5c78c-svk2k" Oct 29 23:10:20.456416 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:20.455998 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:10:20.529392 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:20.528634 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-755453-9bd867db-hdlwb" Oct 29 23:10:20.632365 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:20.626820 5057 patch_prober.go:29] interesting pod/e2e-autores-a9ced3-openshift-default-7d8898bf57-9j6rn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" start-of-body= Oct 29 23:10:20.632365 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:20.626884 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-9j6rn" podUID="f521c155-eff0-4b0b-b4cd-e55abde4ed41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" Oct 29 23:10:20.667559 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:20.667526 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-3e5b21-f69958676-g78g2" Oct 29 23:10:20.945803 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:20.945708 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-755453-788f5c78c-svk2k"] Oct 29 23:10:21.132906 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:21.131775 5057 patch_prober.go:29] interesting pod/e2e-shared-a-w4-af237d-openshift-default-64f945b797-62lhb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body= Oct 29 23:10:21.132906 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:21.131837 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-62lhb" podUID="21f95b5c-4c31-4836-a8ba-41dc2949edea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" Oct 29 23:10:21.239029 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:21.238987 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-755453-788f5c78c-svk2k" event={"ID":"0e08ea7b-6096-4253-a757-b092cf07bc7c","Type":"ContainerStarted","Data":"e15440c29d67c92cf76444e89c189c274e005fdcd089d174448a8814ee0c202a"} Oct 29 23:10:21.239229 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:21.239104 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-755453-788f5c78c-svk2k" event={"ID":"0e08ea7b-6096-4253-a757-b092cf07bc7c","Type":"ContainerStarted","Data":"8a777633783aa760a65bff963846cdd4b36a40dc54931b2ccebcc7b43c665d18"} Oct 29 23:10:21.239293 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:21.239258 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-755453-788f5c78c-svk2k" Oct 29 23:10:21.314622 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:21.312298 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-755453-788f5c78c-svk2k" podStartSLOduration=2.312278964 podStartE2EDuration="2.312278964s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:10:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:10:21.280678936 +0000 UTC m=+2202.063892363" watchObservedRunningTime="2026-10-29 23:10:21.312278964 +0000 UTC m=+2202.095492391" Oct 29 23:10:21.330446 ip-10-0-140-74 kubenswrapper[5057]: W1029 23:10:21.329035 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6760c386_e49a_4d99_b811_31a8d9df5b41.slice/crio-a9b37d08537dcbeb0f9f61565f174653a49b789bcfa8e8c6d6d85224a9d8f047 WatchSource:0}: Error finding container a9b37d08537dcbeb0f9f61565f174653a49b789bcfa8e8c6d6d85224a9d8f047: Status 404 returned error can't find the container with id a9b37d08537dcbeb0f9f61565f174653a49b789bcfa8e8c6d6d85224a9d8f047 Oct 29 23:10:21.338357 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:21.338304 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-755453-9bd867db-hdlwb"] Oct 29 23:10:21.614243 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:21.612446 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-9j6rn" Oct 29 23:10:21.614243 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:21.614102 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-9j6rn" Oct 29 23:10:21.615547 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:21.615522 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-9j6rn" Oct 29 23:10:22.126497 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:22.126463 5057 patch_prober.go:29] interesting pod/e2e-shared-a-w4-af237d-openshift-default-64f945b797-62lhb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body= Oct 29 23:10:22.127120 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:22.126516 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-62lhb" podUID="21f95b5c-4c31-4836-a8ba-41dc2949edea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" Oct 29 23:10:22.253189 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:22.253150 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-755453-9bd867db-hdlwb" event={"ID":"6760c386-e49a-4d99-b811-31a8d9df5b41","Type":"ContainerStarted","Data":"5ec94988b8438146a3c75eb716cc8925ef57b4b532a947dcc1c377448579dc3d"} Oct 29 23:10:22.253375 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:22.253217 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-755453-9bd867db-hdlwb" event={"ID":"6760c386-e49a-4d99-b811-31a8d9df5b41","Type":"ContainerStarted","Data":"a9b37d08537dcbeb0f9f61565f174653a49b789bcfa8e8c6d6d85224a9d8f047"} Oct 29 23:10:22.253375 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:22.253248 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-755453-9bd867db-hdlwb" Oct 29 23:10:22.303402 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:22.303252 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-755453-9bd867db-hdlwb" podStartSLOduration=3.303232101 podStartE2EDuration="3.303232101s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:10:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:10:22.292213403 +0000 UTC m=+2203.075426831" watchObservedRunningTime="2026-10-29 23:10:22.303232101 +0000 UTC m=+2203.086445528" Oct 29 23:10:22.346015 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:22.345983 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:10:23.126794 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:23.126762 5057 patch_prober.go:29] interesting pod/e2e-shared-a-w4-af237d-openshift-default-64f945b797-62lhb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body= Oct 29 23:10:23.127196 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:23.126811 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-62lhb" podUID="21f95b5c-4c31-4836-a8ba-41dc2949edea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" Oct 29 23:10:24.021647 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:24.021608 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:10:24.126957 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:24.126920 5057 patch_prober.go:29] interesting pod/e2e-shared-a-w4-af237d-openshift-default-64f945b797-62lhb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body= Oct 29 23:10:24.127468 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:24.126984 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-62lhb" podUID="21f95b5c-4c31-4836-a8ba-41dc2949edea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" Oct 29 23:10:25.130642 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:25.130604 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-62lhb" Oct 29 23:10:25.131066 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:25.131006 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-62lhb" Oct 29 23:10:25.131742 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:25.131722 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-62lhb" Oct 29 23:10:25.497315 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:25.497276 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-7smjh"] Oct 29 23:10:25.497669 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:25.497515 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-7smjh" podUID="7bc2103e-768a-4393-9522-68b0fc370219" containerName="istio-proxy" containerID="cri-o://151c14950da1cf77656122259bc0c192ec6e36821c5b2a64e2e6dc54a4d6d3ba" gracePeriod=30 Oct 29 23:10:26.140960 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:26.140922 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:10:28.887823 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:28.887782 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:10:30.191955 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:30.191916 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-autores-a9ced3/autores-model-a9ced3-kserve-c48c69885-lwwnp"] Oct 29 23:10:30.223713 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:30.223675 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-a9ced3/autores-model-a9ced3-kserve-c48c69885-lwwnp"] Oct 29 23:10:30.223895 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:30.223814 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-a9ced3/autores-model-a9ced3-kserve-c48c69885-lwwnp" Oct 29 23:10:30.236804 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:30.236775 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:10:30.250146 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:30.250116 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-a9ced3\"/\"default-dockercfg-8nf7x\"" Oct 29 23:10:30.250299 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:30.250167 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-a9ced3\"/\"openshift-service-ca.crt\"" Oct 29 23:10:30.250299 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:30.250235 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-a9ced3\"/\"autores-model-a9ced3-kserve-self-signed-certs\"" Oct 29 23:10:30.250413 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:30.250299 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-a9ced3\"/\"kube-root-ca.crt\"" Oct 29 23:10:30.325492 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:30.325460 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/902d109b-92a3-491e-8784-fe3f27a48dec-home\") pod \"autores-model-a9ced3-kserve-c48c69885-lwwnp\" (UID: \"902d109b-92a3-491e-8784-fe3f27a48dec\") " pod="ai-tenant-e2e-autores-a9ced3/autores-model-a9ced3-kserve-c48c69885-lwwnp" Oct 29 23:10:30.325675 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:30.325579 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lvtk8\" (UniqueName: \"kubernetes.io/projected/902d109b-92a3-491e-8784-fe3f27a48dec-kube-api-access-lvtk8\") pod \"autores-model-a9ced3-kserve-c48c69885-lwwnp\" (UID: \"902d109b-92a3-491e-8784-fe3f27a48dec\") " pod="ai-tenant-e2e-autores-a9ced3/autores-model-a9ced3-kserve-c48c69885-lwwnp" Oct 29 23:10:30.325675 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:30.325645 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/902d109b-92a3-491e-8784-fe3f27a48dec-tmp-dir\") pod \"autores-model-a9ced3-kserve-c48c69885-lwwnp\" (UID: \"902d109b-92a3-491e-8784-fe3f27a48dec\") " pod="ai-tenant-e2e-autores-a9ced3/autores-model-a9ced3-kserve-c48c69885-lwwnp" Oct 29 23:10:30.325793 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:30.325735 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/902d109b-92a3-491e-8784-fe3f27a48dec-dshm\") pod \"autores-model-a9ced3-kserve-c48c69885-lwwnp\" (UID: \"902d109b-92a3-491e-8784-fe3f27a48dec\") " pod="ai-tenant-e2e-autores-a9ced3/autores-model-a9ced3-kserve-c48c69885-lwwnp" Oct 29 23:10:30.325847 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:30.325794 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/902d109b-92a3-491e-8784-fe3f27a48dec-model-cache\") pod \"autores-model-a9ced3-kserve-c48c69885-lwwnp\" (UID: \"902d109b-92a3-491e-8784-fe3f27a48dec\") " pod="ai-tenant-e2e-autores-a9ced3/autores-model-a9ced3-kserve-c48c69885-lwwnp" Oct 29 23:10:30.325959 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:30.325939 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/902d109b-92a3-491e-8784-fe3f27a48dec-tls-certs\") pod \"autores-model-a9ced3-kserve-c48c69885-lwwnp\" (UID: \"902d109b-92a3-491e-8784-fe3f27a48dec\") " pod="ai-tenant-e2e-autores-a9ced3/autores-model-a9ced3-kserve-c48c69885-lwwnp" Oct 29 23:10:30.427434 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:30.427394 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/902d109b-92a3-491e-8784-fe3f27a48dec-tls-certs\") pod \"autores-model-a9ced3-kserve-c48c69885-lwwnp\" (UID: \"902d109b-92a3-491e-8784-fe3f27a48dec\") " pod="ai-tenant-e2e-autores-a9ced3/autores-model-a9ced3-kserve-c48c69885-lwwnp" Oct 29 23:10:30.427636 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:30.427557 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/902d109b-92a3-491e-8784-fe3f27a48dec-home\") pod \"autores-model-a9ced3-kserve-c48c69885-lwwnp\" (UID: \"902d109b-92a3-491e-8784-fe3f27a48dec\") " pod="ai-tenant-e2e-autores-a9ced3/autores-model-a9ced3-kserve-c48c69885-lwwnp" Oct 29 23:10:30.427730 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:30.427655 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lvtk8\" (UniqueName: \"kubernetes.io/projected/902d109b-92a3-491e-8784-fe3f27a48dec-kube-api-access-lvtk8\") pod \"autores-model-a9ced3-kserve-c48c69885-lwwnp\" (UID: \"902d109b-92a3-491e-8784-fe3f27a48dec\") " pod="ai-tenant-e2e-autores-a9ced3/autores-model-a9ced3-kserve-c48c69885-lwwnp" Oct 29 23:10:30.427791 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:30.427732 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/902d109b-92a3-491e-8784-fe3f27a48dec-tmp-dir\") pod \"autores-model-a9ced3-kserve-c48c69885-lwwnp\" (UID: \"902d109b-92a3-491e-8784-fe3f27a48dec\") " pod="ai-tenant-e2e-autores-a9ced3/autores-model-a9ced3-kserve-c48c69885-lwwnp" Oct 29 23:10:30.427851 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:30.427814 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/902d109b-92a3-491e-8784-fe3f27a48dec-dshm\") pod \"autores-model-a9ced3-kserve-c48c69885-lwwnp\" (UID: \"902d109b-92a3-491e-8784-fe3f27a48dec\") " pod="ai-tenant-e2e-autores-a9ced3/autores-model-a9ced3-kserve-c48c69885-lwwnp" Oct 29 23:10:30.427911 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:30.427893 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/902d109b-92a3-491e-8784-fe3f27a48dec-model-cache\") pod \"autores-model-a9ced3-kserve-c48c69885-lwwnp\" (UID: \"902d109b-92a3-491e-8784-fe3f27a48dec\") " pod="ai-tenant-e2e-autores-a9ced3/autores-model-a9ced3-kserve-c48c69885-lwwnp" Oct 29 23:10:30.428021 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:30.427996 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/902d109b-92a3-491e-8784-fe3f27a48dec-home\") pod \"autores-model-a9ced3-kserve-c48c69885-lwwnp\" (UID: \"902d109b-92a3-491e-8784-fe3f27a48dec\") " pod="ai-tenant-e2e-autores-a9ced3/autores-model-a9ced3-kserve-c48c69885-lwwnp" Oct 29 23:10:30.428095 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:30.428016 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/902d109b-92a3-491e-8784-fe3f27a48dec-tmp-dir\") pod \"autores-model-a9ced3-kserve-c48c69885-lwwnp\" (UID: \"902d109b-92a3-491e-8784-fe3f27a48dec\") " pod="ai-tenant-e2e-autores-a9ced3/autores-model-a9ced3-kserve-c48c69885-lwwnp" Oct 29 23:10:30.428275 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:30.428252 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/902d109b-92a3-491e-8784-fe3f27a48dec-model-cache\") pod \"autores-model-a9ced3-kserve-c48c69885-lwwnp\" (UID: \"902d109b-92a3-491e-8784-fe3f27a48dec\") " pod="ai-tenant-e2e-autores-a9ced3/autores-model-a9ced3-kserve-c48c69885-lwwnp" Oct 29 23:10:30.430180 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:30.430157 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/902d109b-92a3-491e-8784-fe3f27a48dec-tls-certs\") pod \"autores-model-a9ced3-kserve-c48c69885-lwwnp\" (UID: \"902d109b-92a3-491e-8784-fe3f27a48dec\") " pod="ai-tenant-e2e-autores-a9ced3/autores-model-a9ced3-kserve-c48c69885-lwwnp" Oct 29 23:10:30.430754 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:30.430736 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/902d109b-92a3-491e-8784-fe3f27a48dec-dshm\") pod \"autores-model-a9ced3-kserve-c48c69885-lwwnp\" (UID: \"902d109b-92a3-491e-8784-fe3f27a48dec\") " pod="ai-tenant-e2e-autores-a9ced3/autores-model-a9ced3-kserve-c48c69885-lwwnp" Oct 29 23:10:30.454161 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:30.454138 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lvtk8\" (UniqueName: \"kubernetes.io/projected/902d109b-92a3-491e-8784-fe3f27a48dec-kube-api-access-lvtk8\") pod \"autores-model-a9ced3-kserve-c48c69885-lwwnp\" (UID: \"902d109b-92a3-491e-8784-fe3f27a48dec\") " pod="ai-tenant-e2e-autores-a9ced3/autores-model-a9ced3-kserve-c48c69885-lwwnp" Oct 29 23:10:30.541892 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:30.541842 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-a9ced3/autores-model-a9ced3-kserve-c48c69885-lwwnp" Oct 29 23:10:30.918287 ip-10-0-140-74 kubenswrapper[5057]: W1029 23:10:30.918151 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod902d109b_92a3_491e_8784_fe3f27a48dec.slice/crio-58d3898dc427781e8de8500be30671b59a42f87fd7e1918579f61c2118689ee8 WatchSource:0}: Error finding container 58d3898dc427781e8de8500be30671b59a42f87fd7e1918579f61c2118689ee8: Status 404 returned error can't find the container with id 58d3898dc427781e8de8500be30671b59a42f87fd7e1918579f61c2118689ee8 Oct 29 23:10:30.920191 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:30.919730 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-a9ced3/autores-model-a9ced3-kserve-c48c69885-lwwnp"] Oct 29 23:10:30.927703 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:30.927672 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-7smjh" Oct 29 23:10:31.045855 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:31.045783 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7bc2103e-768a-4393-9522-68b0fc370219-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7bc2103e-768a-4393-9522-68b0fc370219-istio-envoy\") pod \"7bc2103e-768a-4393-9522-68b0fc370219\" (UID: \"7bc2103e-768a-4393-9522-68b0fc370219\") " Oct 29 23:10:31.046000 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:31.045888 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7bc2103e-768a-4393-9522-68b0fc370219-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7bc2103e-768a-4393-9522-68b0fc370219-istio-data\") pod \"7bc2103e-768a-4393-9522-68b0fc370219\" (UID: \"7bc2103e-768a-4393-9522-68b0fc370219\") " Oct 29 23:10:31.046489 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:31.046431 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7bc2103e-768a-4393-9522-68b0fc370219-kube-api-access-njw8k\" (UniqueName: \"kubernetes.io/projected/7bc2103e-768a-4393-9522-68b0fc370219-kube-api-access-njw8k\") pod \"7bc2103e-768a-4393-9522-68b0fc370219\" (UID: \"7bc2103e-768a-4393-9522-68b0fc370219\") " Oct 29 23:10:31.046794 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:31.046527 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7bc2103e-768a-4393-9522-68b0fc370219-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7bc2103e-768a-4393-9522-68b0fc370219-workload-certs\") pod \"7bc2103e-768a-4393-9522-68b0fc370219\" (UID: \"7bc2103e-768a-4393-9522-68b0fc370219\") " Oct 29 23:10:31.046794 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:31.046643 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7bc2103e-768a-4393-9522-68b0fc370219-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bc2103e-768a-4393-9522-68b0fc370219-workload-socket\") pod \"7bc2103e-768a-4393-9522-68b0fc370219\" (UID: \"7bc2103e-768a-4393-9522-68b0fc370219\") " Oct 29 23:10:31.046794 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:31.046680 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7bc2103e-768a-4393-9522-68b0fc370219-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7bc2103e-768a-4393-9522-68b0fc370219-istiod-ca-cert\") pod \"7bc2103e-768a-4393-9522-68b0fc370219\" (UID: \"7bc2103e-768a-4393-9522-68b0fc370219\") " Oct 29 23:10:31.047742 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:31.047324 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7bc2103e-768a-4393-9522-68b0fc370219-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7bc2103e-768a-4393-9522-68b0fc370219-istio-podinfo\") pod \"7bc2103e-768a-4393-9522-68b0fc370219\" (UID: \"7bc2103e-768a-4393-9522-68b0fc370219\") " Oct 29 23:10:31.047812 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:31.047791 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7bc2103e-768a-4393-9522-68b0fc370219-istio-token\" (UniqueName: \"kubernetes.io/projected/7bc2103e-768a-4393-9522-68b0fc370219-istio-token\") pod \"7bc2103e-768a-4393-9522-68b0fc370219\" (UID: \"7bc2103e-768a-4393-9522-68b0fc370219\") " Oct 29 23:10:31.047912 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:31.047895 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7bc2103e-768a-4393-9522-68b0fc370219-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bc2103e-768a-4393-9522-68b0fc370219-credential-socket\") pod \"7bc2103e-768a-4393-9522-68b0fc370219\" (UID: \"7bc2103e-768a-4393-9522-68b0fc370219\") " Oct 29 23:10:31.049190 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:31.049165 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bc2103e-768a-4393-9522-68b0fc370219-workload-certs" pod "7bc2103e-768a-4393-9522-68b0fc370219" (UID: "7bc2103e-768a-4393-9522-68b0fc370219"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:10:31.049883 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:31.049418 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7bc2103e-768a-4393-9522-68b0fc370219-istiod-ca-cert" pod "7bc2103e-768a-4393-9522-68b0fc370219" (UID: "7bc2103e-768a-4393-9522-68b0fc370219"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:10:31.050008 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:31.049509 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bc2103e-768a-4393-9522-68b0fc370219-istio-data" pod "7bc2103e-768a-4393-9522-68b0fc370219" (UID: "7bc2103e-768a-4393-9522-68b0fc370219"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:10:31.050127 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:31.049548 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bc2103e-768a-4393-9522-68b0fc370219-workload-socket" pod "7bc2103e-768a-4393-9522-68b0fc370219" (UID: "7bc2103e-768a-4393-9522-68b0fc370219"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:10:31.050290 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:31.050270 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bc2103e-768a-4393-9522-68b0fc370219-credential-socket" pod "7bc2103e-768a-4393-9522-68b0fc370219" (UID: "7bc2103e-768a-4393-9522-68b0fc370219"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:10:31.053451 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:31.053422 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bc2103e-768a-4393-9522-68b0fc370219-kube-api-access-njw8k" pod "7bc2103e-768a-4393-9522-68b0fc370219" (UID: "7bc2103e-768a-4393-9522-68b0fc370219"). InnerVolumeSpecName "kube-api-access-njw8k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:10:31.055774 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:31.055751 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bc2103e-768a-4393-9522-68b0fc370219-istio-token" pod "7bc2103e-768a-4393-9522-68b0fc370219" (UID: "7bc2103e-768a-4393-9522-68b0fc370219"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:10:31.055874 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:31.055749 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7bc2103e-768a-4393-9522-68b0fc370219-istio-podinfo" pod "7bc2103e-768a-4393-9522-68b0fc370219" (UID: "7bc2103e-768a-4393-9522-68b0fc370219"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:10:31.057620 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:31.057591 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bc2103e-768a-4393-9522-68b0fc370219-istio-envoy" pod "7bc2103e-768a-4393-9522-68b0fc370219" (UID: "7bc2103e-768a-4393-9522-68b0fc370219"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:10:31.149913 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:31.149879 5057 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7bc2103e-768a-4393-9522-68b0fc370219-istio-data\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:10:31.149913 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:31.149913 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-njw8k\" (UniqueName: \"kubernetes.io/projected/7bc2103e-768a-4393-9522-68b0fc370219-kube-api-access-njw8k\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:10:31.150112 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:31.149923 5057 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7bc2103e-768a-4393-9522-68b0fc370219-workload-certs\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:10:31.150112 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:31.149933 5057 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bc2103e-768a-4393-9522-68b0fc370219-workload-socket\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:10:31.150112 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:31.149941 5057 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7bc2103e-768a-4393-9522-68b0fc370219-istiod-ca-cert\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:10:31.150112 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:31.149949 5057 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7bc2103e-768a-4393-9522-68b0fc370219-istio-podinfo\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:10:31.150112 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:31.149957 5057 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7bc2103e-768a-4393-9522-68b0fc370219-istio-token\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:10:31.150112 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:31.149964 5057 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bc2103e-768a-4393-9522-68b0fc370219-credential-socket\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:10:31.150112 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:31.149971 5057 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7bc2103e-768a-4393-9522-68b0fc370219-istio-envoy\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:10:31.366133 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:31.366092 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-a9ced3/autores-model-a9ced3-kserve-c48c69885-lwwnp" event={"ID":"902d109b-92a3-491e-8784-fe3f27a48dec","Type":"ContainerStarted","Data":"3e9ca39fc055490a4b7c77c0e55162a72bee3d5334702ea15e367625e3a968ca"} Oct 29 23:10:31.366608 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:31.366145 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-a9ced3/autores-model-a9ced3-kserve-c48c69885-lwwnp" event={"ID":"902d109b-92a3-491e-8784-fe3f27a48dec","Type":"ContainerStarted","Data":"58d3898dc427781e8de8500be30671b59a42f87fd7e1918579f61c2118689ee8"} Oct 29 23:10:31.368098 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:31.368066 5057 generic.go:356] "Generic (PLEG): container finished" podID="7bc2103e-768a-4393-9522-68b0fc370219" containerID="151c14950da1cf77656122259bc0c192ec6e36821c5b2a64e2e6dc54a4d6d3ba" exitCode=0 Oct 29 23:10:31.368257 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:31.368186 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-7smjh" event={"ID":"7bc2103e-768a-4393-9522-68b0fc370219","Type":"ContainerDied","Data":"151c14950da1cf77656122259bc0c192ec6e36821c5b2a64e2e6dc54a4d6d3ba"} Oct 29 23:10:31.368257 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:31.368233 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-7smjh" event={"ID":"7bc2103e-768a-4393-9522-68b0fc370219","Type":"ContainerDied","Data":"c460d10527202accda73cf505984dd3c983ec9d0a7c327d7decab3a6dfd6c1dd"} Oct 29 23:10:31.368413 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:31.368262 5057 scope.go:122] "RemoveContainer" containerID="151c14950da1cf77656122259bc0c192ec6e36821c5b2a64e2e6dc54a4d6d3ba" Oct 29 23:10:31.368475 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:31.368457 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-7smjh" Oct 29 23:10:31.395515 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:31.395438 5057 scope.go:122] "RemoveContainer" containerID="151c14950da1cf77656122259bc0c192ec6e36821c5b2a64e2e6dc54a4d6d3ba" Oct 29 23:10:31.396025 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:10:31.395997 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"151c14950da1cf77656122259bc0c192ec6e36821c5b2a64e2e6dc54a4d6d3ba\": container with ID starting with 151c14950da1cf77656122259bc0c192ec6e36821c5b2a64e2e6dc54a4d6d3ba not found: ID does not exist" containerID="151c14950da1cf77656122259bc0c192ec6e36821c5b2a64e2e6dc54a4d6d3ba" Oct 29 23:10:31.396143 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:31.396036 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"151c14950da1cf77656122259bc0c192ec6e36821c5b2a64e2e6dc54a4d6d3ba"} err="failed to get container status \"151c14950da1cf77656122259bc0c192ec6e36821c5b2a64e2e6dc54a4d6d3ba\": rpc error: code = NotFound desc = could not find container \"151c14950da1cf77656122259bc0c192ec6e36821c5b2a64e2e6dc54a4d6d3ba\": container with ID starting with 151c14950da1cf77656122259bc0c192ec6e36821c5b2a64e2e6dc54a4d6d3ba not found: ID does not exist" Oct 29 23:10:31.418957 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:31.418923 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:10:31.424652 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:31.424603 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-autores-a9ced3/autores-model-a9ced3-kserve-c48c69885-lwwnp" podStartSLOduration=1.424586153 podStartE2EDuration="1.424586153s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:10:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:10:31.407561983 +0000 UTC m=+2212.190775410" watchObservedRunningTime="2026-10-29 23:10:31.424586153 +0000 UTC m=+2212.207799580" Oct 29 23:10:31.447464 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:31.447438 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-7smjh"] Oct 29 23:10:31.452577 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:31.452555 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-7smjh"] Oct 29 23:10:31.532422 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:31.532391 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:10:31.542440 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:31.542406 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-a9ced3/autores-model-a9ced3-kserve-c48c69885-lwwnp" Oct 29 23:10:31.644789 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:31.644693 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:10:32.257622 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:32.257586 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-755453-788f5c78c-svk2k" Oct 29 23:10:32.877504 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:32.877451 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:10:33.264787 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:33.264760 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-755453-9bd867db-hdlwb" Oct 29 23:10:33.310881 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:33.310844 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:10:33.336312 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:33.336280 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7bc2103e-768a-4393-9522-68b0fc370219" path="/var/lib/kubelet/pods/7bc2103e-768a-4393-9522-68b0fc370219/volumes" Oct 29 23:10:33.448507 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:33.448445 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:10:33.809884 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:33.809832 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:10:33.929810 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:33.929775 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:10:34.101406 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:34.091567 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:10:35.168908 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:35.168868 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:10:35.278900 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:35.278853 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:10:35.389615 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:35.389580 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:10:37.040003 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:37.039967 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:10:37.412588 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:37.412304 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v2tfm"] Oct 29 23:10:38.158938 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:38.158897 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:10:38.277907 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:38.277873 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:10:39.828561 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:39.828511 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:10:39.998272 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:39.998234 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:10:40.110300 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:40.110201 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:10:40.173545 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:40.173509 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ed3bf2-dd855d495-fj85v"] Oct 29 23:10:40.174478 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:40.174458 5057 cpu_manager.go:414] "containerMap: removing container" podUID="7bc2103e-768a-4393-9522-68b0fc370219" containerName="istio-proxy" Oct 29 23:10:40.174478 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:40.174478 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7bc2103e-768a-4393-9522-68b0fc370219" containerName="istio-proxy" Oct 29 23:10:40.174630 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:40.174620 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="7bc2103e-768a-4393-9522-68b0fc370219" containerName="istio-proxy" Oct 29 23:10:40.178788 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:40.178756 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ed3bf2-dd855d495-fj85v" Oct 29 23:10:40.197031 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:40.197005 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ed3bf2-dd855d495-fj85v"] Oct 29 23:10:40.201790 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:40.200762 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-ed3bf2\"" Oct 29 23:10:40.201790 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:40.200819 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-ed3bf2-dockercfg-9rfsm\"" Oct 29 23:10:40.410251 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:40.410158 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/695c8042-ace4-420f-bdaf-11ca7e09ef5f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-ed3bf2-dd855d495-fj85v\" (UID: \"695c8042-ace4-420f-bdaf-11ca7e09ef5f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ed3bf2-dd855d495-fj85v" Oct 29 23:10:40.410483 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:40.410467 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-888cr\" (UniqueName: \"kubernetes.io/projected/695c8042-ace4-420f-bdaf-11ca7e09ef5f-kube-api-access-888cr\") pod \"payload-pre-processing-e2e-shared-b-w2-ed3bf2-dd855d495-fj85v\" (UID: \"695c8042-ace4-420f-bdaf-11ca7e09ef5f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ed3bf2-dd855d495-fj85v" Oct 29 23:10:40.427359 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:40.425858 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-ed3bf2-7cbcbd6758-2gkwj"] Oct 29 23:10:40.438775 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:40.437883 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ed3bf2-7cbcbd6758-2gkwj" Oct 29 23:10:40.487113 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:40.487082 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-ed3bf2-7cbcbd6758-2gkwj"] Oct 29 23:10:40.512630 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:40.512591 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnlqc\" (UniqueName: \"kubernetes.io/projected/56962a7e-7f67-4151-b04a-c4161d2de549-kube-api-access-qnlqc\") pod \"payload-processing-e2e-shared-b-w2-ed3bf2-7cbcbd6758-2gkwj\" (UID: \"56962a7e-7f67-4151-b04a-c4161d2de549\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ed3bf2-7cbcbd6758-2gkwj" Oct 29 23:10:40.512813 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:40.512657 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56962a7e-7f67-4151-b04a-c4161d2de549-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-ed3bf2-7cbcbd6758-2gkwj\" (UID: \"56962a7e-7f67-4151-b04a-c4161d2de549\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ed3bf2-7cbcbd6758-2gkwj" Oct 29 23:10:40.512813 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:40.512707 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/695c8042-ace4-420f-bdaf-11ca7e09ef5f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-ed3bf2-dd855d495-fj85v\" (UID: \"695c8042-ace4-420f-bdaf-11ca7e09ef5f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ed3bf2-dd855d495-fj85v" Oct 29 23:10:40.513350 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:40.513306 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-888cr\" (UniqueName: \"kubernetes.io/projected/695c8042-ace4-420f-bdaf-11ca7e09ef5f-kube-api-access-888cr\") pod \"payload-pre-processing-e2e-shared-b-w2-ed3bf2-dd855d495-fj85v\" (UID: \"695c8042-ace4-420f-bdaf-11ca7e09ef5f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ed3bf2-dd855d495-fj85v" Oct 29 23:10:40.515863 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:40.514349 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/695c8042-ace4-420f-bdaf-11ca7e09ef5f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-ed3bf2-dd855d495-fj85v\" (UID: \"695c8042-ace4-420f-bdaf-11ca7e09ef5f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ed3bf2-dd855d495-fj85v" Oct 29 23:10:40.541490 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:40.541462 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-888cr\" (UniqueName: \"kubernetes.io/projected/695c8042-ace4-420f-bdaf-11ca7e09ef5f-kube-api-access-888cr\") pod \"payload-pre-processing-e2e-shared-b-w2-ed3bf2-dd855d495-fj85v\" (UID: \"695c8042-ace4-420f-bdaf-11ca7e09ef5f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ed3bf2-dd855d495-fj85v" Oct 29 23:10:40.546771 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:40.543186 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-autores-a9ced3/autores-model-a9ced3-kserve-c48c69885-lwwnp" Oct 29 23:10:40.562800 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:40.562771 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-autores-a9ced3/autores-model-a9ced3-kserve-c48c69885-lwwnp" Oct 29 23:10:40.617290 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:40.617260 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnlqc\" (UniqueName: \"kubernetes.io/projected/56962a7e-7f67-4151-b04a-c4161d2de549-kube-api-access-qnlqc\") pod \"payload-processing-e2e-shared-b-w2-ed3bf2-7cbcbd6758-2gkwj\" (UID: \"56962a7e-7f67-4151-b04a-c4161d2de549\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ed3bf2-7cbcbd6758-2gkwj" Oct 29 23:10:40.617457 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:40.617297 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56962a7e-7f67-4151-b04a-c4161d2de549-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-ed3bf2-7cbcbd6758-2gkwj\" (UID: \"56962a7e-7f67-4151-b04a-c4161d2de549\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ed3bf2-7cbcbd6758-2gkwj" Oct 29 23:10:40.618270 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:40.618233 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56962a7e-7f67-4151-b04a-c4161d2de549-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-ed3bf2-7cbcbd6758-2gkwj\" (UID: \"56962a7e-7f67-4151-b04a-c4161d2de549\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ed3bf2-7cbcbd6758-2gkwj" Oct 29 23:10:40.643214 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:40.643186 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnlqc\" (UniqueName: \"kubernetes.io/projected/56962a7e-7f67-4151-b04a-c4161d2de549-kube-api-access-qnlqc\") pod \"payload-processing-e2e-shared-b-w2-ed3bf2-7cbcbd6758-2gkwj\" (UID: \"56962a7e-7f67-4151-b04a-c4161d2de549\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ed3bf2-7cbcbd6758-2gkwj" Oct 29 23:10:40.794127 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:40.794093 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ed3bf2-7cbcbd6758-2gkwj" Oct 29 23:10:40.801814 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:40.801786 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ed3bf2-dd855d495-fj85v" Oct 29 23:10:41.108158 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:41.108100 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-ed3bf2-7cbcbd6758-2gkwj"] Oct 29 23:10:41.144806 ip-10-0-140-74 kubenswrapper[5057]: W1029 23:10:41.144778 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod695c8042_ace4_420f_bdaf_11ca7e09ef5f.slice/crio-aac3c58bc99e59162f229bc530500eebade7d9b7de4ef21c7dcd4fda5c6b9c04 WatchSource:0}: Error finding container aac3c58bc99e59162f229bc530500eebade7d9b7de4ef21c7dcd4fda5c6b9c04: Status 404 returned error can't find the container with id aac3c58bc99e59162f229bc530500eebade7d9b7de4ef21c7dcd4fda5c6b9c04 Oct 29 23:10:41.145941 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:41.145917 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ed3bf2-dd855d495-fj85v"] Oct 29 23:10:41.550537 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:41.550451 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ed3bf2-dd855d495-fj85v" event={"ID":"695c8042-ace4-420f-bdaf-11ca7e09ef5f","Type":"ContainerStarted","Data":"d21b8bfa1035c412881c363719348641f520a2be385a9318dfad63b541eb798d"} Oct 29 23:10:41.550537 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:41.550517 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ed3bf2-dd855d495-fj85v" event={"ID":"695c8042-ace4-420f-bdaf-11ca7e09ef5f","Type":"ContainerStarted","Data":"aac3c58bc99e59162f229bc530500eebade7d9b7de4ef21c7dcd4fda5c6b9c04"} Oct 29 23:10:41.550852 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:41.550830 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ed3bf2-dd855d495-fj85v" Oct 29 23:10:41.552749 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:41.552722 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ed3bf2-7cbcbd6758-2gkwj" event={"ID":"56962a7e-7f67-4151-b04a-c4161d2de549","Type":"ContainerStarted","Data":"c1474dda4e4d0b5a231c642a7efa78b6c0bc880aec7432a2396b95379b37ce36"} Oct 29 23:10:41.552898 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:41.552775 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ed3bf2-7cbcbd6758-2gkwj" event={"ID":"56962a7e-7f67-4151-b04a-c4161d2de549","Type":"ContainerStarted","Data":"3a780ecb4825e316aac3531e77733f553506580f7bb00d95cc849ade2c567f85"} Oct 29 23:10:41.552898 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:41.552811 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ed3bf2-7cbcbd6758-2gkwj" Oct 29 23:10:41.557296 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:41.557262 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-autores-a9ced3/autores-model-a9ced3-kserve-c48c69885-lwwnp" Oct 29 23:10:41.596271 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:41.596221 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ed3bf2-dd855d495-fj85v" podStartSLOduration=1.5962062430000001 podStartE2EDuration="1.596206243s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:10:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:10:41.591857495 +0000 UTC m=+2222.375070921" watchObservedRunningTime="2026-10-29 23:10:41.596206243 +0000 UTC m=+2222.379419683" Oct 29 23:10:41.635513 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:41.635463 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ed3bf2-7cbcbd6758-2gkwj" podStartSLOduration=1.635449141 podStartE2EDuration="1.635449141s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:10:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:10:41.631027831 +0000 UTC m=+2222.414241257" watchObservedRunningTime="2026-10-29 23:10:41.635449141 +0000 UTC m=+2222.418662567" Oct 29 23:10:42.286150 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:42.286089 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:10:42.400965 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:42.400924 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:10:44.461355 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:44.458215 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:10:46.111151 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:46.111113 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:10:47.718985 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:47.718949 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:10:49.031368 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:10:49.031316 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d2dnn" containerName="extractor" Oct 29 23:10:49.390917 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:49.390821 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:10:50.883496 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:50.883457 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:10:51.004033 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:51.003860 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:10:51.118651 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:51.118613 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:10:52.566346 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:52.566294 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ed3bf2-dd855d495-fj85v" Oct 29 23:10:52.566831 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:52.566413 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ed3bf2-7cbcbd6758-2gkwj" Oct 29 23:10:52.834518 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:52.834437 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:10:54.604781 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:54.604741 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:10:56.302920 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:56.302871 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:10:57.115643 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:57.115601 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:10:57.240544 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:57.240505 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:10:58.012617 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:58.012580 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:10:58.367700 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:58.367614 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-ngbdj"] Oct 29 23:10:58.374070 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:58.374042 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-ngbdj" Oct 29 23:10:58.444395 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:58.436486 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-ngbdj"] Oct 29 23:10:58.447502 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:58.447469 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7884e38e-4574-43f4-af11-f9414580568f-workload-socket\") pod \"e2e-autores-a9ced3-openshift-default-7d8898bf57-ngbdj\" (UID: \"7884e38e-4574-43f4-af11-f9414580568f\") " pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-ngbdj" Oct 29 23:10:58.447661 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:58.447594 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7884e38e-4574-43f4-af11-f9414580568f-credential-socket\") pod \"e2e-autores-a9ced3-openshift-default-7d8898bf57-ngbdj\" (UID: \"7884e38e-4574-43f4-af11-f9414580568f\") " pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-ngbdj" Oct 29 23:10:58.447743 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:58.447688 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7884e38e-4574-43f4-af11-f9414580568f-workload-certs\") pod \"e2e-autores-a9ced3-openshift-default-7d8898bf57-ngbdj\" (UID: \"7884e38e-4574-43f4-af11-f9414580568f\") " pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-ngbdj" Oct 29 23:10:58.447743 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:58.447731 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7884e38e-4574-43f4-af11-f9414580568f-istio-token\") pod \"e2e-autores-a9ced3-openshift-default-7d8898bf57-ngbdj\" (UID: \"7884e38e-4574-43f4-af11-f9414580568f\") " pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-ngbdj" Oct 29 23:10:58.447859 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:58.447796 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7884e38e-4574-43f4-af11-f9414580568f-istio-data\") pod \"e2e-autores-a9ced3-openshift-default-7d8898bf57-ngbdj\" (UID: \"7884e38e-4574-43f4-af11-f9414580568f\") " pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-ngbdj" Oct 29 23:10:58.447918 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:58.447887 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7884e38e-4574-43f4-af11-f9414580568f-istio-podinfo\") pod \"e2e-autores-a9ced3-openshift-default-7d8898bf57-ngbdj\" (UID: \"7884e38e-4574-43f4-af11-f9414580568f\") " pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-ngbdj" Oct 29 23:10:58.447981 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:58.447927 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rtfb\" (UniqueName: \"kubernetes.io/projected/7884e38e-4574-43f4-af11-f9414580568f-kube-api-access-9rtfb\") pod \"e2e-autores-a9ced3-openshift-default-7d8898bf57-ngbdj\" (UID: \"7884e38e-4574-43f4-af11-f9414580568f\") " pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-ngbdj" Oct 29 23:10:58.448042 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:58.448017 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7884e38e-4574-43f4-af11-f9414580568f-istio-envoy\") pod \"e2e-autores-a9ced3-openshift-default-7d8898bf57-ngbdj\" (UID: \"7884e38e-4574-43f4-af11-f9414580568f\") " pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-ngbdj" Oct 29 23:10:58.448099 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:58.448051 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7884e38e-4574-43f4-af11-f9414580568f-istiod-ca-cert\") pod \"e2e-autores-a9ced3-openshift-default-7d8898bf57-ngbdj\" (UID: \"7884e38e-4574-43f4-af11-f9414580568f\") " pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-ngbdj" Oct 29 23:10:58.549533 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:58.549503 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7884e38e-4574-43f4-af11-f9414580568f-workload-socket\") pod \"e2e-autores-a9ced3-openshift-default-7d8898bf57-ngbdj\" (UID: \"7884e38e-4574-43f4-af11-f9414580568f\") " pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-ngbdj" Oct 29 23:10:58.549690 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:58.549570 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7884e38e-4574-43f4-af11-f9414580568f-credential-socket\") pod \"e2e-autores-a9ced3-openshift-default-7d8898bf57-ngbdj\" (UID: \"7884e38e-4574-43f4-af11-f9414580568f\") " pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-ngbdj" Oct 29 23:10:58.549690 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:58.549613 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7884e38e-4574-43f4-af11-f9414580568f-workload-certs\") pod \"e2e-autores-a9ced3-openshift-default-7d8898bf57-ngbdj\" (UID: \"7884e38e-4574-43f4-af11-f9414580568f\") " pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-ngbdj" Oct 29 23:10:58.549690 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:58.549635 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7884e38e-4574-43f4-af11-f9414580568f-istio-token\") pod \"e2e-autores-a9ced3-openshift-default-7d8898bf57-ngbdj\" (UID: \"7884e38e-4574-43f4-af11-f9414580568f\") " pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-ngbdj" Oct 29 23:10:58.549690 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:58.549671 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7884e38e-4574-43f4-af11-f9414580568f-istio-data\") pod \"e2e-autores-a9ced3-openshift-default-7d8898bf57-ngbdj\" (UID: \"7884e38e-4574-43f4-af11-f9414580568f\") " pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-ngbdj" Oct 29 23:10:58.549899 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:58.549718 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7884e38e-4574-43f4-af11-f9414580568f-istio-podinfo\") pod \"e2e-autores-a9ced3-openshift-default-7d8898bf57-ngbdj\" (UID: \"7884e38e-4574-43f4-af11-f9414580568f\") " pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-ngbdj" Oct 29 23:10:58.549899 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:58.549738 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rtfb\" (UniqueName: \"kubernetes.io/projected/7884e38e-4574-43f4-af11-f9414580568f-kube-api-access-9rtfb\") pod \"e2e-autores-a9ced3-openshift-default-7d8898bf57-ngbdj\" (UID: \"7884e38e-4574-43f4-af11-f9414580568f\") " pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-ngbdj" Oct 29 23:10:58.549899 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:58.549795 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7884e38e-4574-43f4-af11-f9414580568f-istio-envoy\") pod \"e2e-autores-a9ced3-openshift-default-7d8898bf57-ngbdj\" (UID: \"7884e38e-4574-43f4-af11-f9414580568f\") " pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-ngbdj" Oct 29 23:10:58.549899 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:58.549816 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7884e38e-4574-43f4-af11-f9414580568f-istiod-ca-cert\") pod \"e2e-autores-a9ced3-openshift-default-7d8898bf57-ngbdj\" (UID: \"7884e38e-4574-43f4-af11-f9414580568f\") " pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-ngbdj" Oct 29 23:10:58.550130 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:58.549936 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7884e38e-4574-43f4-af11-f9414580568f-workload-socket\") pod \"e2e-autores-a9ced3-openshift-default-7d8898bf57-ngbdj\" (UID: \"7884e38e-4574-43f4-af11-f9414580568f\") " pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-ngbdj" Oct 29 23:10:58.550130 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:58.550016 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7884e38e-4574-43f4-af11-f9414580568f-credential-socket\") pod \"e2e-autores-a9ced3-openshift-default-7d8898bf57-ngbdj\" (UID: \"7884e38e-4574-43f4-af11-f9414580568f\") " pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-ngbdj" Oct 29 23:10:58.550130 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:58.550037 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7884e38e-4574-43f4-af11-f9414580568f-workload-certs\") pod \"e2e-autores-a9ced3-openshift-default-7d8898bf57-ngbdj\" (UID: \"7884e38e-4574-43f4-af11-f9414580568f\") " pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-ngbdj" Oct 29 23:10:58.550252 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:58.550133 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7884e38e-4574-43f4-af11-f9414580568f-istio-data\") pod \"e2e-autores-a9ced3-openshift-default-7d8898bf57-ngbdj\" (UID: \"7884e38e-4574-43f4-af11-f9414580568f\") " pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-ngbdj" Oct 29 23:10:58.550560 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:58.550534 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7884e38e-4574-43f4-af11-f9414580568f-istiod-ca-cert\") pod \"e2e-autores-a9ced3-openshift-default-7d8898bf57-ngbdj\" (UID: \"7884e38e-4574-43f4-af11-f9414580568f\") " pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-ngbdj" Oct 29 23:10:58.551989 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:58.551971 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7884e38e-4574-43f4-af11-f9414580568f-istio-envoy\") pod \"e2e-autores-a9ced3-openshift-default-7d8898bf57-ngbdj\" (UID: \"7884e38e-4574-43f4-af11-f9414580568f\") " pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-ngbdj" Oct 29 23:10:58.552224 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:58.552208 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7884e38e-4574-43f4-af11-f9414580568f-istio-podinfo\") pod \"e2e-autores-a9ced3-openshift-default-7d8898bf57-ngbdj\" (UID: \"7884e38e-4574-43f4-af11-f9414580568f\") " pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-ngbdj" Oct 29 23:10:58.572595 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:58.572567 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7884e38e-4574-43f4-af11-f9414580568f-istio-token\") pod \"e2e-autores-a9ced3-openshift-default-7d8898bf57-ngbdj\" (UID: \"7884e38e-4574-43f4-af11-f9414580568f\") " pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-ngbdj" Oct 29 23:10:58.572711 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:58.572657 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rtfb\" (UniqueName: \"kubernetes.io/projected/7884e38e-4574-43f4-af11-f9414580568f-kube-api-access-9rtfb\") pod \"e2e-autores-a9ced3-openshift-default-7d8898bf57-ngbdj\" (UID: \"7884e38e-4574-43f4-af11-f9414580568f\") " pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-ngbdj" Oct 29 23:10:58.734880 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:58.734846 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-ngbdj" Oct 29 23:10:59.060707 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:59.060676 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-ngbdj"] Oct 29 23:10:59.061323 ip-10-0-140-74 kubenswrapper[5057]: W1029 23:10:59.061294 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7884e38e_4574_43f4_af11_f9414580568f.slice/crio-393719fe756016c9321bfda1292355d71145402249a0fd78c789c4c39c81048a WatchSource:0}: Error finding container 393719fe756016c9321bfda1292355d71145402249a0fd78c789c4c39c81048a: Status 404 returned error can't find the container with id 393719fe756016c9321bfda1292355d71145402249a0fd78c789c4c39c81048a Oct 29 23:10:59.064180 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:59.064148 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 23:10:59.064285 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:59.064221 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 23:10:59.064285 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:59.064251 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 23:10:59.700868 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:59.700828 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:10:59.732281 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:59.732241 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-ngbdj" event={"ID":"7884e38e-4574-43f4-af11-f9414580568f","Type":"ContainerStarted","Data":"e7ed362e274d797012f28a969ceb923a7f7b13d1844d500edd7176907516d200"} Oct 29 23:10:59.732505 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:59.732300 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-ngbdj" event={"ID":"7884e38e-4574-43f4-af11-f9414580568f","Type":"ContainerStarted","Data":"393719fe756016c9321bfda1292355d71145402249a0fd78c789c4c39c81048a"} Oct 29 23:10:59.736041 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:59.736016 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-ngbdj" Oct 29 23:10:59.803233 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:10:59.803164 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-ngbdj" podStartSLOduration=1.803143479 podStartE2EDuration="1.803143479s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:10:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:10:59.789802777 +0000 UTC m=+2240.573016204" watchObservedRunningTime="2026-10-29 23:10:59.803143479 +0000 UTC m=+2240.586356905" Oct 29 23:11:00.736695 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:00.736660 5057 patch_prober.go:29] interesting pod/e2e-autores-a9ced3-openshift-default-7d8898bf57-ngbdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" start-of-body= Oct 29 23:11:00.737080 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:00.736734 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-ngbdj" podUID="7884e38e-4574-43f4-af11-f9414580568f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" Oct 29 23:11:01.608608 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:01.608522 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:11:01.735281 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:01.735246 5057 patch_prober.go:29] interesting pod/e2e-autores-a9ced3-openshift-default-7d8898bf57-ngbdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" start-of-body= Oct 29 23:11:01.735468 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:01.735294 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-ngbdj" podUID="7884e38e-4574-43f4-af11-f9414580568f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" Oct 29 23:11:01.933406 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:01.923085 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-a9ced3/autores-model-a9ced3-kserve-c48c69885-lwwnp"] Oct 29 23:11:01.984912 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:01.984872 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:11:02.736782 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:02.736746 5057 patch_prober.go:29] interesting pod/e2e-autores-a9ced3-openshift-default-7d8898bf57-ngbdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" start-of-body= Oct 29 23:11:02.736962 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:02.736802 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-ngbdj" podUID="7884e38e-4574-43f4-af11-f9414580568f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" Oct 29 23:11:03.735416 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:03.735382 5057 patch_prober.go:29] interesting pod/e2e-autores-a9ced3-openshift-default-7d8898bf57-ngbdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" start-of-body= Oct 29 23:11:03.735812 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:03.735448 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-ngbdj" podUID="7884e38e-4574-43f4-af11-f9414580568f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" Oct 29 23:11:04.016283 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:04.016173 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:11:04.735223 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:04.735163 5057 patch_prober.go:29] interesting pod/e2e-autores-a9ced3-openshift-default-7d8898bf57-ngbdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" start-of-body= Oct 29 23:11:04.735429 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:04.735240 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-ngbdj" podUID="7884e38e-4574-43f4-af11-f9414580568f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" Oct 29 23:11:05.833443 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:05.828466 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:11:06.495240 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:06.495200 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-ngbdj" Oct 29 23:11:06.495824 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:06.495714 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-ngbdj" Oct 29 23:11:06.497698 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:06.497682 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-ngbdj" Oct 29 23:11:07.744973 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:07.744918 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:11:08.042297 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:08.042210 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:11:09.862001 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:09.861875 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:11:09.970868 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:09.970828 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:11:10.080677 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:10.080640 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:11:12.023563 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:12.023530 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:11:13.031968 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:13.031930 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:11:13.147948 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:13.147905 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:11:14.185217 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:14.185162 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:11:14.329211 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:14.329176 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:11:16.019293 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:16.019252 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:11:17.242080 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:17.242043 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-autores-a9ced3/autores-model-a9ced3-kserve-c48c69885-lwwnp" podUID="902d109b-92a3-491e-8784-fe3f27a48dec" containerName="main" containerID="cri-o://3e9ca39fc055490a4b7c77c0e55162a72bee3d5334702ea15e367625e3a968ca" gracePeriod=45 Oct 29 23:11:17.819140 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:17.819107 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-a9ced3/autores-model-a9ced3-kserve-c48c69885-lwwnp" Oct 29 23:11:17.871173 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:17.871141 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/902d109b-92a3-491e-8784-fe3f27a48dec-kube-api-access-lvtk8\" (UniqueName: \"kubernetes.io/projected/902d109b-92a3-491e-8784-fe3f27a48dec-kube-api-access-lvtk8\") pod \"902d109b-92a3-491e-8784-fe3f27a48dec\" (UID: \"902d109b-92a3-491e-8784-fe3f27a48dec\") " Oct 29 23:11:17.871388 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:17.871257 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/902d109b-92a3-491e-8784-fe3f27a48dec-home\" (UniqueName: \"kubernetes.io/empty-dir/902d109b-92a3-491e-8784-fe3f27a48dec-home\") pod \"902d109b-92a3-491e-8784-fe3f27a48dec\" (UID: \"902d109b-92a3-491e-8784-fe3f27a48dec\") " Oct 29 23:11:17.871388 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:17.871351 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/902d109b-92a3-491e-8784-fe3f27a48dec-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/902d109b-92a3-491e-8784-fe3f27a48dec-tmp-dir\") pod \"902d109b-92a3-491e-8784-fe3f27a48dec\" (UID: \"902d109b-92a3-491e-8784-fe3f27a48dec\") " Oct 29 23:11:17.871519 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:17.871399 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/902d109b-92a3-491e-8784-fe3f27a48dec-tls-certs\" (UniqueName: \"kubernetes.io/secret/902d109b-92a3-491e-8784-fe3f27a48dec-tls-certs\") pod \"902d109b-92a3-491e-8784-fe3f27a48dec\" (UID: \"902d109b-92a3-491e-8784-fe3f27a48dec\") " Oct 29 23:11:17.871519 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:17.871454 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/902d109b-92a3-491e-8784-fe3f27a48dec-dshm\" (UniqueName: \"kubernetes.io/empty-dir/902d109b-92a3-491e-8784-fe3f27a48dec-dshm\") pod \"902d109b-92a3-491e-8784-fe3f27a48dec\" (UID: \"902d109b-92a3-491e-8784-fe3f27a48dec\") " Oct 29 23:11:17.871639 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:17.871587 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/902d109b-92a3-491e-8784-fe3f27a48dec-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/902d109b-92a3-491e-8784-fe3f27a48dec-model-cache\") pod \"902d109b-92a3-491e-8784-fe3f27a48dec\" (UID: \"902d109b-92a3-491e-8784-fe3f27a48dec\") " Oct 29 23:11:17.871639 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:17.871595 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/902d109b-92a3-491e-8784-fe3f27a48dec-home" pod "902d109b-92a3-491e-8784-fe3f27a48dec" (UID: "902d109b-92a3-491e-8784-fe3f27a48dec"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:11:17.871793 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:17.871643 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/902d109b-92a3-491e-8784-fe3f27a48dec-tmp-dir" pod "902d109b-92a3-491e-8784-fe3f27a48dec" (UID: "902d109b-92a3-491e-8784-fe3f27a48dec"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:11:17.871903 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:17.871884 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/902d109b-92a3-491e-8784-fe3f27a48dec-model-cache" pod "902d109b-92a3-491e-8784-fe3f27a48dec" (UID: "902d109b-92a3-491e-8784-fe3f27a48dec"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:11:17.872570 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:17.872547 5057 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/902d109b-92a3-491e-8784-fe3f27a48dec-home\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:11:17.872672 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:17.872572 5057 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/902d109b-92a3-491e-8784-fe3f27a48dec-tmp-dir\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:11:17.872672 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:17.872586 5057 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/902d109b-92a3-491e-8784-fe3f27a48dec-model-cache\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:11:17.874038 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:17.874016 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/902d109b-92a3-491e-8784-fe3f27a48dec-tls-certs" pod "902d109b-92a3-491e-8784-fe3f27a48dec" (UID: "902d109b-92a3-491e-8784-fe3f27a48dec"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:11:17.874238 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:17.874216 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/902d109b-92a3-491e-8784-fe3f27a48dec-dshm" pod "902d109b-92a3-491e-8784-fe3f27a48dec" (UID: "902d109b-92a3-491e-8784-fe3f27a48dec"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:11:17.874384 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:17.874313 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/902d109b-92a3-491e-8784-fe3f27a48dec-kube-api-access-lvtk8" pod "902d109b-92a3-491e-8784-fe3f27a48dec" (UID: "902d109b-92a3-491e-8784-fe3f27a48dec"). InnerVolumeSpecName "kube-api-access-lvtk8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:11:17.945536 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:17.945497 5057 generic.go:356] "Generic (PLEG): container finished" podID="902d109b-92a3-491e-8784-fe3f27a48dec" containerID="3e9ca39fc055490a4b7c77c0e55162a72bee3d5334702ea15e367625e3a968ca" exitCode=0 Oct 29 23:11:17.945707 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:17.945543 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-a9ced3/autores-model-a9ced3-kserve-c48c69885-lwwnp" Oct 29 23:11:17.945707 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:17.945560 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-a9ced3/autores-model-a9ced3-kserve-c48c69885-lwwnp" event={"ID":"902d109b-92a3-491e-8784-fe3f27a48dec","Type":"ContainerDied","Data":"3e9ca39fc055490a4b7c77c0e55162a72bee3d5334702ea15e367625e3a968ca"} Oct 29 23:11:17.945707 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:17.945591 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-a9ced3/autores-model-a9ced3-kserve-c48c69885-lwwnp" event={"ID":"902d109b-92a3-491e-8784-fe3f27a48dec","Type":"ContainerDied","Data":"58d3898dc427781e8de8500be30671b59a42f87fd7e1918579f61c2118689ee8"} Oct 29 23:11:17.945707 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:17.945607 5057 scope.go:122] "RemoveContainer" containerID="3e9ca39fc055490a4b7c77c0e55162a72bee3d5334702ea15e367625e3a968ca" Oct 29 23:11:17.958854 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:17.958828 5057 scope.go:122] "RemoveContainer" containerID="3e9ca39fc055490a4b7c77c0e55162a72bee3d5334702ea15e367625e3a968ca" Oct 29 23:11:17.959129 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:11:17.959109 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e9ca39fc055490a4b7c77c0e55162a72bee3d5334702ea15e367625e3a968ca\": container with ID starting with 3e9ca39fc055490a4b7c77c0e55162a72bee3d5334702ea15e367625e3a968ca not found: ID does not exist" containerID="3e9ca39fc055490a4b7c77c0e55162a72bee3d5334702ea15e367625e3a968ca" Oct 29 23:11:17.959207 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:17.959137 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e9ca39fc055490a4b7c77c0e55162a72bee3d5334702ea15e367625e3a968ca"} err="failed to get container status \"3e9ca39fc055490a4b7c77c0e55162a72bee3d5334702ea15e367625e3a968ca\": rpc error: code = NotFound desc = could not find container \"3e9ca39fc055490a4b7c77c0e55162a72bee3d5334702ea15e367625e3a968ca\": container with ID starting with 3e9ca39fc055490a4b7c77c0e55162a72bee3d5334702ea15e367625e3a968ca not found: ID does not exist" Oct 29 23:11:17.974025 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:17.973996 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lvtk8\" (UniqueName: \"kubernetes.io/projected/902d109b-92a3-491e-8784-fe3f27a48dec-kube-api-access-lvtk8\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:11:17.974110 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:17.974022 5057 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/902d109b-92a3-491e-8784-fe3f27a48dec-tls-certs\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:11:17.974110 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:17.974040 5057 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/902d109b-92a3-491e-8784-fe3f27a48dec-dshm\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:11:17.999983 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:17.999956 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-a9ced3/autores-model-a9ced3-kserve-c48c69885-lwwnp"] Oct 29 23:11:18.007265 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:18.007241 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-autores-a9ced3/autores-model-a9ced3-kserve-c48c69885-lwwnp"] Oct 29 23:11:18.008877 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:18.008844 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:11:18.671982 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:18.671943 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-autores-a9ced3/autores-model-a9ced3-kserve-c48c69885-lwwnp" podUID="902d109b-92a3-491e-8784-fe3f27a48dec" containerName="main" probeResult="failure" output="Get \"https://10.132.0.170:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:11:19.030305 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:11:19.030262 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d2dnn" containerName="extractor" Oct 29 23:11:19.332964 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:19.332888 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="902d109b-92a3-491e-8784-fe3f27a48dec" path="/var/lib/kubelet/pods/902d109b-92a3-491e-8784-fe3f27a48dec/volumes" Oct 29 23:11:19.893553 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:19.893522 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:11:21.548979 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:21.548939 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:11:21.666861 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:21.666269 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:11:22.006516 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:22.006478 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:11:22.493796 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:22.493759 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:11:22.693546 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:22.693507 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:11:23.164961 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:23.164925 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:11:23.881828 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:23.881788 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:11:24.057670 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:24.057631 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:11:24.438551 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:24.438513 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:11:24.718283 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:24.718252 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:11:26.690501 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:26.690463 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:11:27.811212 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:27.811174 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:11:27.929960 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:27.929922 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:11:28.285199 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:28.285168 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:11:28.555968 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:28.555876 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:11:29.496546 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:29.494449 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:11:29.603781 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:29.603737 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:11:29.657394 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:29.657356 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-b8f5c4-cdd8644c9-hpdp5"] Oct 29 23:11:29.658274 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:29.658257 5057 cpu_manager.go:414] "containerMap: removing container" podUID="902d109b-92a3-491e-8784-fe3f27a48dec" containerName="main" Oct 29 23:11:29.658361 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:29.658276 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="902d109b-92a3-491e-8784-fe3f27a48dec" containerName="main" Oct 29 23:11:29.658466 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:29.658456 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="902d109b-92a3-491e-8784-fe3f27a48dec" containerName="main" Oct 29 23:11:29.663730 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:29.663710 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-b8f5c4-cdd8644c9-hpdp5" Oct 29 23:11:29.681498 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:29.681474 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-b8f5c4-cdd8644c9-hpdp5"] Oct 29 23:11:29.695131 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:29.695104 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w6-b8f5c4\"" Oct 29 23:11:29.695463 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:29.695443 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w6-b8f5c4-dockercfg-hxl4q\"" Oct 29 23:11:29.734684 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:29.734647 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6c69267-32e0-4a27-a376-d9918d3f2e3b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w6-b8f5c4-cdd8644c9-hpdp5\" (UID: \"e6c69267-32e0-4a27-a376-d9918d3f2e3b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-b8f5c4-cdd8644c9-hpdp5" Oct 29 23:11:29.734847 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:29.734700 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwfbd\" (UniqueName: \"kubernetes.io/projected/e6c69267-32e0-4a27-a376-d9918d3f2e3b-kube-api-access-pwfbd\") pod \"payload-pre-processing-e2e-shared-b-w6-b8f5c4-cdd8644c9-hpdp5\" (UID: \"e6c69267-32e0-4a27-a376-d9918d3f2e3b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-b8f5c4-cdd8644c9-hpdp5" Oct 29 23:11:29.752517 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:29.752446 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-b8f5c4-584577d98b-tpwp2"] Oct 29 23:11:29.758501 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:29.758479 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-b8f5c4-584577d98b-tpwp2" Oct 29 23:11:29.774794 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:29.774766 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-b8f5c4-584577d98b-tpwp2"] Oct 29 23:11:29.844983 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:29.844927 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6c69267-32e0-4a27-a376-d9918d3f2e3b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w6-b8f5c4-cdd8644c9-hpdp5\" (UID: \"e6c69267-32e0-4a27-a376-d9918d3f2e3b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-b8f5c4-cdd8644c9-hpdp5" Oct 29 23:11:29.857771 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:29.846001 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6c69267-32e0-4a27-a376-d9918d3f2e3b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w6-b8f5c4-cdd8644c9-hpdp5\" (UID: \"e6c69267-32e0-4a27-a376-d9918d3f2e3b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-b8f5c4-cdd8644c9-hpdp5" Oct 29 23:11:29.858081 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:29.858061 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwfbd\" (UniqueName: \"kubernetes.io/projected/e6c69267-32e0-4a27-a376-d9918d3f2e3b-kube-api-access-pwfbd\") pod \"payload-pre-processing-e2e-shared-b-w6-b8f5c4-cdd8644c9-hpdp5\" (UID: \"e6c69267-32e0-4a27-a376-d9918d3f2e3b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-b8f5c4-cdd8644c9-hpdp5" Oct 29 23:11:29.858326 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:29.858305 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/074dc860-7008-4c4f-9c9e-0fe9059b1947-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w6-b8f5c4-584577d98b-tpwp2\" (UID: \"074dc860-7008-4c4f-9c9e-0fe9059b1947\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-b8f5c4-584577d98b-tpwp2" Oct 29 23:11:29.858784 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:29.858760 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4lzvk\" (UniqueName: \"kubernetes.io/projected/074dc860-7008-4c4f-9c9e-0fe9059b1947-kube-api-access-4lzvk\") pod \"payload-processing-e2e-shared-b-w6-b8f5c4-584577d98b-tpwp2\" (UID: \"074dc860-7008-4c4f-9c9e-0fe9059b1947\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-b8f5c4-584577d98b-tpwp2" Oct 29 23:11:29.909523 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:29.899412 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwfbd\" (UniqueName: \"kubernetes.io/projected/e6c69267-32e0-4a27-a376-d9918d3f2e3b-kube-api-access-pwfbd\") pod \"payload-pre-processing-e2e-shared-b-w6-b8f5c4-cdd8644c9-hpdp5\" (UID: \"e6c69267-32e0-4a27-a376-d9918d3f2e3b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-b8f5c4-cdd8644c9-hpdp5" Oct 29 23:11:29.973907 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:29.973863 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4lzvk\" (UniqueName: \"kubernetes.io/projected/074dc860-7008-4c4f-9c9e-0fe9059b1947-kube-api-access-4lzvk\") pod \"payload-processing-e2e-shared-b-w6-b8f5c4-584577d98b-tpwp2\" (UID: \"074dc860-7008-4c4f-9c9e-0fe9059b1947\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-b8f5c4-584577d98b-tpwp2" Oct 29 23:11:29.974748 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:29.974721 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/074dc860-7008-4c4f-9c9e-0fe9059b1947-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w6-b8f5c4-584577d98b-tpwp2\" (UID: \"074dc860-7008-4c4f-9c9e-0fe9059b1947\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-b8f5c4-584577d98b-tpwp2" Oct 29 23:11:29.975877 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:29.975723 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/074dc860-7008-4c4f-9c9e-0fe9059b1947-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w6-b8f5c4-584577d98b-tpwp2\" (UID: \"074dc860-7008-4c4f-9c9e-0fe9059b1947\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-b8f5c4-584577d98b-tpwp2" Oct 29 23:11:29.987476 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:29.986983 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-b8f5c4-cdd8644c9-hpdp5" Oct 29 23:11:30.018613 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:30.018583 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4lzvk\" (UniqueName: \"kubernetes.io/projected/074dc860-7008-4c4f-9c9e-0fe9059b1947-kube-api-access-4lzvk\") pod \"payload-processing-e2e-shared-b-w6-b8f5c4-584577d98b-tpwp2\" (UID: \"074dc860-7008-4c4f-9c9e-0fe9059b1947\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-b8f5c4-584577d98b-tpwp2" Oct 29 23:11:30.082757 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:30.082513 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-b8f5c4-584577d98b-tpwp2" Oct 29 23:11:30.549051 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:30.546935 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-b8f5c4-cdd8644c9-hpdp5"] Oct 29 23:11:30.549051 ip-10-0-140-74 kubenswrapper[5057]: W1029 23:11:30.547897 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6c69267_32e0_4a27_a376_d9918d3f2e3b.slice/crio-0a3f9f5dc69ce624677091fb1efb08b8177790c929e103a31a326181e5590983 WatchSource:0}: Error finding container 0a3f9f5dc69ce624677091fb1efb08b8177790c929e103a31a326181e5590983: Status 404 returned error can't find the container with id 0a3f9f5dc69ce624677091fb1efb08b8177790c929e103a31a326181e5590983 Oct 29 23:11:30.550765 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:30.550736 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:11:30.579493 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:30.579403 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-b8f5c4-584577d98b-tpwp2"] Oct 29 23:11:31.074811 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:31.074771 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-b8f5c4-584577d98b-tpwp2" event={"ID":"074dc860-7008-4c4f-9c9e-0fe9059b1947","Type":"ContainerStarted","Data":"8dbf9fe6b03d54fa2705a50cb4b71157296dc4dd80f4aa33754a7d86334f3178"} Oct 29 23:11:31.075003 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:31.074851 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-b8f5c4-584577d98b-tpwp2" event={"ID":"074dc860-7008-4c4f-9c9e-0fe9059b1947","Type":"ContainerStarted","Data":"bfa48e61bc90a5cc416507c3042d79ce002e5792e16f7ccb3d807732da46825c"} Oct 29 23:11:31.075003 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:31.074887 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-b8f5c4-584577d98b-tpwp2" Oct 29 23:11:31.076542 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:31.076518 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-b8f5c4-cdd8644c9-hpdp5" event={"ID":"e6c69267-32e0-4a27-a376-d9918d3f2e3b","Type":"ContainerStarted","Data":"31e3629a311ce5d04d9e20fc37fe8042d0a167af1e267f1769c698ee9fc31567"} Oct 29 23:11:31.076656 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:31.076558 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-b8f5c4-cdd8644c9-hpdp5" event={"ID":"e6c69267-32e0-4a27-a376-d9918d3f2e3b","Type":"ContainerStarted","Data":"0a3f9f5dc69ce624677091fb1efb08b8177790c929e103a31a326181e5590983"} Oct 29 23:11:31.076656 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:31.076607 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-b8f5c4-cdd8644c9-hpdp5" Oct 29 23:11:31.118345 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:31.118277 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-b8f5c4-584577d98b-tpwp2" podStartSLOduration=2.118260274 podStartE2EDuration="2.118260274s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:11:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:11:31.11405092 +0000 UTC m=+2271.897264346" watchObservedRunningTime="2026-10-29 23:11:31.118260274 +0000 UTC m=+2271.901473701" Oct 29 23:11:31.156623 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:31.156577 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-b8f5c4-cdd8644c9-hpdp5" podStartSLOduration=2.156563807 podStartE2EDuration="2.156563807s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:11:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:11:31.152599209 +0000 UTC m=+2271.935812638" watchObservedRunningTime="2026-10-29 23:11:31.156563807 +0000 UTC m=+2271.939777233" Oct 29 23:11:31.378506 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:31.378418 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:11:31.491707 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:31.491671 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:11:31.719563 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:31.719526 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-9j6rn"] Oct 29 23:11:31.720371 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:31.720311 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-9j6rn" podUID="f521c155-eff0-4b0b-b4cd-e55abde4ed41" containerName="istio-proxy" containerID="cri-o://677b5bd71da6734023894e303e2563a729dc8609d8a2fd6a966cb4bc19e3c3db" gracePeriod=30 Oct 29 23:11:31.725572 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:31.725548 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-ngbdj"] Oct 29 23:11:31.725769 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:31.725745 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-ngbdj" podUID="7884e38e-4574-43f4-af11-f9414580568f" containerName="istio-proxy" containerID="cri-o://e7ed362e274d797012f28a969ceb923a7f7b13d1844d500edd7176907516d200" gracePeriod=30 Oct 29 23:11:31.729934 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:31.729909 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:11:32.386615 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:32.386572 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:11:32.494373 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:32.494324 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:11:32.878615 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:32.878580 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:11:33.078358 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:33.078060 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:11:33.924468 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:33.924428 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:11:34.321916 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:34.321877 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:11:35.983854 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:35.983818 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:11:36.498850 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:36.498815 5057 patch_prober.go:29] interesting pod/e2e-autores-a9ced3-openshift-default-7d8898bf57-ngbdj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 23:11:36.499025 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:36.498877 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-ngbdj" podUID="7884e38e-4574-43f4-af11-f9414580568f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 23:11:36.616367 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:36.616292 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:11:36.617034 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:36.617006 5057 patch_prober.go:29] interesting pod/e2e-autores-a9ced3-openshift-default-7d8898bf57-9j6rn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 23:11:36.617172 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:36.617065 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-9j6rn" podUID="f521c155-eff0-4b0b-b4cd-e55abde4ed41" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 23:11:36.725738 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:36.725704 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:11:37.056311 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:37.056279 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-ngbdj" Oct 29 23:11:37.096467 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:37.096439 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-9j6rn" Oct 29 23:11:37.099911 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:37.099886 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7884e38e-4574-43f4-af11-f9414580568f-istio-token\" (UniqueName: \"kubernetes.io/projected/7884e38e-4574-43f4-af11-f9414580568f-istio-token\") pod \"7884e38e-4574-43f4-af11-f9414580568f\" (UID: \"7884e38e-4574-43f4-af11-f9414580568f\") " Oct 29 23:11:37.100545 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:37.100512 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7884e38e-4574-43f4-af11-f9414580568f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7884e38e-4574-43f4-af11-f9414580568f-istio-envoy\") pod \"7884e38e-4574-43f4-af11-f9414580568f\" (UID: \"7884e38e-4574-43f4-af11-f9414580568f\") " Oct 29 23:11:37.100833 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:37.100807 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7884e38e-4574-43f4-af11-f9414580568f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7884e38e-4574-43f4-af11-f9414580568f-workload-certs\") pod \"7884e38e-4574-43f4-af11-f9414580568f\" (UID: \"7884e38e-4574-43f4-af11-f9414580568f\") " Oct 29 23:11:37.100942 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:37.100875 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7884e38e-4574-43f4-af11-f9414580568f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7884e38e-4574-43f4-af11-f9414580568f-workload-socket\") pod \"7884e38e-4574-43f4-af11-f9414580568f\" (UID: \"7884e38e-4574-43f4-af11-f9414580568f\") " Oct 29 23:11:37.100942 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:37.100919 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7884e38e-4574-43f4-af11-f9414580568f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7884e38e-4574-43f4-af11-f9414580568f-istio-podinfo\") pod \"7884e38e-4574-43f4-af11-f9414580568f\" (UID: \"7884e38e-4574-43f4-af11-f9414580568f\") " Oct 29 23:11:37.101039 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:37.100992 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7884e38e-4574-43f4-af11-f9414580568f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7884e38e-4574-43f4-af11-f9414580568f-istio-data\") pod \"7884e38e-4574-43f4-af11-f9414580568f\" (UID: \"7884e38e-4574-43f4-af11-f9414580568f\") " Oct 29 23:11:37.101129 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:37.101113 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7884e38e-4574-43f4-af11-f9414580568f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7884e38e-4574-43f4-af11-f9414580568f-credential-socket\") pod \"7884e38e-4574-43f4-af11-f9414580568f\" (UID: \"7884e38e-4574-43f4-af11-f9414580568f\") " Oct 29 23:11:37.101194 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:37.101178 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7884e38e-4574-43f4-af11-f9414580568f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7884e38e-4574-43f4-af11-f9414580568f-istiod-ca-cert\") pod \"7884e38e-4574-43f4-af11-f9414580568f\" (UID: \"7884e38e-4574-43f4-af11-f9414580568f\") " Oct 29 23:11:37.101281 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:37.101265 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7884e38e-4574-43f4-af11-f9414580568f-kube-api-access-9rtfb\" (UniqueName: \"kubernetes.io/projected/7884e38e-4574-43f4-af11-f9414580568f-kube-api-access-9rtfb\") pod \"7884e38e-4574-43f4-af11-f9414580568f\" (UID: \"7884e38e-4574-43f4-af11-f9414580568f\") " Oct 29 23:11:37.103109 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:37.102960 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7884e38e-4574-43f4-af11-f9414580568f-istio-data" pod "7884e38e-4574-43f4-af11-f9414580568f" (UID: "7884e38e-4574-43f4-af11-f9414580568f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:11:37.103109 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:37.102990 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7884e38e-4574-43f4-af11-f9414580568f-workload-socket" pod "7884e38e-4574-43f4-af11-f9414580568f" (UID: "7884e38e-4574-43f4-af11-f9414580568f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:11:37.103109 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:37.102886 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7884e38e-4574-43f4-af11-f9414580568f-credential-socket" pod "7884e38e-4574-43f4-af11-f9414580568f" (UID: "7884e38e-4574-43f4-af11-f9414580568f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:11:37.103309 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:37.103275 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7884e38e-4574-43f4-af11-f9414580568f-istiod-ca-cert" pod "7884e38e-4574-43f4-af11-f9414580568f" (UID: "7884e38e-4574-43f4-af11-f9414580568f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:11:37.103393 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:37.103354 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7884e38e-4574-43f4-af11-f9414580568f-workload-certs" pod "7884e38e-4574-43f4-af11-f9414580568f" (UID: "7884e38e-4574-43f4-af11-f9414580568f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:11:37.106070 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:37.105746 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7884e38e-4574-43f4-af11-f9414580568f-istio-token" pod "7884e38e-4574-43f4-af11-f9414580568f" (UID: "7884e38e-4574-43f4-af11-f9414580568f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:11:37.106207 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:37.106173 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7884e38e-4574-43f4-af11-f9414580568f-istio-podinfo" pod "7884e38e-4574-43f4-af11-f9414580568f" (UID: "7884e38e-4574-43f4-af11-f9414580568f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:11:37.108638 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:37.108612 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7884e38e-4574-43f4-af11-f9414580568f-kube-api-access-9rtfb" pod "7884e38e-4574-43f4-af11-f9414580568f" (UID: "7884e38e-4574-43f4-af11-f9414580568f"). InnerVolumeSpecName "kube-api-access-9rtfb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:11:37.109955 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:37.109932 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7884e38e-4574-43f4-af11-f9414580568f-istio-envoy" pod "7884e38e-4574-43f4-af11-f9414580568f" (UID: "7884e38e-4574-43f4-af11-f9414580568f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:11:37.171901 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:37.171866 5057 generic.go:356] "Generic (PLEG): container finished" podID="f521c155-eff0-4b0b-b4cd-e55abde4ed41" containerID="677b5bd71da6734023894e303e2563a729dc8609d8a2fd6a966cb4bc19e3c3db" exitCode=0 Oct 29 23:11:37.172096 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:37.171956 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-9j6rn" event={"ID":"f521c155-eff0-4b0b-b4cd-e55abde4ed41","Type":"ContainerDied","Data":"677b5bd71da6734023894e303e2563a729dc8609d8a2fd6a966cb4bc19e3c3db"} Oct 29 23:11:37.172096 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:37.171993 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-9j6rn" event={"ID":"f521c155-eff0-4b0b-b4cd-e55abde4ed41","Type":"ContainerDied","Data":"4bfb4774928800727c634a9ae215db57ae3f072633424f4ae11b5c5abfd25ddd"} Oct 29 23:11:37.172096 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:37.172013 5057 scope.go:122] "RemoveContainer" containerID="677b5bd71da6734023894e303e2563a729dc8609d8a2fd6a966cb4bc19e3c3db" Oct 29 23:11:37.172285 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:37.172124 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-9j6rn" Oct 29 23:11:37.178349 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:37.178301 5057 generic.go:356] "Generic (PLEG): container finished" podID="7884e38e-4574-43f4-af11-f9414580568f" containerID="e7ed362e274d797012f28a969ceb923a7f7b13d1844d500edd7176907516d200" exitCode=0 Oct 29 23:11:37.178648 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:37.178623 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-ngbdj" event={"ID":"7884e38e-4574-43f4-af11-f9414580568f","Type":"ContainerDied","Data":"e7ed362e274d797012f28a969ceb923a7f7b13d1844d500edd7176907516d200"} Oct 29 23:11:37.178755 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:37.178684 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-ngbdj" event={"ID":"7884e38e-4574-43f4-af11-f9414580568f","Type":"ContainerDied","Data":"393719fe756016c9321bfda1292355d71145402249a0fd78c789c4c39c81048a"} Oct 29 23:11:37.178820 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:37.178768 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-ngbdj" Oct 29 23:11:37.203549 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:37.203232 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f521c155-eff0-4b0b-b4cd-e55abde4ed41-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f521c155-eff0-4b0b-b4cd-e55abde4ed41-istio-envoy\") pod \"f521c155-eff0-4b0b-b4cd-e55abde4ed41\" (UID: \"f521c155-eff0-4b0b-b4cd-e55abde4ed41\") " Oct 29 23:11:37.203549 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:37.203289 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f521c155-eff0-4b0b-b4cd-e55abde4ed41-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f521c155-eff0-4b0b-b4cd-e55abde4ed41-workload-certs\") pod \"f521c155-eff0-4b0b-b4cd-e55abde4ed41\" (UID: \"f521c155-eff0-4b0b-b4cd-e55abde4ed41\") " Oct 29 23:11:37.203549 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:37.203439 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f521c155-eff0-4b0b-b4cd-e55abde4ed41-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f521c155-eff0-4b0b-b4cd-e55abde4ed41-istiod-ca-cert\") pod \"f521c155-eff0-4b0b-b4cd-e55abde4ed41\" (UID: \"f521c155-eff0-4b0b-b4cd-e55abde4ed41\") " Oct 29 23:11:37.203758 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:37.203564 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f521c155-eff0-4b0b-b4cd-e55abde4ed41-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f521c155-eff0-4b0b-b4cd-e55abde4ed41-istio-podinfo\") pod \"f521c155-eff0-4b0b-b4cd-e55abde4ed41\" (UID: \"f521c155-eff0-4b0b-b4cd-e55abde4ed41\") " Oct 29 23:11:37.203758 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:37.203596 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f521c155-eff0-4b0b-b4cd-e55abde4ed41-kube-api-access-gcwbw\" (UniqueName: \"kubernetes.io/projected/f521c155-eff0-4b0b-b4cd-e55abde4ed41-kube-api-access-gcwbw\") pod \"f521c155-eff0-4b0b-b4cd-e55abde4ed41\" (UID: \"f521c155-eff0-4b0b-b4cd-e55abde4ed41\") " Oct 29 23:11:37.203758 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:37.203623 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f521c155-eff0-4b0b-b4cd-e55abde4ed41-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f521c155-eff0-4b0b-b4cd-e55abde4ed41-credential-socket\") pod \"f521c155-eff0-4b0b-b4cd-e55abde4ed41\" (UID: \"f521c155-eff0-4b0b-b4cd-e55abde4ed41\") " Oct 29 23:11:37.203758 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:37.203685 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f521c155-eff0-4b0b-b4cd-e55abde4ed41-istio-token\" (UniqueName: \"kubernetes.io/projected/f521c155-eff0-4b0b-b4cd-e55abde4ed41-istio-token\") pod \"f521c155-eff0-4b0b-b4cd-e55abde4ed41\" (UID: \"f521c155-eff0-4b0b-b4cd-e55abde4ed41\") " Oct 29 23:11:37.203758 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:37.203739 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f521c155-eff0-4b0b-b4cd-e55abde4ed41-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f521c155-eff0-4b0b-b4cd-e55abde4ed41-workload-socket\") pod \"f521c155-eff0-4b0b-b4cd-e55abde4ed41\" (UID: \"f521c155-eff0-4b0b-b4cd-e55abde4ed41\") " Oct 29 23:11:37.203758 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:37.203760 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f521c155-eff0-4b0b-b4cd-e55abde4ed41-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f521c155-eff0-4b0b-b4cd-e55abde4ed41-istio-data\") pod \"f521c155-eff0-4b0b-b4cd-e55abde4ed41\" (UID: \"f521c155-eff0-4b0b-b4cd-e55abde4ed41\") " Oct 29 23:11:37.204285 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:37.204266 5057 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7884e38e-4574-43f4-af11-f9414580568f-istio-token\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:11:37.204285 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:37.204285 5057 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7884e38e-4574-43f4-af11-f9414580568f-istio-envoy\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:11:37.204829 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:37.204294 5057 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7884e38e-4574-43f4-af11-f9414580568f-workload-certs\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:11:37.204829 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:37.204303 5057 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7884e38e-4574-43f4-af11-f9414580568f-workload-socket\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:11:37.204829 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:37.204312 5057 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7884e38e-4574-43f4-af11-f9414580568f-istio-podinfo\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:11:37.204829 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:37.204321 5057 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7884e38e-4574-43f4-af11-f9414580568f-istio-data\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:11:37.204829 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:37.204541 5057 scope.go:122] "RemoveContainer" containerID="677b5bd71da6734023894e303e2563a729dc8609d8a2fd6a966cb4bc19e3c3db" Oct 29 23:11:37.205092 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:37.204924 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f521c155-eff0-4b0b-b4cd-e55abde4ed41-credential-socket" pod "f521c155-eff0-4b0b-b4cd-e55abde4ed41" (UID: "f521c155-eff0-4b0b-b4cd-e55abde4ed41"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:11:37.205159 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:37.205120 5057 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7884e38e-4574-43f4-af11-f9414580568f-credential-socket\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:11:37.205159 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:37.205146 5057 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7884e38e-4574-43f4-af11-f9414580568f-istiod-ca-cert\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:11:37.205269 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:37.205165 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9rtfb\" (UniqueName: \"kubernetes.io/projected/7884e38e-4574-43f4-af11-f9414580568f-kube-api-access-9rtfb\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:11:37.205403 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:37.205380 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f521c155-eff0-4b0b-b4cd-e55abde4ed41-istio-data" pod "f521c155-eff0-4b0b-b4cd-e55abde4ed41" (UID: "f521c155-eff0-4b0b-b4cd-e55abde4ed41"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:11:37.205510 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:37.205478 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f521c155-eff0-4b0b-b4cd-e55abde4ed41-workload-certs" pod "f521c155-eff0-4b0b-b4cd-e55abde4ed41" (UID: "f521c155-eff0-4b0b-b4cd-e55abde4ed41"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:11:37.205645 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:11:37.205608 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"677b5bd71da6734023894e303e2563a729dc8609d8a2fd6a966cb4bc19e3c3db\": container with ID starting with 677b5bd71da6734023894e303e2563a729dc8609d8a2fd6a966cb4bc19e3c3db not found: ID does not exist" containerID="677b5bd71da6734023894e303e2563a729dc8609d8a2fd6a966cb4bc19e3c3db" Oct 29 23:11:37.205704 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:37.205657 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"677b5bd71da6734023894e303e2563a729dc8609d8a2fd6a966cb4bc19e3c3db"} err="failed to get container status \"677b5bd71da6734023894e303e2563a729dc8609d8a2fd6a966cb4bc19e3c3db\": rpc error: code = NotFound desc = could not find container \"677b5bd71da6734023894e303e2563a729dc8609d8a2fd6a966cb4bc19e3c3db\": container with ID starting with 677b5bd71da6734023894e303e2563a729dc8609d8a2fd6a966cb4bc19e3c3db not found: ID does not exist" Oct 29 23:11:37.205704 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:37.205692 5057 scope.go:122] "RemoveContainer" containerID="e7ed362e274d797012f28a969ceb923a7f7b13d1844d500edd7176907516d200" Oct 29 23:11:37.206064 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:37.206045 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f521c155-eff0-4b0b-b4cd-e55abde4ed41-istiod-ca-cert" pod "f521c155-eff0-4b0b-b4cd-e55abde4ed41" (UID: "f521c155-eff0-4b0b-b4cd-e55abde4ed41"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:11:37.206643 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:37.206615 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f521c155-eff0-4b0b-b4cd-e55abde4ed41-workload-socket" pod "f521c155-eff0-4b0b-b4cd-e55abde4ed41" (UID: "f521c155-eff0-4b0b-b4cd-e55abde4ed41"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:11:37.209840 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:37.209814 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f521c155-eff0-4b0b-b4cd-e55abde4ed41-istio-podinfo" pod "f521c155-eff0-4b0b-b4cd-e55abde4ed41" (UID: "f521c155-eff0-4b0b-b4cd-e55abde4ed41"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:11:37.210062 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:37.210041 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f521c155-eff0-4b0b-b4cd-e55abde4ed41-kube-api-access-gcwbw" pod "f521c155-eff0-4b0b-b4cd-e55abde4ed41" (UID: "f521c155-eff0-4b0b-b4cd-e55abde4ed41"). InnerVolumeSpecName "kube-api-access-gcwbw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:11:37.210689 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:37.210667 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f521c155-eff0-4b0b-b4cd-e55abde4ed41-istio-envoy" pod "f521c155-eff0-4b0b-b4cd-e55abde4ed41" (UID: "f521c155-eff0-4b0b-b4cd-e55abde4ed41"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:11:37.222146 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:37.222125 5057 scope.go:122] "RemoveContainer" containerID="e7ed362e274d797012f28a969ceb923a7f7b13d1844d500edd7176907516d200" Oct 29 23:11:37.223344 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:11:37.223310 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7ed362e274d797012f28a969ceb923a7f7b13d1844d500edd7176907516d200\": container with ID starting with e7ed362e274d797012f28a969ceb923a7f7b13d1844d500edd7176907516d200 not found: ID does not exist" containerID="e7ed362e274d797012f28a969ceb923a7f7b13d1844d500edd7176907516d200" Oct 29 23:11:37.223650 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:37.223622 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7ed362e274d797012f28a969ceb923a7f7b13d1844d500edd7176907516d200"} err="failed to get container status \"e7ed362e274d797012f28a969ceb923a7f7b13d1844d500edd7176907516d200\": rpc error: code = NotFound desc = could not find container \"e7ed362e274d797012f28a969ceb923a7f7b13d1844d500edd7176907516d200\": container with ID starting with e7ed362e274d797012f28a969ceb923a7f7b13d1844d500edd7176907516d200 not found: ID does not exist" Oct 29 23:11:37.224087 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:37.224053 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f521c155-eff0-4b0b-b4cd-e55abde4ed41-istio-token" pod "f521c155-eff0-4b0b-b4cd-e55abde4ed41" (UID: "f521c155-eff0-4b0b-b4cd-e55abde4ed41"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:11:37.269351 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:37.262321 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-ngbdj"] Oct 29 23:11:37.283571 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:37.269718 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-ngbdj"] Oct 29 23:11:37.325156 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:37.313996 5057 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f521c155-eff0-4b0b-b4cd-e55abde4ed41-istio-podinfo\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:11:37.325156 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:37.314033 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gcwbw\" (UniqueName: \"kubernetes.io/projected/f521c155-eff0-4b0b-b4cd-e55abde4ed41-kube-api-access-gcwbw\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:11:37.325156 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:37.314051 5057 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f521c155-eff0-4b0b-b4cd-e55abde4ed41-credential-socket\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:11:37.325156 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:37.314067 5057 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f521c155-eff0-4b0b-b4cd-e55abde4ed41-istio-token\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:11:37.325156 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:37.314081 5057 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f521c155-eff0-4b0b-b4cd-e55abde4ed41-workload-socket\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:11:37.325156 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:37.314093 5057 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f521c155-eff0-4b0b-b4cd-e55abde4ed41-istio-data\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:11:37.325156 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:37.314105 5057 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f521c155-eff0-4b0b-b4cd-e55abde4ed41-istio-envoy\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:11:37.325156 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:37.314118 5057 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f521c155-eff0-4b0b-b4cd-e55abde4ed41-workload-certs\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:11:37.325156 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:37.314130 5057 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f521c155-eff0-4b0b-b4cd-e55abde4ed41-istiod-ca-cert\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:11:37.356125 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:37.355958 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7884e38e-4574-43f4-af11-f9414580568f" path="/var/lib/kubelet/pods/7884e38e-4574-43f4-af11-f9414580568f/volumes" Oct 29 23:11:37.357040 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:37.357018 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v2tfm"] Oct 29 23:11:37.506746 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:37.505886 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w4-af237d/auth-test-model-af237d-kserve-7444475bdf-5tw7n"] Oct 29 23:11:37.508903 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:37.508867 5057 cpu_manager.go:414] "containerMap: removing container" podUID="7884e38e-4574-43f4-af11-f9414580568f" containerName="istio-proxy" Oct 29 23:11:37.508903 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:37.508897 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7884e38e-4574-43f4-af11-f9414580568f" containerName="istio-proxy" Oct 29 23:11:37.509113 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:37.508997 5057 cpu_manager.go:414] "containerMap: removing container" podUID="f521c155-eff0-4b0b-b4cd-e55abde4ed41" containerName="istio-proxy" Oct 29 23:11:37.509113 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:37.509008 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f521c155-eff0-4b0b-b4cd-e55abde4ed41" containerName="istio-proxy" Oct 29 23:11:37.509277 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:37.509253 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="f521c155-eff0-4b0b-b4cd-e55abde4ed41" containerName="istio-proxy" Oct 29 23:11:37.509412 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:37.509315 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="7884e38e-4574-43f4-af11-f9414580568f" containerName="istio-proxy" Oct 29 23:11:37.564435 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:37.562476 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:11:37.570977 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:37.569570 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-9j6rn"] Oct 29 23:11:37.570977 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:37.569685 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-af237d/auth-test-model-af237d-kserve-7444475bdf-5tw7n" Oct 29 23:11:37.589359 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:37.582080 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-af237d/auth-test-model-af237d-kserve-7444475bdf-5tw7n"] Oct 29 23:11:37.589359 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:37.583423 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-9j6rn"] Oct 29 23:11:37.602414 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:37.599042 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-af237d\"/\"auth-test-model-af237d-kserve-self-signed-certs\"" Oct 29 23:11:37.602414 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:37.599210 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-af237d\"/\"default-dockercfg-smz99\"" Oct 29 23:11:37.602414 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:37.599890 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-af237d\"/\"openshift-service-ca.crt\"" Oct 29 23:11:37.602414 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:37.599964 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-af237d\"/\"kube-root-ca.crt\"" Oct 29 23:11:37.726471 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:37.726437 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/813724c2-17b1-4473-9c1a-3ee7d092ecd5-tls-certs\") pod \"auth-test-model-af237d-kserve-7444475bdf-5tw7n\" (UID: \"813724c2-17b1-4473-9c1a-3ee7d092ecd5\") " pod="ai-tenant-e2e-shared-a-w4-af237d/auth-test-model-af237d-kserve-7444475bdf-5tw7n" Oct 29 23:11:37.726676 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:37.726482 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/813724c2-17b1-4473-9c1a-3ee7d092ecd5-tmp-dir\") pod \"auth-test-model-af237d-kserve-7444475bdf-5tw7n\" (UID: \"813724c2-17b1-4473-9c1a-3ee7d092ecd5\") " pod="ai-tenant-e2e-shared-a-w4-af237d/auth-test-model-af237d-kserve-7444475bdf-5tw7n" Oct 29 23:11:37.726676 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:37.726622 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t52v9\" (UniqueName: \"kubernetes.io/projected/813724c2-17b1-4473-9c1a-3ee7d092ecd5-kube-api-access-t52v9\") pod \"auth-test-model-af237d-kserve-7444475bdf-5tw7n\" (UID: \"813724c2-17b1-4473-9c1a-3ee7d092ecd5\") " pod="ai-tenant-e2e-shared-a-w4-af237d/auth-test-model-af237d-kserve-7444475bdf-5tw7n" Oct 29 23:11:37.726873 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:37.726855 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/813724c2-17b1-4473-9c1a-3ee7d092ecd5-dshm\") pod \"auth-test-model-af237d-kserve-7444475bdf-5tw7n\" (UID: \"813724c2-17b1-4473-9c1a-3ee7d092ecd5\") " pod="ai-tenant-e2e-shared-a-w4-af237d/auth-test-model-af237d-kserve-7444475bdf-5tw7n" Oct 29 23:11:37.726935 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:37.726883 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/813724c2-17b1-4473-9c1a-3ee7d092ecd5-home\") pod \"auth-test-model-af237d-kserve-7444475bdf-5tw7n\" (UID: \"813724c2-17b1-4473-9c1a-3ee7d092ecd5\") " pod="ai-tenant-e2e-shared-a-w4-af237d/auth-test-model-af237d-kserve-7444475bdf-5tw7n" Oct 29 23:11:37.726935 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:37.726908 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/813724c2-17b1-4473-9c1a-3ee7d092ecd5-model-cache\") pod \"auth-test-model-af237d-kserve-7444475bdf-5tw7n\" (UID: \"813724c2-17b1-4473-9c1a-3ee7d092ecd5\") " pod="ai-tenant-e2e-shared-a-w4-af237d/auth-test-model-af237d-kserve-7444475bdf-5tw7n" Oct 29 23:11:37.834441 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:37.830699 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/813724c2-17b1-4473-9c1a-3ee7d092ecd5-dshm\") pod \"auth-test-model-af237d-kserve-7444475bdf-5tw7n\" (UID: \"813724c2-17b1-4473-9c1a-3ee7d092ecd5\") " pod="ai-tenant-e2e-shared-a-w4-af237d/auth-test-model-af237d-kserve-7444475bdf-5tw7n" Oct 29 23:11:37.834441 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:37.830755 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/813724c2-17b1-4473-9c1a-3ee7d092ecd5-home\") pod \"auth-test-model-af237d-kserve-7444475bdf-5tw7n\" (UID: \"813724c2-17b1-4473-9c1a-3ee7d092ecd5\") " pod="ai-tenant-e2e-shared-a-w4-af237d/auth-test-model-af237d-kserve-7444475bdf-5tw7n" Oct 29 23:11:37.834441 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:37.830793 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/813724c2-17b1-4473-9c1a-3ee7d092ecd5-model-cache\") pod \"auth-test-model-af237d-kserve-7444475bdf-5tw7n\" (UID: \"813724c2-17b1-4473-9c1a-3ee7d092ecd5\") " pod="ai-tenant-e2e-shared-a-w4-af237d/auth-test-model-af237d-kserve-7444475bdf-5tw7n" Oct 29 23:11:37.834441 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:37.830953 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/813724c2-17b1-4473-9c1a-3ee7d092ecd5-tls-certs\") pod \"auth-test-model-af237d-kserve-7444475bdf-5tw7n\" (UID: \"813724c2-17b1-4473-9c1a-3ee7d092ecd5\") " pod="ai-tenant-e2e-shared-a-w4-af237d/auth-test-model-af237d-kserve-7444475bdf-5tw7n" Oct 29 23:11:37.834441 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:37.830982 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/813724c2-17b1-4473-9c1a-3ee7d092ecd5-tmp-dir\") pod \"auth-test-model-af237d-kserve-7444475bdf-5tw7n\" (UID: \"813724c2-17b1-4473-9c1a-3ee7d092ecd5\") " pod="ai-tenant-e2e-shared-a-w4-af237d/auth-test-model-af237d-kserve-7444475bdf-5tw7n" Oct 29 23:11:37.834441 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:37.831062 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t52v9\" (UniqueName: \"kubernetes.io/projected/813724c2-17b1-4473-9c1a-3ee7d092ecd5-kube-api-access-t52v9\") pod \"auth-test-model-af237d-kserve-7444475bdf-5tw7n\" (UID: \"813724c2-17b1-4473-9c1a-3ee7d092ecd5\") " pod="ai-tenant-e2e-shared-a-w4-af237d/auth-test-model-af237d-kserve-7444475bdf-5tw7n" Oct 29 23:11:37.834441 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:37.832592 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/813724c2-17b1-4473-9c1a-3ee7d092ecd5-model-cache\") pod \"auth-test-model-af237d-kserve-7444475bdf-5tw7n\" (UID: \"813724c2-17b1-4473-9c1a-3ee7d092ecd5\") " pod="ai-tenant-e2e-shared-a-w4-af237d/auth-test-model-af237d-kserve-7444475bdf-5tw7n" Oct 29 23:11:37.834441 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:37.832857 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/813724c2-17b1-4473-9c1a-3ee7d092ecd5-home\") pod \"auth-test-model-af237d-kserve-7444475bdf-5tw7n\" (UID: \"813724c2-17b1-4473-9c1a-3ee7d092ecd5\") " pod="ai-tenant-e2e-shared-a-w4-af237d/auth-test-model-af237d-kserve-7444475bdf-5tw7n" Oct 29 23:11:37.834441 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:37.833570 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/813724c2-17b1-4473-9c1a-3ee7d092ecd5-tmp-dir\") pod \"auth-test-model-af237d-kserve-7444475bdf-5tw7n\" (UID: \"813724c2-17b1-4473-9c1a-3ee7d092ecd5\") " pod="ai-tenant-e2e-shared-a-w4-af237d/auth-test-model-af237d-kserve-7444475bdf-5tw7n" Oct 29 23:11:37.842353 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:37.841182 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/813724c2-17b1-4473-9c1a-3ee7d092ecd5-dshm\") pod \"auth-test-model-af237d-kserve-7444475bdf-5tw7n\" (UID: \"813724c2-17b1-4473-9c1a-3ee7d092ecd5\") " pod="ai-tenant-e2e-shared-a-w4-af237d/auth-test-model-af237d-kserve-7444475bdf-5tw7n" Oct 29 23:11:37.848348 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:37.846224 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/813724c2-17b1-4473-9c1a-3ee7d092ecd5-tls-certs\") pod \"auth-test-model-af237d-kserve-7444475bdf-5tw7n\" (UID: \"813724c2-17b1-4473-9c1a-3ee7d092ecd5\") " pod="ai-tenant-e2e-shared-a-w4-af237d/auth-test-model-af237d-kserve-7444475bdf-5tw7n" Oct 29 23:11:37.870306 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:37.870280 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t52v9\" (UniqueName: \"kubernetes.io/projected/813724c2-17b1-4473-9c1a-3ee7d092ecd5-kube-api-access-t52v9\") pod \"auth-test-model-af237d-kserve-7444475bdf-5tw7n\" (UID: \"813724c2-17b1-4473-9c1a-3ee7d092ecd5\") " pod="ai-tenant-e2e-shared-a-w4-af237d/auth-test-model-af237d-kserve-7444475bdf-5tw7n" Oct 29 23:11:37.913511 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:37.913472 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-af237d/auth-test-model-af237d-kserve-7444475bdf-5tw7n" Oct 29 23:11:38.272380 ip-10-0-140-74 kubenswrapper[5057]: W1029 23:11:38.272298 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod813724c2_17b1_4473_9c1a_3ee7d092ecd5.slice/crio-cd8354326439486c501664523fc327b9f8628788ba56f0fe834698624893a98b WatchSource:0}: Error finding container cd8354326439486c501664523fc327b9f8628788ba56f0fe834698624893a98b: Status 404 returned error can't find the container with id cd8354326439486c501664523fc327b9f8628788ba56f0fe834698624893a98b Oct 29 23:11:38.274511 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:38.274146 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-af237d/auth-test-model-af237d-kserve-7444475bdf-5tw7n"] Oct 29 23:11:38.550222 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:38.550137 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:11:39.208200 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:39.208160 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-af237d/auth-test-model-af237d-kserve-7444475bdf-5tw7n" event={"ID":"813724c2-17b1-4473-9c1a-3ee7d092ecd5","Type":"ContainerStarted","Data":"e974f95ebb4ff0cc0f75186e1477d7ca31a6d6f2e6f8dfc03025c6671b4a3544"} Oct 29 23:11:39.208486 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:39.208216 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-af237d/auth-test-model-af237d-kserve-7444475bdf-5tw7n" event={"ID":"813724c2-17b1-4473-9c1a-3ee7d092ecd5","Type":"ContainerStarted","Data":"cd8354326439486c501664523fc327b9f8628788ba56f0fe834698624893a98b"} Oct 29 23:11:39.253840 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:39.253791 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w4-af237d/auth-test-model-af237d-kserve-7444475bdf-5tw7n" podStartSLOduration=2.253777466 podStartE2EDuration="2.253777466s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:11:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:11:39.251202957 +0000 UTC m=+2280.034416396" watchObservedRunningTime="2026-10-29 23:11:39.253777466 +0000 UTC m=+2280.036990891" Oct 29 23:11:39.332753 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:39.332719 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f521c155-eff0-4b0b-b4cd-e55abde4ed41" path="/var/lib/kubelet/pods/f521c155-eff0-4b0b-b4cd-e55abde4ed41/volumes" Oct 29 23:11:39.561421 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:39.561278 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:11:39.682000 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:39.681965 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:11:39.737670 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:39.737635 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-be0401-79b7645f8c-xjktr"] Oct 29 23:11:39.743622 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:39.743598 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-be0401-79b7645f8c-xjktr" Oct 29 23:11:39.758467 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:39.758440 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-be0401\"" Oct 29 23:11:39.760269 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:39.760242 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-be0401-79b7645f8c-xjktr"] Oct 29 23:11:39.765811 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:39.765788 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-be0401-dockercfg-pmjm7\"" Oct 29 23:11:39.862411 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:39.857785 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-be0401-6fc87448b6-vxln4"] Oct 29 23:11:39.905574 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:39.905533 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-be0401-6fc87448b6-vxln4"] Oct 29 23:11:39.905797 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:39.905677 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-be0401-6fc87448b6-vxln4" Oct 29 23:11:39.922282 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:39.920583 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w4-af237d/auth-test-model-af237d-kserve-7444475bdf-5tw7n" Oct 29 23:11:39.922282 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:39.921822 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/832a7cbc-eb85-4516-92f5-aa93400f3618-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-be0401-79b7645f8c-xjktr\" (UID: \"832a7cbc-eb85-4516-92f5-aa93400f3618\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-be0401-79b7645f8c-xjktr" Oct 29 23:11:39.922282 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:39.922082 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wv88m\" (UniqueName: \"kubernetes.io/projected/832a7cbc-eb85-4516-92f5-aa93400f3618-kube-api-access-wv88m\") pod \"payload-pre-processing-e2e-explicit-be0401-79b7645f8c-xjktr\" (UID: \"832a7cbc-eb85-4516-92f5-aa93400f3618\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-be0401-79b7645f8c-xjktr" Oct 29 23:11:40.024022 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:40.023985 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wv88m\" (UniqueName: \"kubernetes.io/projected/832a7cbc-eb85-4516-92f5-aa93400f3618-kube-api-access-wv88m\") pod \"payload-pre-processing-e2e-explicit-be0401-79b7645f8c-xjktr\" (UID: \"832a7cbc-eb85-4516-92f5-aa93400f3618\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-be0401-79b7645f8c-xjktr" Oct 29 23:11:40.024234 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:40.024123 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3beaf203-5283-45eb-bac3-c094e6cf8212-plugins-config-volume\") pod \"payload-processing-e2e-explicit-be0401-6fc87448b6-vxln4\" (UID: \"3beaf203-5283-45eb-bac3-c094e6cf8212\") " pod="openshift-ingress/payload-processing-e2e-explicit-be0401-6fc87448b6-vxln4" Oct 29 23:11:40.024344 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:40.024313 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/832a7cbc-eb85-4516-92f5-aa93400f3618-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-be0401-79b7645f8c-xjktr\" (UID: \"832a7cbc-eb85-4516-92f5-aa93400f3618\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-be0401-79b7645f8c-xjktr" Oct 29 23:11:40.024504 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:40.024486 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8zxl\" (UniqueName: \"kubernetes.io/projected/3beaf203-5283-45eb-bac3-c094e6cf8212-kube-api-access-x8zxl\") pod \"payload-processing-e2e-explicit-be0401-6fc87448b6-vxln4\" (UID: \"3beaf203-5283-45eb-bac3-c094e6cf8212\") " pod="openshift-ingress/payload-processing-e2e-explicit-be0401-6fc87448b6-vxln4" Oct 29 23:11:40.025221 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:40.025197 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/832a7cbc-eb85-4516-92f5-aa93400f3618-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-be0401-79b7645f8c-xjktr\" (UID: \"832a7cbc-eb85-4516-92f5-aa93400f3618\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-be0401-79b7645f8c-xjktr" Oct 29 23:11:40.047797 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:40.047766 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wv88m\" (UniqueName: \"kubernetes.io/projected/832a7cbc-eb85-4516-92f5-aa93400f3618-kube-api-access-wv88m\") pod \"payload-pre-processing-e2e-explicit-be0401-79b7645f8c-xjktr\" (UID: \"832a7cbc-eb85-4516-92f5-aa93400f3618\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-be0401-79b7645f8c-xjktr" Oct 29 23:11:40.062322 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:40.062294 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-be0401-79b7645f8c-xjktr" Oct 29 23:11:40.128814 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:40.127728 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8zxl\" (UniqueName: \"kubernetes.io/projected/3beaf203-5283-45eb-bac3-c094e6cf8212-kube-api-access-x8zxl\") pod \"payload-processing-e2e-explicit-be0401-6fc87448b6-vxln4\" (UID: \"3beaf203-5283-45eb-bac3-c094e6cf8212\") " pod="openshift-ingress/payload-processing-e2e-explicit-be0401-6fc87448b6-vxln4" Oct 29 23:11:40.128814 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:40.127940 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3beaf203-5283-45eb-bac3-c094e6cf8212-plugins-config-volume\") pod \"payload-processing-e2e-explicit-be0401-6fc87448b6-vxln4\" (UID: \"3beaf203-5283-45eb-bac3-c094e6cf8212\") " pod="openshift-ingress/payload-processing-e2e-explicit-be0401-6fc87448b6-vxln4" Oct 29 23:11:40.128814 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:40.128681 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3beaf203-5283-45eb-bac3-c094e6cf8212-plugins-config-volume\") pod \"payload-processing-e2e-explicit-be0401-6fc87448b6-vxln4\" (UID: \"3beaf203-5283-45eb-bac3-c094e6cf8212\") " pod="openshift-ingress/payload-processing-e2e-explicit-be0401-6fc87448b6-vxln4" Oct 29 23:11:40.151837 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:40.151794 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8zxl\" (UniqueName: \"kubernetes.io/projected/3beaf203-5283-45eb-bac3-c094e6cf8212-kube-api-access-x8zxl\") pod \"payload-processing-e2e-explicit-be0401-6fc87448b6-vxln4\" (UID: \"3beaf203-5283-45eb-bac3-c094e6cf8212\") " pod="openshift-ingress/payload-processing-e2e-explicit-be0401-6fc87448b6-vxln4" Oct 29 23:11:40.262973 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:40.262942 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-be0401-6fc87448b6-vxln4" Oct 29 23:11:40.406226 ip-10-0-140-74 kubenswrapper[5057]: W1029 23:11:40.406004 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod832a7cbc_eb85_4516_92f5_aa93400f3618.slice/crio-6efae9e54cca62e99a06c5bb4e087c7f5379d4694dc696decaca1d8842d2afc8 WatchSource:0}: Error finding container 6efae9e54cca62e99a06c5bb4e087c7f5379d4694dc696decaca1d8842d2afc8: Status 404 returned error can't find the container with id 6efae9e54cca62e99a06c5bb4e087c7f5379d4694dc696decaca1d8842d2afc8 Oct 29 23:11:40.429293 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:40.429265 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-be0401-79b7645f8c-xjktr"] Oct 29 23:11:40.620009 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:40.619985 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-be0401-6fc87448b6-vxln4"] Oct 29 23:11:40.621402 ip-10-0-140-74 kubenswrapper[5057]: W1029 23:11:40.621369 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3beaf203_5283_45eb_bac3_c094e6cf8212.slice/crio-6e0be85a97fd8b01dfb5c3ffaa823f6c2b8663a2278d60691101ed88e00f41a2 WatchSource:0}: Error finding container 6e0be85a97fd8b01dfb5c3ffaa823f6c2b8663a2278d60691101ed88e00f41a2: Status 404 returned error can't find the container with id 6e0be85a97fd8b01dfb5c3ffaa823f6c2b8663a2278d60691101ed88e00f41a2 Oct 29 23:11:40.755848 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:40.755800 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:11:41.238121 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:41.238081 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-be0401-79b7645f8c-xjktr" event={"ID":"832a7cbc-eb85-4516-92f5-aa93400f3618","Type":"ContainerStarted","Data":"24a12ac377b8c97646d53e585ae796bbd85ff23428c199ae66e10675142c9907"} Oct 29 23:11:41.238325 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:41.238147 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-be0401-79b7645f8c-xjktr" event={"ID":"832a7cbc-eb85-4516-92f5-aa93400f3618","Type":"ContainerStarted","Data":"6efae9e54cca62e99a06c5bb4e087c7f5379d4694dc696decaca1d8842d2afc8"} Oct 29 23:11:41.238325 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:41.238176 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-be0401-79b7645f8c-xjktr" Oct 29 23:11:41.240122 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:41.240090 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-be0401-6fc87448b6-vxln4" event={"ID":"3beaf203-5283-45eb-bac3-c094e6cf8212","Type":"ContainerStarted","Data":"33c616c6c870e9cdb8b292ddb58ea93e2a3b5fc5ddfe4730f454fddcf90b3b6d"} Oct 29 23:11:41.240238 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:41.240143 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-explicit-be0401-6fc87448b6-vxln4" Oct 29 23:11:41.240238 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:41.240161 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-be0401-6fc87448b6-vxln4" event={"ID":"3beaf203-5283-45eb-bac3-c094e6cf8212","Type":"ContainerStarted","Data":"6e0be85a97fd8b01dfb5c3ffaa823f6c2b8663a2278d60691101ed88e00f41a2"} Oct 29 23:11:41.298868 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:41.298815 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-explicit-be0401-79b7645f8c-xjktr" podStartSLOduration=2.2988016780000002 podStartE2EDuration="2.298801678s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:11:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:11:41.285685043 +0000 UTC m=+2282.068898469" watchObservedRunningTime="2026-10-29 23:11:41.298801678 +0000 UTC m=+2282.082015103" Oct 29 23:11:41.329236 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:41.328231 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-explicit-be0401-6fc87448b6-vxln4" podStartSLOduration=2.328216995 podStartE2EDuration="2.328216995s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:11:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:11:41.325860519 +0000 UTC m=+2282.109073958" watchObservedRunningTime="2026-10-29 23:11:41.328216995 +0000 UTC m=+2282.111430422" Oct 29 23:11:42.097890 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:42.097856 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-b8f5c4-cdd8644c9-hpdp5" Oct 29 23:11:42.098370 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:42.098298 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-b8f5c4-584577d98b-tpwp2" Oct 29 23:11:42.639814 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:42.639775 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:11:44.152005 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:44.151968 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w0-4504f1/auth-test-model-4504f1-kserve-7694bcddfc-dt6vg"] Oct 29 23:11:44.159863 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:44.159838 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-4504f1/auth-test-model-4504f1-kserve-7694bcddfc-dt6vg" Oct 29 23:11:44.176408 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:44.176376 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-4504f1/auth-test-model-4504f1-kserve-7694bcddfc-dt6vg"] Oct 29 23:11:44.182392 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:44.180450 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-4504f1\"/\"openshift-service-ca.crt\"" Oct 29 23:11:44.182392 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:44.180562 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-4504f1\"/\"kube-root-ca.crt\"" Oct 29 23:11:44.207723 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:44.207588 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-4504f1\"/\"auth-test-model-4504f1-kserve-self-signed-certs\"" Oct 29 23:11:44.207723 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:44.207693 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-4504f1\"/\"default-dockercfg-gqfj8\"" Oct 29 23:11:44.216208 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:44.216179 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:11:44.231601 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:44.231571 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8d7303c8-8d98-4ff1-bb7c-2018d73a5033-home\") pod \"auth-test-model-4504f1-kserve-7694bcddfc-dt6vg\" (UID: \"8d7303c8-8d98-4ff1-bb7c-2018d73a5033\") " pod="ai-tenant-e2e-shared-a-w0-4504f1/auth-test-model-4504f1-kserve-7694bcddfc-dt6vg" Oct 29 23:11:44.231786 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:44.231761 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8d7303c8-8d98-4ff1-bb7c-2018d73a5033-dshm\") pod \"auth-test-model-4504f1-kserve-7694bcddfc-dt6vg\" (UID: \"8d7303c8-8d98-4ff1-bb7c-2018d73a5033\") " pod="ai-tenant-e2e-shared-a-w0-4504f1/auth-test-model-4504f1-kserve-7694bcddfc-dt6vg" Oct 29 23:11:44.232014 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:44.231989 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrfpb\" (UniqueName: \"kubernetes.io/projected/8d7303c8-8d98-4ff1-bb7c-2018d73a5033-kube-api-access-lrfpb\") pod \"auth-test-model-4504f1-kserve-7694bcddfc-dt6vg\" (UID: \"8d7303c8-8d98-4ff1-bb7c-2018d73a5033\") " pod="ai-tenant-e2e-shared-a-w0-4504f1/auth-test-model-4504f1-kserve-7694bcddfc-dt6vg" Oct 29 23:11:44.232071 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:44.232039 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8d7303c8-8d98-4ff1-bb7c-2018d73a5033-model-cache\") pod \"auth-test-model-4504f1-kserve-7694bcddfc-dt6vg\" (UID: \"8d7303c8-8d98-4ff1-bb7c-2018d73a5033\") " pod="ai-tenant-e2e-shared-a-w0-4504f1/auth-test-model-4504f1-kserve-7694bcddfc-dt6vg" Oct 29 23:11:44.232221 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:44.232204 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d7303c8-8d98-4ff1-bb7c-2018d73a5033-tls-certs\") pod \"auth-test-model-4504f1-kserve-7694bcddfc-dt6vg\" (UID: \"8d7303c8-8d98-4ff1-bb7c-2018d73a5033\") " pod="ai-tenant-e2e-shared-a-w0-4504f1/auth-test-model-4504f1-kserve-7694bcddfc-dt6vg" Oct 29 23:11:44.232288 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:44.232272 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d7303c8-8d98-4ff1-bb7c-2018d73a5033-tmp-dir\") pod \"auth-test-model-4504f1-kserve-7694bcddfc-dt6vg\" (UID: \"8d7303c8-8d98-4ff1-bb7c-2018d73a5033\") " pod="ai-tenant-e2e-shared-a-w0-4504f1/auth-test-model-4504f1-kserve-7694bcddfc-dt6vg" Oct 29 23:11:44.333958 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:44.333924 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8d7303c8-8d98-4ff1-bb7c-2018d73a5033-dshm\") pod \"auth-test-model-4504f1-kserve-7694bcddfc-dt6vg\" (UID: \"8d7303c8-8d98-4ff1-bb7c-2018d73a5033\") " pod="ai-tenant-e2e-shared-a-w0-4504f1/auth-test-model-4504f1-kserve-7694bcddfc-dt6vg" Oct 29 23:11:44.334146 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:44.334100 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrfpb\" (UniqueName: \"kubernetes.io/projected/8d7303c8-8d98-4ff1-bb7c-2018d73a5033-kube-api-access-lrfpb\") pod \"auth-test-model-4504f1-kserve-7694bcddfc-dt6vg\" (UID: \"8d7303c8-8d98-4ff1-bb7c-2018d73a5033\") " pod="ai-tenant-e2e-shared-a-w0-4504f1/auth-test-model-4504f1-kserve-7694bcddfc-dt6vg" Oct 29 23:11:44.334146 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:44.334136 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8d7303c8-8d98-4ff1-bb7c-2018d73a5033-model-cache\") pod \"auth-test-model-4504f1-kserve-7694bcddfc-dt6vg\" (UID: \"8d7303c8-8d98-4ff1-bb7c-2018d73a5033\") " pod="ai-tenant-e2e-shared-a-w0-4504f1/auth-test-model-4504f1-kserve-7694bcddfc-dt6vg" Oct 29 23:11:44.334281 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:44.334245 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d7303c8-8d98-4ff1-bb7c-2018d73a5033-tls-certs\") pod \"auth-test-model-4504f1-kserve-7694bcddfc-dt6vg\" (UID: \"8d7303c8-8d98-4ff1-bb7c-2018d73a5033\") " pod="ai-tenant-e2e-shared-a-w0-4504f1/auth-test-model-4504f1-kserve-7694bcddfc-dt6vg" Oct 29 23:11:44.334355 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:44.334318 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d7303c8-8d98-4ff1-bb7c-2018d73a5033-tmp-dir\") pod \"auth-test-model-4504f1-kserve-7694bcddfc-dt6vg\" (UID: \"8d7303c8-8d98-4ff1-bb7c-2018d73a5033\") " pod="ai-tenant-e2e-shared-a-w0-4504f1/auth-test-model-4504f1-kserve-7694bcddfc-dt6vg" Oct 29 23:11:44.334442 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:44.334428 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8d7303c8-8d98-4ff1-bb7c-2018d73a5033-home\") pod \"auth-test-model-4504f1-kserve-7694bcddfc-dt6vg\" (UID: \"8d7303c8-8d98-4ff1-bb7c-2018d73a5033\") " pod="ai-tenant-e2e-shared-a-w0-4504f1/auth-test-model-4504f1-kserve-7694bcddfc-dt6vg" Oct 29 23:11:44.334647 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:44.334617 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8d7303c8-8d98-4ff1-bb7c-2018d73a5033-model-cache\") pod \"auth-test-model-4504f1-kserve-7694bcddfc-dt6vg\" (UID: \"8d7303c8-8d98-4ff1-bb7c-2018d73a5033\") " pod="ai-tenant-e2e-shared-a-w0-4504f1/auth-test-model-4504f1-kserve-7694bcddfc-dt6vg" Oct 29 23:11:44.334729 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:44.334666 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d7303c8-8d98-4ff1-bb7c-2018d73a5033-tmp-dir\") pod \"auth-test-model-4504f1-kserve-7694bcddfc-dt6vg\" (UID: \"8d7303c8-8d98-4ff1-bb7c-2018d73a5033\") " pod="ai-tenant-e2e-shared-a-w0-4504f1/auth-test-model-4504f1-kserve-7694bcddfc-dt6vg" Oct 29 23:11:44.334800 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:44.334780 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8d7303c8-8d98-4ff1-bb7c-2018d73a5033-home\") pod \"auth-test-model-4504f1-kserve-7694bcddfc-dt6vg\" (UID: \"8d7303c8-8d98-4ff1-bb7c-2018d73a5033\") " pod="ai-tenant-e2e-shared-a-w0-4504f1/auth-test-model-4504f1-kserve-7694bcddfc-dt6vg" Oct 29 23:11:44.336497 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:44.336469 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8d7303c8-8d98-4ff1-bb7c-2018d73a5033-dshm\") pod \"auth-test-model-4504f1-kserve-7694bcddfc-dt6vg\" (UID: \"8d7303c8-8d98-4ff1-bb7c-2018d73a5033\") " pod="ai-tenant-e2e-shared-a-w0-4504f1/auth-test-model-4504f1-kserve-7694bcddfc-dt6vg" Oct 29 23:11:44.337018 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:44.336995 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d7303c8-8d98-4ff1-bb7c-2018d73a5033-tls-certs\") pod \"auth-test-model-4504f1-kserve-7694bcddfc-dt6vg\" (UID: \"8d7303c8-8d98-4ff1-bb7c-2018d73a5033\") " pod="ai-tenant-e2e-shared-a-w0-4504f1/auth-test-model-4504f1-kserve-7694bcddfc-dt6vg" Oct 29 23:11:44.356343 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:44.356311 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrfpb\" (UniqueName: \"kubernetes.io/projected/8d7303c8-8d98-4ff1-bb7c-2018d73a5033-kube-api-access-lrfpb\") pod \"auth-test-model-4504f1-kserve-7694bcddfc-dt6vg\" (UID: \"8d7303c8-8d98-4ff1-bb7c-2018d73a5033\") " pod="ai-tenant-e2e-shared-a-w0-4504f1/auth-test-model-4504f1-kserve-7694bcddfc-dt6vg" Oct 29 23:11:44.491124 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:44.491085 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-4504f1/auth-test-model-4504f1-kserve-7694bcddfc-dt6vg" Oct 29 23:11:44.823528 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:44.823379 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:11:44.836523 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:44.836490 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-4504f1/auth-test-model-4504f1-kserve-7694bcddfc-dt6vg"] Oct 29 23:11:45.277663 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:45.277624 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-4504f1/auth-test-model-4504f1-kserve-7694bcddfc-dt6vg" event={"ID":"8d7303c8-8d98-4ff1-bb7c-2018d73a5033","Type":"ContainerStarted","Data":"ffbd0e5175827b44d40f512779ee80119e511fbda77e7c45c2507d89f43c3e2a"} Oct 29 23:11:45.278072 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:45.277677 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-4504f1/auth-test-model-4504f1-kserve-7694bcddfc-dt6vg" event={"ID":"8d7303c8-8d98-4ff1-bb7c-2018d73a5033","Type":"ContainerStarted","Data":"3cbd17fd945785763b2070ae021600f680edc1a3bf4ddd25fbaa7f67b96663ac"} Oct 29 23:11:45.323261 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:45.323201 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w0-4504f1/auth-test-model-4504f1-kserve-7694bcddfc-dt6vg" podStartSLOduration=1.323184177 podStartE2EDuration="1.323184177s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:11:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:11:45.317483238 +0000 UTC m=+2286.100696656" watchObservedRunningTime="2026-10-29 23:11:45.323184177 +0000 UTC m=+2286.106397605" Oct 29 23:11:45.491499 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:45.491453 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w0-4504f1/auth-test-model-4504f1-kserve-7694bcddfc-dt6vg" Oct 29 23:11:46.840259 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:46.840226 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:11:47.914652 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:47.914615 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w4-af237d/auth-test-model-af237d-kserve-7444475bdf-5tw7n" Oct 29 23:11:47.927075 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:47.927053 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w4-af237d/auth-test-model-af237d-kserve-7444475bdf-5tw7n" Oct 29 23:11:48.314843 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:48.314810 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w4-af237d/auth-test-model-af237d-kserve-7444475bdf-5tw7n" Oct 29 23:11:48.590753 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:48.590663 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:11:49.029655 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:11:49.029612 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d2dnn" containerName="extractor" Oct 29 23:11:49.396841 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:49.396762 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:11:50.199279 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:50.199240 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:11:51.346764 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:51.346731 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:11:52.090152 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:52.090117 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:11:52.252170 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:52.252140 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-be0401-79b7645f8c-xjktr" Oct 29 23:11:52.252380 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:52.252221 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-explicit-be0401-6fc87448b6-vxln4" Oct 29 23:11:53.924826 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:53.924793 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:11:54.491939 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:54.491903 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w0-4504f1/auth-test-model-4504f1-kserve-7694bcddfc-dt6vg" Oct 29 23:11:54.504408 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:54.504383 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w0-4504f1/auth-test-model-4504f1-kserve-7694bcddfc-dt6vg" Oct 29 23:11:55.387402 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:55.387372 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w0-4504f1/auth-test-model-4504f1-kserve-7694bcddfc-dt6vg" Oct 29 23:11:55.941134 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:55.941094 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:11:57.815265 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:57.815227 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:11:59.418978 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:59.418933 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:11:59.571177 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:59.571101 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:11:59.723277 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:11:59.723239 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:12:01.314807 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:12:01.314724 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:12:01.665167 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:12:01.665075 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:12:02.715396 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:12:02.715322 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:12:02.825624 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:12:02.825591 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:12:03.111476 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:12:03.111361 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:12:03.233726 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:12:03.233689 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:12:03.532271 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:12:03.526451 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:12:05.854680 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:12:05.854633 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:12:08.348993 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:12:08.348954 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:12:08.985419 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:12:08.985380 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:12:09.111889 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:12:09.111853 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:12:10.227876 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:12:10.227841 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:12:10.681311 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:12:10.681224 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:12:10.802568 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:12:10.802517 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:12:10.803101 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:12:10.803079 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-c880c7-5d864f4568-xqcv8"] Oct 29 23:12:10.809692 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:12:10.809668 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-c880c7-5d864f4568-xqcv8" Oct 29 23:12:10.829202 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:12:10.829171 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-c880c7-5d864f4568-xqcv8"] Oct 29 23:12:10.830643 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:12:10.830620 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w5-c880c7\"" Oct 29 23:12:10.863975 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:12:10.863948 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-c880c7-58cdf8ddd7-fj789"] Oct 29 23:12:10.869318 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:12:10.869293 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-c880c7-58cdf8ddd7-fj789" Oct 29 23:12:10.894580 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:12:10.894547 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w5-c880c7-dockercfg-cr4hm\"" Oct 29 23:12:10.895353 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:12:10.895311 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-c880c7-58cdf8ddd7-fj789"] Oct 29 23:12:10.895815 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:12:10.895785 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b882cd5-e248-46d5-83d5-cdc7f3db72ba-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w5-c880c7-5d864f4568-xqcv8\" (UID: \"7b882cd5-e248-46d5-83d5-cdc7f3db72ba\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-c880c7-5d864f4568-xqcv8" Oct 29 23:12:10.895938 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:12:10.895919 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgvsh\" (UniqueName: \"kubernetes.io/projected/7b882cd5-e248-46d5-83d5-cdc7f3db72ba-kube-api-access-fgvsh\") pod \"payload-processing-e2e-shared-a-w5-c880c7-5d864f4568-xqcv8\" (UID: \"7b882cd5-e248-46d5-83d5-cdc7f3db72ba\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-c880c7-5d864f4568-xqcv8" Oct 29 23:12:11.014942 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:12:11.014898 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgvsh\" (UniqueName: \"kubernetes.io/projected/7b882cd5-e248-46d5-83d5-cdc7f3db72ba-kube-api-access-fgvsh\") pod \"payload-processing-e2e-shared-a-w5-c880c7-5d864f4568-xqcv8\" (UID: \"7b882cd5-e248-46d5-83d5-cdc7f3db72ba\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-c880c7-5d864f4568-xqcv8" Oct 29 23:12:11.015231 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:12:11.015076 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sm9bg\" (UniqueName: \"kubernetes.io/projected/babbd117-8898-4d39-9f04-e4d165379a0a-kube-api-access-sm9bg\") pod \"payload-pre-processing-e2e-shared-a-w5-c880c7-58cdf8ddd7-fj789\" (UID: \"babbd117-8898-4d39-9f04-e4d165379a0a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-c880c7-58cdf8ddd7-fj789" Oct 29 23:12:11.015231 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:12:11.015150 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/babbd117-8898-4d39-9f04-e4d165379a0a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w5-c880c7-58cdf8ddd7-fj789\" (UID: \"babbd117-8898-4d39-9f04-e4d165379a0a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-c880c7-58cdf8ddd7-fj789" Oct 29 23:12:11.016447 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:12:11.015385 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b882cd5-e248-46d5-83d5-cdc7f3db72ba-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w5-c880c7-5d864f4568-xqcv8\" (UID: \"7b882cd5-e248-46d5-83d5-cdc7f3db72ba\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-c880c7-5d864f4568-xqcv8" Oct 29 23:12:11.016447 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:12:11.016264 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b882cd5-e248-46d5-83d5-cdc7f3db72ba-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w5-c880c7-5d864f4568-xqcv8\" (UID: \"7b882cd5-e248-46d5-83d5-cdc7f3db72ba\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-c880c7-5d864f4568-xqcv8" Oct 29 23:12:11.039270 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:12:11.039236 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgvsh\" (UniqueName: \"kubernetes.io/projected/7b882cd5-e248-46d5-83d5-cdc7f3db72ba-kube-api-access-fgvsh\") pod \"payload-processing-e2e-shared-a-w5-c880c7-5d864f4568-xqcv8\" (UID: \"7b882cd5-e248-46d5-83d5-cdc7f3db72ba\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-c880c7-5d864f4568-xqcv8" Oct 29 23:12:11.117206 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:12:11.117164 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sm9bg\" (UniqueName: \"kubernetes.io/projected/babbd117-8898-4d39-9f04-e4d165379a0a-kube-api-access-sm9bg\") pod \"payload-pre-processing-e2e-shared-a-w5-c880c7-58cdf8ddd7-fj789\" (UID: \"babbd117-8898-4d39-9f04-e4d165379a0a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-c880c7-58cdf8ddd7-fj789" Oct 29 23:12:11.117963 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:12:11.117257 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/babbd117-8898-4d39-9f04-e4d165379a0a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w5-c880c7-58cdf8ddd7-fj789\" (UID: \"babbd117-8898-4d39-9f04-e4d165379a0a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-c880c7-58cdf8ddd7-fj789" Oct 29 23:12:11.118283 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:12:11.118260 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/babbd117-8898-4d39-9f04-e4d165379a0a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w5-c880c7-58cdf8ddd7-fj789\" (UID: \"babbd117-8898-4d39-9f04-e4d165379a0a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-c880c7-58cdf8ddd7-fj789" Oct 29 23:12:11.130891 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:12:11.130860 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-c880c7-5d864f4568-xqcv8" Oct 29 23:12:11.143888 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:12:11.143863 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sm9bg\" (UniqueName: \"kubernetes.io/projected/babbd117-8898-4d39-9f04-e4d165379a0a-kube-api-access-sm9bg\") pod \"payload-pre-processing-e2e-shared-a-w5-c880c7-58cdf8ddd7-fj789\" (UID: \"babbd117-8898-4d39-9f04-e4d165379a0a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-c880c7-58cdf8ddd7-fj789" Oct 29 23:12:11.197722 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:12:11.197668 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-c880c7-58cdf8ddd7-fj789" Oct 29 23:12:11.510514 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:12:11.510487 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-c880c7-5d864f4568-xqcv8"] Oct 29 23:12:11.627439 ip-10-0-140-74 kubenswrapper[5057]: W1029 23:12:11.627401 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbabbd117_8898_4d39_9f04_e4d165379a0a.slice/crio-8990e4b14d44e2cbd1a6fceffd122d53c63271a286a0f5b91a06ac4cc5deb87a WatchSource:0}: Error finding container 8990e4b14d44e2cbd1a6fceffd122d53c63271a286a0f5b91a06ac4cc5deb87a: Status 404 returned error can't find the container with id 8990e4b14d44e2cbd1a6fceffd122d53c63271a286a0f5b91a06ac4cc5deb87a Oct 29 23:12:11.627571 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:12:11.627508 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-c880c7-58cdf8ddd7-fj789"] Oct 29 23:12:11.676030 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:12:11.676000 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-c880c7-58cdf8ddd7-fj789" event={"ID":"babbd117-8898-4d39-9f04-e4d165379a0a","Type":"ContainerStarted","Data":"8990e4b14d44e2cbd1a6fceffd122d53c63271a286a0f5b91a06ac4cc5deb87a"} Oct 29 23:12:11.679238 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:12:11.679204 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-c880c7-5d864f4568-xqcv8" event={"ID":"7b882cd5-e248-46d5-83d5-cdc7f3db72ba","Type":"ContainerStarted","Data":"83d27099d1380f07d976647486fafad63b526484eb1060021d0bfdef3fed710f"} Oct 29 23:12:11.679383 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:12:11.679256 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-c880c7-5d864f4568-xqcv8" event={"ID":"7b882cd5-e248-46d5-83d5-cdc7f3db72ba","Type":"ContainerStarted","Data":"88b0efcc58d58193a8c904bbd4263a228dfe97be234b9b67987d8e7f46ac4166"} Oct 29 23:12:11.679703 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:12:11.679680 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-c880c7-5d864f4568-xqcv8" Oct 29 23:12:11.727008 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:12:11.726952 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-c880c7-5d864f4568-xqcv8" podStartSLOduration=1.726932862 podStartE2EDuration="1.726932862s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:12:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:12:11.718708826 +0000 UTC m=+2312.501922252" watchObservedRunningTime="2026-10-29 23:12:11.726932862 +0000 UTC m=+2312.510146288" Oct 29 23:12:12.155352 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:12:12.155285 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:12:12.273781 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:12:12.273345 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:12:12.422794 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:12:12.422688 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:12:12.548249 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:12:12.548197 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:12:12.691772 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:12:12.691680 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-c880c7-58cdf8ddd7-fj789" event={"ID":"babbd117-8898-4d39-9f04-e4d165379a0a","Type":"ContainerStarted","Data":"27cad8308fa7ddaf60e17e1650ac4b36bf82ec7345002409746a03c202cda7a4"} Oct 29 23:12:12.691947 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:12:12.691768 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-c880c7-58cdf8ddd7-fj789" Oct 29 23:12:12.736492 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:12:12.736434 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-c880c7-58cdf8ddd7-fj789" podStartSLOduration=2.736420921 podStartE2EDuration="2.736420921s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:12:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:12:12.731375431 +0000 UTC m=+2313.514588856" watchObservedRunningTime="2026-10-29 23:12:12.736420921 +0000 UTC m=+2313.519634347" Oct 29 23:12:14.104155 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:12:14.104117 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:12:15.734216 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:12:15.734183 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:12:15.907300 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:12:15.907243 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:12:17.321758 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:12:17.321719 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:12:18.992144 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:12:18.992107 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:12:19.029029 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:12:19.028990 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d2dnn" containerName="extractor" Oct 29 23:12:20.709318 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:12:20.709280 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:12:22.447953 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:12:22.447904 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:12:22.694565 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:12:22.694538 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-c880c7-5d864f4568-xqcv8" Oct 29 23:12:23.704262 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:12:23.704226 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-c880c7-58cdf8ddd7-fj789" Oct 29 23:12:24.103217 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:12:24.103113 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:12:25.684856 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:12:25.684818 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:12:27.324568 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:12:27.324517 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:12:28.906023 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:12:28.905979 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:12:30.439440 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:12:30.439393 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:12:31.865499 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:12:31.865452 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:12:32.031518 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:12:32.031482 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:12:33.695322 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:12:33.695286 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:12:34.065947 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:12:34.065905 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:12:35.387053 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:12:35.387002 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:12:36.964019 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:12:36.963980 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:12:38.515778 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:12:38.515744 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:12:38.784561 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:12:38.784471 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:12:39.982512 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:12:39.982472 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:12:40.126142 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:12:40.126097 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:12:40.242748 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:12:40.242257 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:12:40.576016 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:12:40.575829 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:12:40.833399 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:12:40.833114 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:12:41.274211 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:12:41.274172 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v2tfm"] Oct 29 23:12:42.799584 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:12:42.799550 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:12:45.899228 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:12:45.899187 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:12:46.286294 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:12:46.286250 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:12:46.419081 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:12:46.419045 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:12:48.034264 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:12:48.034233 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:12:48.236012 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:12:48.235974 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:12:48.251603 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:12:48.251566 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-fba83e-76fd9c4d99-56dzs"] Oct 29 23:12:48.259478 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:12:48.259449 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fba83e-76fd9c4d99-56dzs" Oct 29 23:12:48.273283 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:12:48.273221 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-fba83e-76fd9c4d99-56dzs"] Oct 29 23:12:48.279364 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:12:48.279320 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-fba83e\"" Oct 29 23:12:48.344529 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:12:48.344439 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-fba83e-6f5cbb76f8-gs9rk"] Oct 29 23:12:48.351514 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:12:48.350916 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-fba83e-6f5cbb76f8-gs9rk" Oct 29 23:12:48.351514 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:12:48.351205 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6slx\" (UniqueName: \"kubernetes.io/projected/0d2f8a82-947a-43b7-a4fd-a68dded7c872-kube-api-access-q6slx\") pod \"payload-pre-processing-e2e-inf-a-fba83e-76fd9c4d99-56dzs\" (UID: \"0d2f8a82-947a-43b7-a4fd-a68dded7c872\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fba83e-76fd9c4d99-56dzs" Oct 29 23:12:48.351514 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:12:48.351390 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d2f8a82-947a-43b7-a4fd-a68dded7c872-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-fba83e-76fd9c4d99-56dzs\" (UID: \"0d2f8a82-947a-43b7-a4fd-a68dded7c872\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fba83e-76fd9c4d99-56dzs" Oct 29 23:12:48.368525 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:12:48.368496 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-fba83e-6f5cbb76f8-gs9rk"] Oct 29 23:12:48.389626 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:12:48.389590 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:12:48.444840 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:12:48.444770 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-fba83e-dockercfg-plcgp\"" Oct 29 23:12:48.453541 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:12:48.453508 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56a3f0c8-6d9f-428a-9ba5-f22d397c56ce-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-fba83e-6f5cbb76f8-gs9rk\" (UID: \"56a3f0c8-6d9f-428a-9ba5-f22d397c56ce\") " pod="openshift-ingress/payload-processing-e2e-inf-a-fba83e-6f5cbb76f8-gs9rk" Oct 29 23:12:48.453818 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:12:48.453571 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6slx\" (UniqueName: \"kubernetes.io/projected/0d2f8a82-947a-43b7-a4fd-a68dded7c872-kube-api-access-q6slx\") pod \"payload-pre-processing-e2e-inf-a-fba83e-76fd9c4d99-56dzs\" (UID: \"0d2f8a82-947a-43b7-a4fd-a68dded7c872\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fba83e-76fd9c4d99-56dzs" Oct 29 23:12:48.453818 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:12:48.453674 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d2f8a82-947a-43b7-a4fd-a68dded7c872-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-fba83e-76fd9c4d99-56dzs\" (UID: \"0d2f8a82-947a-43b7-a4fd-a68dded7c872\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fba83e-76fd9c4d99-56dzs" Oct 29 23:12:48.454931 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:12:48.453843 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2vq9\" (UniqueName: \"kubernetes.io/projected/56a3f0c8-6d9f-428a-9ba5-f22d397c56ce-kube-api-access-m2vq9\") pod \"payload-processing-e2e-inf-a-fba83e-6f5cbb76f8-gs9rk\" (UID: \"56a3f0c8-6d9f-428a-9ba5-f22d397c56ce\") " pod="openshift-ingress/payload-processing-e2e-inf-a-fba83e-6f5cbb76f8-gs9rk" Oct 29 23:12:48.455201 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:12:48.455176 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d2f8a82-947a-43b7-a4fd-a68dded7c872-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-fba83e-76fd9c4d99-56dzs\" (UID: \"0d2f8a82-947a-43b7-a4fd-a68dded7c872\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fba83e-76fd9c4d99-56dzs" Oct 29 23:12:48.478010 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:12:48.477981 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6slx\" (UniqueName: \"kubernetes.io/projected/0d2f8a82-947a-43b7-a4fd-a68dded7c872-kube-api-access-q6slx\") pod \"payload-pre-processing-e2e-inf-a-fba83e-76fd9c4d99-56dzs\" (UID: \"0d2f8a82-947a-43b7-a4fd-a68dded7c872\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fba83e-76fd9c4d99-56dzs" Oct 29 23:12:48.555990 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:12:48.555954 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2vq9\" (UniqueName: \"kubernetes.io/projected/56a3f0c8-6d9f-428a-9ba5-f22d397c56ce-kube-api-access-m2vq9\") pod \"payload-processing-e2e-inf-a-fba83e-6f5cbb76f8-gs9rk\" (UID: \"56a3f0c8-6d9f-428a-9ba5-f22d397c56ce\") " pod="openshift-ingress/payload-processing-e2e-inf-a-fba83e-6f5cbb76f8-gs9rk" Oct 29 23:12:48.556164 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:12:48.556109 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56a3f0c8-6d9f-428a-9ba5-f22d397c56ce-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-fba83e-6f5cbb76f8-gs9rk\" (UID: \"56a3f0c8-6d9f-428a-9ba5-f22d397c56ce\") " pod="openshift-ingress/payload-processing-e2e-inf-a-fba83e-6f5cbb76f8-gs9rk" Oct 29 23:12:48.557135 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:12:48.557107 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56a3f0c8-6d9f-428a-9ba5-f22d397c56ce-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-fba83e-6f5cbb76f8-gs9rk\" (UID: \"56a3f0c8-6d9f-428a-9ba5-f22d397c56ce\") " pod="openshift-ingress/payload-processing-e2e-inf-a-fba83e-6f5cbb76f8-gs9rk" Oct 29 23:12:48.589401 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:12:48.582614 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fba83e-76fd9c4d99-56dzs" Oct 29 23:12:48.602267 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:12:48.602180 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2vq9\" (UniqueName: \"kubernetes.io/projected/56a3f0c8-6d9f-428a-9ba5-f22d397c56ce-kube-api-access-m2vq9\") pod \"payload-processing-e2e-inf-a-fba83e-6f5cbb76f8-gs9rk\" (UID: \"56a3f0c8-6d9f-428a-9ba5-f22d397c56ce\") " pod="openshift-ingress/payload-processing-e2e-inf-a-fba83e-6f5cbb76f8-gs9rk" Oct 29 23:12:48.674350 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:12:48.670902 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-fba83e-6f5cbb76f8-gs9rk" Oct 29 23:12:48.919674 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:12:48.918691 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-fba83e-76fd9c4d99-56dzs"] Oct 29 23:12:48.920730 ip-10-0-140-74 kubenswrapper[5057]: W1029 23:12:48.920639 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d2f8a82_947a_43b7_a4fd_a68dded7c872.slice/crio-b18e231e5b49be8f14127222fee0beab7c46bbb63e4ea5fb0d2d950614740f81 WatchSource:0}: Error finding container b18e231e5b49be8f14127222fee0beab7c46bbb63e4ea5fb0d2d950614740f81: Status 404 returned error can't find the container with id b18e231e5b49be8f14127222fee0beab7c46bbb63e4ea5fb0d2d950614740f81 Oct 29 23:12:49.046423 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:12:49.046382 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d2dnn" containerName="extractor" Oct 29 23:12:49.116410 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:12:49.116372 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fba83e-76fd9c4d99-56dzs" event={"ID":"0d2f8a82-947a-43b7-a4fd-a68dded7c872","Type":"ContainerStarted","Data":"3065b8e517bb24b274141413adcdfde01326c3c2cc9d3e774e27c31873ac50f8"} Oct 29 23:12:49.116578 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:12:49.116442 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fba83e-76fd9c4d99-56dzs" event={"ID":"0d2f8a82-947a-43b7-a4fd-a68dded7c872","Type":"ContainerStarted","Data":"b18e231e5b49be8f14127222fee0beab7c46bbb63e4ea5fb0d2d950614740f81"} Oct 29 23:12:49.116578 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:12:49.116477 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fba83e-76fd9c4d99-56dzs" Oct 29 23:12:49.123845 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:12:49.123765 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-fba83e-6f5cbb76f8-gs9rk"] Oct 29 23:12:49.123996 ip-10-0-140-74 kubenswrapper[5057]: W1029 23:12:49.123970 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56a3f0c8_6d9f_428a_9ba5_f22d397c56ce.slice/crio-2029a32e8ff8dd6bdf400d5c776199eb105b7ea2bc9e13cf81755133a12e3236 WatchSource:0}: Error finding container 2029a32e8ff8dd6bdf400d5c776199eb105b7ea2bc9e13cf81755133a12e3236: Status 404 returned error can't find the container with id 2029a32e8ff8dd6bdf400d5c776199eb105b7ea2bc9e13cf81755133a12e3236 Oct 29 23:12:49.158829 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:12:49.158783 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fba83e-76fd9c4d99-56dzs" podStartSLOduration=1.158769107 podStartE2EDuration="1.158769107s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:12:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:12:49.1584438 +0000 UTC m=+2349.941657239" watchObservedRunningTime="2026-10-29 23:12:49.158769107 +0000 UTC m=+2349.941982588" Oct 29 23:12:49.186307 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:12:49.186280 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:12:50.125969 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:12:50.125930 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-fba83e-6f5cbb76f8-gs9rk" event={"ID":"56a3f0c8-6d9f-428a-9ba5-f22d397c56ce","Type":"ContainerStarted","Data":"053db1251fba784cbc50d1d01ae7472dbc35a76f1d809de9a78d77e21aa6bfb5"} Oct 29 23:12:50.126425 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:12:50.125990 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-fba83e-6f5cbb76f8-gs9rk" event={"ID":"56a3f0c8-6d9f-428a-9ba5-f22d397c56ce","Type":"ContainerStarted","Data":"2029a32e8ff8dd6bdf400d5c776199eb105b7ea2bc9e13cf81755133a12e3236"} Oct 29 23:12:50.126425 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:12:50.126035 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-fba83e-6f5cbb76f8-gs9rk" Oct 29 23:12:50.173462 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:12:50.173388 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-fba83e-6f5cbb76f8-gs9rk" podStartSLOduration=2.173368239 podStartE2EDuration="2.173368239s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:12:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:12:50.166104061 +0000 UTC m=+2350.949317499" watchObservedRunningTime="2026-10-29 23:12:50.173368239 +0000 UTC m=+2350.956581667" Oct 29 23:12:51.550961 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:12:51.550912 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:12:53.290892 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:12:53.290855 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:12:53.402463 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:12:53.402424 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:12:53.753802 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:12:53.753747 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:12:55.299686 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:12:55.299630 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:12:57.026959 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:12:57.026917 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:12:58.838938 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:12:58.838899 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:13:00.129966 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:13:00.129933 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fba83e-76fd9c4d99-56dzs" Oct 29 23:13:00.657613 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:13:00.657578 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:13:01.139752 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:13:01.139722 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-fba83e-6f5cbb76f8-gs9rk" Oct 29 23:13:02.584012 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:13:02.583962 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:13:04.227114 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:13:04.227075 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:13:05.822833 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:13:05.822797 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:13:07.526782 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:13:07.526742 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:13:08.060179 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:13:08.060130 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-nxkfc"] Oct 29 23:13:08.071045 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:13:08.071009 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-nxkfc" Oct 29 23:13:08.094160 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:13:08.094126 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-nxkfc"] Oct 29 23:13:08.096139 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:13:08.096108 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-32717a-openshift-default-dockercfg-4th7d\"" Oct 29 23:13:08.141067 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:13:08.141027 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e52f656e-cf07-4fa7-aed0-21d17554a880-credential-socket\") pod \"e2e-shared-a-w3-32717a-openshift-default-69bcd68955-nxkfc\" (UID: \"e52f656e-cf07-4fa7-aed0-21d17554a880\") " pod="openshift-ingress/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-nxkfc" Oct 29 23:13:08.141258 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:13:08.141160 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-msbbt\" (UniqueName: \"kubernetes.io/projected/e52f656e-cf07-4fa7-aed0-21d17554a880-kube-api-access-msbbt\") pod \"e2e-shared-a-w3-32717a-openshift-default-69bcd68955-nxkfc\" (UID: \"e52f656e-cf07-4fa7-aed0-21d17554a880\") " pod="openshift-ingress/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-nxkfc" Oct 29 23:13:08.141425 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:13:08.141290 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e52f656e-cf07-4fa7-aed0-21d17554a880-istio-token\") pod \"e2e-shared-a-w3-32717a-openshift-default-69bcd68955-nxkfc\" (UID: \"e52f656e-cf07-4fa7-aed0-21d17554a880\") " pod="openshift-ingress/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-nxkfc" Oct 29 23:13:08.141518 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:13:08.141461 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e52f656e-cf07-4fa7-aed0-21d17554a880-workload-socket\") pod \"e2e-shared-a-w3-32717a-openshift-default-69bcd68955-nxkfc\" (UID: \"e52f656e-cf07-4fa7-aed0-21d17554a880\") " pod="openshift-ingress/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-nxkfc" Oct 29 23:13:08.141576 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:13:08.141520 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e52f656e-cf07-4fa7-aed0-21d17554a880-istio-podinfo\") pod \"e2e-shared-a-w3-32717a-openshift-default-69bcd68955-nxkfc\" (UID: \"e52f656e-cf07-4fa7-aed0-21d17554a880\") " pod="openshift-ingress/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-nxkfc" Oct 29 23:13:08.141658 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:13:08.141639 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e52f656e-cf07-4fa7-aed0-21d17554a880-istio-envoy\") pod \"e2e-shared-a-w3-32717a-openshift-default-69bcd68955-nxkfc\" (UID: \"e52f656e-cf07-4fa7-aed0-21d17554a880\") " pod="openshift-ingress/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-nxkfc" Oct 29 23:13:08.141735 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:13:08.141718 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e52f656e-cf07-4fa7-aed0-21d17554a880-workload-certs\") pod \"e2e-shared-a-w3-32717a-openshift-default-69bcd68955-nxkfc\" (UID: \"e52f656e-cf07-4fa7-aed0-21d17554a880\") " pod="openshift-ingress/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-nxkfc" Oct 29 23:13:08.141884 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:13:08.141852 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e52f656e-cf07-4fa7-aed0-21d17554a880-istio-data\") pod \"e2e-shared-a-w3-32717a-openshift-default-69bcd68955-nxkfc\" (UID: \"e52f656e-cf07-4fa7-aed0-21d17554a880\") " pod="openshift-ingress/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-nxkfc" Oct 29 23:13:08.142016 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:13:08.141996 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e52f656e-cf07-4fa7-aed0-21d17554a880-istiod-ca-cert\") pod \"e2e-shared-a-w3-32717a-openshift-default-69bcd68955-nxkfc\" (UID: \"e52f656e-cf07-4fa7-aed0-21d17554a880\") " pod="openshift-ingress/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-nxkfc" Oct 29 23:13:08.243075 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:13:08.243035 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e52f656e-cf07-4fa7-aed0-21d17554a880-workload-certs\") pod \"e2e-shared-a-w3-32717a-openshift-default-69bcd68955-nxkfc\" (UID: \"e52f656e-cf07-4fa7-aed0-21d17554a880\") " pod="openshift-ingress/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-nxkfc" Oct 29 23:13:08.243274 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:13:08.243098 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e52f656e-cf07-4fa7-aed0-21d17554a880-istio-data\") pod \"e2e-shared-a-w3-32717a-openshift-default-69bcd68955-nxkfc\" (UID: \"e52f656e-cf07-4fa7-aed0-21d17554a880\") " pod="openshift-ingress/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-nxkfc" Oct 29 23:13:08.243274 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:13:08.243166 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e52f656e-cf07-4fa7-aed0-21d17554a880-istiod-ca-cert\") pod \"e2e-shared-a-w3-32717a-openshift-default-69bcd68955-nxkfc\" (UID: \"e52f656e-cf07-4fa7-aed0-21d17554a880\") " pod="openshift-ingress/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-nxkfc" Oct 29 23:13:08.243274 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:13:08.243236 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e52f656e-cf07-4fa7-aed0-21d17554a880-credential-socket\") pod \"e2e-shared-a-w3-32717a-openshift-default-69bcd68955-nxkfc\" (UID: \"e52f656e-cf07-4fa7-aed0-21d17554a880\") " pod="openshift-ingress/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-nxkfc" Oct 29 23:13:08.243501 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:13:08.243300 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-msbbt\" (UniqueName: \"kubernetes.io/projected/e52f656e-cf07-4fa7-aed0-21d17554a880-kube-api-access-msbbt\") pod \"e2e-shared-a-w3-32717a-openshift-default-69bcd68955-nxkfc\" (UID: \"e52f656e-cf07-4fa7-aed0-21d17554a880\") " pod="openshift-ingress/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-nxkfc" Oct 29 23:13:08.243501 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:13:08.243391 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e52f656e-cf07-4fa7-aed0-21d17554a880-istio-token\") pod \"e2e-shared-a-w3-32717a-openshift-default-69bcd68955-nxkfc\" (UID: \"e52f656e-cf07-4fa7-aed0-21d17554a880\") " pod="openshift-ingress/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-nxkfc" Oct 29 23:13:08.243501 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:13:08.243496 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e52f656e-cf07-4fa7-aed0-21d17554a880-workload-socket\") pod \"e2e-shared-a-w3-32717a-openshift-default-69bcd68955-nxkfc\" (UID: \"e52f656e-cf07-4fa7-aed0-21d17554a880\") " pod="openshift-ingress/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-nxkfc" Oct 29 23:13:08.243668 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:13:08.243547 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e52f656e-cf07-4fa7-aed0-21d17554a880-istio-podinfo\") pod \"e2e-shared-a-w3-32717a-openshift-default-69bcd68955-nxkfc\" (UID: \"e52f656e-cf07-4fa7-aed0-21d17554a880\") " pod="openshift-ingress/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-nxkfc" Oct 29 23:13:08.243668 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:13:08.243563 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e52f656e-cf07-4fa7-aed0-21d17554a880-istio-data\") pod \"e2e-shared-a-w3-32717a-openshift-default-69bcd68955-nxkfc\" (UID: \"e52f656e-cf07-4fa7-aed0-21d17554a880\") " pod="openshift-ingress/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-nxkfc" Oct 29 23:13:08.243791 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:13:08.243650 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e52f656e-cf07-4fa7-aed0-21d17554a880-credential-socket\") pod \"e2e-shared-a-w3-32717a-openshift-default-69bcd68955-nxkfc\" (UID: \"e52f656e-cf07-4fa7-aed0-21d17554a880\") " pod="openshift-ingress/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-nxkfc" Oct 29 23:13:08.243791 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:13:08.243709 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e52f656e-cf07-4fa7-aed0-21d17554a880-istio-envoy\") pod \"e2e-shared-a-w3-32717a-openshift-default-69bcd68955-nxkfc\" (UID: \"e52f656e-cf07-4fa7-aed0-21d17554a880\") " pod="openshift-ingress/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-nxkfc" Oct 29 23:13:08.243930 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:13:08.243907 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e52f656e-cf07-4fa7-aed0-21d17554a880-workload-socket\") pod \"e2e-shared-a-w3-32717a-openshift-default-69bcd68955-nxkfc\" (UID: \"e52f656e-cf07-4fa7-aed0-21d17554a880\") " pod="openshift-ingress/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-nxkfc" Oct 29 23:13:08.243995 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:13:08.243938 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e52f656e-cf07-4fa7-aed0-21d17554a880-workload-certs\") pod \"e2e-shared-a-w3-32717a-openshift-default-69bcd68955-nxkfc\" (UID: \"e52f656e-cf07-4fa7-aed0-21d17554a880\") " pod="openshift-ingress/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-nxkfc" Oct 29 23:13:08.243995 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:13:08.243945 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e52f656e-cf07-4fa7-aed0-21d17554a880-istiod-ca-cert\") pod \"e2e-shared-a-w3-32717a-openshift-default-69bcd68955-nxkfc\" (UID: \"e52f656e-cf07-4fa7-aed0-21d17554a880\") " pod="openshift-ingress/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-nxkfc" Oct 29 23:13:08.245851 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:13:08.245824 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e52f656e-cf07-4fa7-aed0-21d17554a880-istio-envoy\") pod \"e2e-shared-a-w3-32717a-openshift-default-69bcd68955-nxkfc\" (UID: \"e52f656e-cf07-4fa7-aed0-21d17554a880\") " pod="openshift-ingress/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-nxkfc" Oct 29 23:13:08.246189 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:13:08.246168 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e52f656e-cf07-4fa7-aed0-21d17554a880-istio-podinfo\") pod \"e2e-shared-a-w3-32717a-openshift-default-69bcd68955-nxkfc\" (UID: \"e52f656e-cf07-4fa7-aed0-21d17554a880\") " pod="openshift-ingress/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-nxkfc" Oct 29 23:13:08.269039 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:13:08.269010 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e52f656e-cf07-4fa7-aed0-21d17554a880-istio-token\") pod \"e2e-shared-a-w3-32717a-openshift-default-69bcd68955-nxkfc\" (UID: \"e52f656e-cf07-4fa7-aed0-21d17554a880\") " pod="openshift-ingress/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-nxkfc" Oct 29 23:13:08.269039 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:13:08.269026 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-msbbt\" (UniqueName: \"kubernetes.io/projected/e52f656e-cf07-4fa7-aed0-21d17554a880-kube-api-access-msbbt\") pod \"e2e-shared-a-w3-32717a-openshift-default-69bcd68955-nxkfc\" (UID: \"e52f656e-cf07-4fa7-aed0-21d17554a880\") " pod="openshift-ingress/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-nxkfc" Oct 29 23:13:08.395593 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:13:08.395500 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-nxkfc" Oct 29 23:13:08.736069 ip-10-0-140-74 kubenswrapper[5057]: W1029 23:13:08.736042 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode52f656e_cf07_4fa7_aed0_21d17554a880.slice/crio-e0810f120c9eac168f9fe8637f5f2098ce789370cd560b103422383a4edf4b2b WatchSource:0}: Error finding container e0810f120c9eac168f9fe8637f5f2098ce789370cd560b103422383a4edf4b2b: Status 404 returned error can't find the container with id e0810f120c9eac168f9fe8637f5f2098ce789370cd560b103422383a4edf4b2b Oct 29 23:13:08.736543 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:13:08.736519 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-nxkfc"] Oct 29 23:13:08.738949 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:13:08.738916 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 23:13:08.739023 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:13:08.738979 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 23:13:08.739023 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:13:08.739009 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 23:13:09.102104 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:13:09.102009 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:13:09.350842 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:13:09.350804 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-nxkfc" event={"ID":"e52f656e-cf07-4fa7-aed0-21d17554a880","Type":"ContainerStarted","Data":"0249b628f89b9ada1a6072e7ebce456108645a13908d336075140d3d8bf44761"} Oct 29 23:13:09.351007 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:13:09.350862 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-nxkfc" event={"ID":"e52f656e-cf07-4fa7-aed0-21d17554a880","Type":"ContainerStarted","Data":"e0810f120c9eac168f9fe8637f5f2098ce789370cd560b103422383a4edf4b2b"} Oct 29 23:13:09.399404 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:13:09.396556 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-nxkfc" Oct 29 23:13:09.399404 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:13:09.398602 5057 patch_prober.go:29] interesting pod/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-nxkfc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" start-of-body= Oct 29 23:13:09.399404 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:13:09.398653 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-nxkfc" podUID="e52f656e-cf07-4fa7-aed0-21d17554a880" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" Oct 29 23:13:09.400221 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:13:09.400175 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-nxkfc" podStartSLOduration=1.400161312 podStartE2EDuration="1.400161312s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:13:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:13:09.395322886 +0000 UTC m=+2370.178536313" watchObservedRunningTime="2026-10-29 23:13:09.400161312 +0000 UTC m=+2370.183374738" Oct 29 23:13:10.407355 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:13:10.396633 5057 patch_prober.go:29] interesting pod/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-nxkfc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" start-of-body= Oct 29 23:13:10.407355 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:13:10.396690 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-nxkfc" podUID="e52f656e-cf07-4fa7-aed0-21d17554a880" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" Oct 29 23:13:10.712365 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:13:10.712291 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:13:11.413355 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:13:11.402738 5057 patch_prober.go:29] interesting pod/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-nxkfc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" start-of-body= Oct 29 23:13:11.413355 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:13:11.402800 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-nxkfc" podUID="e52f656e-cf07-4fa7-aed0-21d17554a880" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" Oct 29 23:13:11.921298 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:13:11.921244 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:13:12.398356 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:13:12.396516 5057 patch_prober.go:29] interesting pod/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-nxkfc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" start-of-body= Oct 29 23:13:12.398356 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:13:12.396585 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-nxkfc" podUID="e52f656e-cf07-4fa7-aed0-21d17554a880" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" Oct 29 23:13:12.938215 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:13:12.938179 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:13:13.401886 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:13:13.401237 5057 patch_prober.go:29] interesting pod/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-nxkfc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" start-of-body= Oct 29 23:13:13.401886 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:13:13.401308 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-nxkfc" podUID="e52f656e-cf07-4fa7-aed0-21d17554a880" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" Oct 29 23:13:14.399650 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:13:14.399615 5057 patch_prober.go:29] interesting pod/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-nxkfc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" start-of-body= Oct 29 23:13:14.400728 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:13:14.400697 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-nxkfc" podUID="e52f656e-cf07-4fa7-aed0-21d17554a880" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" Oct 29 23:13:15.213216 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:13:15.213155 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:13:15.506508 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:13:15.506433 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-nxkfc" Oct 29 23:13:15.507614 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:13:15.507587 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-nxkfc" Oct 29 23:13:15.508566 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:13:15.508543 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-nxkfc" Oct 29 23:13:16.909084 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:13:16.909041 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:13:18.510170 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:13:18.510127 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:13:19.029678 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:13:19.029639 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d2dnn" containerName="extractor" Oct 29 23:13:20.108532 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:13:20.108490 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:13:21.934087 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:13:21.933994 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:13:23.682881 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:13:23.682828 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:13:23.803161 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:13:23.803126 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:13:24.036682 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:13:24.036647 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:13:25.554988 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:13:25.554955 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:13:25.710445 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:13:25.710408 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:13:26.151905 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:13:26.151872 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:13:26.677883 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:13:26.677847 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:13:27.109423 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:13:27.109386 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:13:27.573426 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:13:27.573389 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:13:29.312538 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:13:29.312499 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:13:30.603760 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:13:30.603671 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:13:30.724614 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:13:30.724572 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:13:31.163495 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:13:31.163455 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:13:32.249850 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:13:32.249795 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:13:32.385349 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:13:32.383200 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:13:33.447094 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:13:33.447060 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:13:34.191021 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:13:34.190979 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:13:35.371723 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:13:35.371681 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:13:36.908350 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:13:36.908292 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:13:37.525584 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:13:37.525543 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:13:38.602130 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:13:38.602098 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:13:39.778839 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:13:39.778809 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-vj6md_aa6a190e-3283-458a-9dfd-2de7cb193fc5/console-operator/2.log" Oct 29 23:13:39.831753 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:13:39.831726 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-vj6md_aa6a190e-3283-458a-9dfd-2de7cb193fc5/console-operator/2.log" Oct 29 23:13:40.300590 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:13:40.300500 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:13:42.118611 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:13:42.118574 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:13:42.604773 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:13:42.604746 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:13:44.218098 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:13:44.218056 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:13:46.021485 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:13:46.021436 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:13:47.688601 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:13:47.688567 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:13:49.031955 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:13:49.031911 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d2dnn" containerName="extractor" Oct 29 23:13:49.299382 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:13:49.299274 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:13:50.955009 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:13:50.954969 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:13:52.592421 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:13:52.592379 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:13:53.464888 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:13:53.464852 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-be0401-6fc87448b6-vxln4"] Oct 29 23:13:53.496449 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:13:53.496418 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-be0401-79b7645f8c-xjktr"] Oct 29 23:13:53.610051 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:13:53.609955 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:13:53.734672 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:13:53.734376 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:13:54.252596 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:13:54.252558 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:13:56.135764 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:13:56.135711 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:13:57.824608 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:13:57.824560 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:13:58.466071 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:13:58.466031 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-explicit-be0401-6fc87448b6-vxln4" podUID="3beaf203-5283-45eb-bac3-c094e6cf8212" containerName="payload-processing" containerID="cri-o://33c616c6c870e9cdb8b292ddb58ea93e2a3b5fc5ddfe4730f454fddcf90b3b6d" gracePeriod=25 Oct 29 23:13:58.497691 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:13:58.497659 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-explicit-be0401-79b7645f8c-xjktr" podUID="832a7cbc-eb85-4516-92f5-aa93400f3618" containerName="payload-pre-processing" containerID="cri-o://24a12ac377b8c97646d53e585ae796bbd85ff23428c199ae66e10675142c9907" gracePeriod=25 Oct 29 23:13:58.837728 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:13:58.837703 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-be0401-6fc87448b6-vxln4" Oct 29 23:13:58.875852 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:13:58.875829 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-be0401-79b7645f8c-xjktr" Oct 29 23:13:58.939070 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:13:58.939028 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/832a7cbc-eb85-4516-92f5-aa93400f3618-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/832a7cbc-eb85-4516-92f5-aa93400f3618-plugins-config-volume\") pod \"832a7cbc-eb85-4516-92f5-aa93400f3618\" (UID: \"832a7cbc-eb85-4516-92f5-aa93400f3618\") " Oct 29 23:13:58.939325 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:13:58.939298 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3beaf203-5283-45eb-bac3-c094e6cf8212-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3beaf203-5283-45eb-bac3-c094e6cf8212-plugins-config-volume\") pod \"3beaf203-5283-45eb-bac3-c094e6cf8212\" (UID: \"3beaf203-5283-45eb-bac3-c094e6cf8212\") " Oct 29 23:13:58.939494 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:13:58.939475 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3beaf203-5283-45eb-bac3-c094e6cf8212-kube-api-access-x8zxl\" (UniqueName: \"kubernetes.io/projected/3beaf203-5283-45eb-bac3-c094e6cf8212-kube-api-access-x8zxl\") pod \"3beaf203-5283-45eb-bac3-c094e6cf8212\" (UID: \"3beaf203-5283-45eb-bac3-c094e6cf8212\") " Oct 29 23:13:58.939587 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:13:58.939571 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/832a7cbc-eb85-4516-92f5-aa93400f3618-kube-api-access-wv88m\" (UniqueName: \"kubernetes.io/projected/832a7cbc-eb85-4516-92f5-aa93400f3618-kube-api-access-wv88m\") pod \"832a7cbc-eb85-4516-92f5-aa93400f3618\" (UID: \"832a7cbc-eb85-4516-92f5-aa93400f3618\") " Oct 29 23:13:58.943085 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:13:58.942778 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3beaf203-5283-45eb-bac3-c094e6cf8212-plugins-config-volume" pod "3beaf203-5283-45eb-bac3-c094e6cf8212" (UID: "3beaf203-5283-45eb-bac3-c094e6cf8212"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:13:58.943709 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:13:58.943400 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/832a7cbc-eb85-4516-92f5-aa93400f3618-plugins-config-volume" pod "832a7cbc-eb85-4516-92f5-aa93400f3618" (UID: "832a7cbc-eb85-4516-92f5-aa93400f3618"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:13:58.943709 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:13:58.943605 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/832a7cbc-eb85-4516-92f5-aa93400f3618-kube-api-access-wv88m" pod "832a7cbc-eb85-4516-92f5-aa93400f3618" (UID: "832a7cbc-eb85-4516-92f5-aa93400f3618"). InnerVolumeSpecName "kube-api-access-wv88m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:13:58.950443 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:13:58.949490 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3beaf203-5283-45eb-bac3-c094e6cf8212-kube-api-access-x8zxl" pod "3beaf203-5283-45eb-bac3-c094e6cf8212" (UID: "3beaf203-5283-45eb-bac3-c094e6cf8212"). InnerVolumeSpecName "kube-api-access-x8zxl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:13:58.961545 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:13:58.961511 5057 generic.go:356] "Generic (PLEG): container finished" podID="3beaf203-5283-45eb-bac3-c094e6cf8212" containerID="33c616c6c870e9cdb8b292ddb58ea93e2a3b5fc5ddfe4730f454fddcf90b3b6d" exitCode=0 Oct 29 23:13:58.961664 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:13:58.961568 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-be0401-6fc87448b6-vxln4" Oct 29 23:13:58.961664 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:13:58.961587 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-be0401-6fc87448b6-vxln4" event={"ID":"3beaf203-5283-45eb-bac3-c094e6cf8212","Type":"ContainerDied","Data":"33c616c6c870e9cdb8b292ddb58ea93e2a3b5fc5ddfe4730f454fddcf90b3b6d"} Oct 29 23:13:58.961772 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:13:58.961673 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-be0401-6fc87448b6-vxln4" event={"ID":"3beaf203-5283-45eb-bac3-c094e6cf8212","Type":"ContainerDied","Data":"6e0be85a97fd8b01dfb5c3ffaa823f6c2b8663a2278d60691101ed88e00f41a2"} Oct 29 23:13:58.961772 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:13:58.961701 5057 scope.go:122] "RemoveContainer" containerID="33c616c6c870e9cdb8b292ddb58ea93e2a3b5fc5ddfe4730f454fddcf90b3b6d" Oct 29 23:13:58.964500 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:13:58.964367 5057 generic.go:356] "Generic (PLEG): container finished" podID="832a7cbc-eb85-4516-92f5-aa93400f3618" containerID="24a12ac377b8c97646d53e585ae796bbd85ff23428c199ae66e10675142c9907" exitCode=0 Oct 29 23:13:58.964798 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:13:58.964650 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-be0401-79b7645f8c-xjktr" event={"ID":"832a7cbc-eb85-4516-92f5-aa93400f3618","Type":"ContainerDied","Data":"24a12ac377b8c97646d53e585ae796bbd85ff23428c199ae66e10675142c9907"} Oct 29 23:13:58.964798 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:13:58.964704 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-be0401-79b7645f8c-xjktr" event={"ID":"832a7cbc-eb85-4516-92f5-aa93400f3618","Type":"ContainerDied","Data":"6efae9e54cca62e99a06c5bb4e087c7f5379d4694dc696decaca1d8842d2afc8"} Oct 29 23:13:58.965281 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:13:58.965262 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-be0401-79b7645f8c-xjktr" Oct 29 23:13:58.979277 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:13:58.979196 5057 scope.go:122] "RemoveContainer" containerID="33c616c6c870e9cdb8b292ddb58ea93e2a3b5fc5ddfe4730f454fddcf90b3b6d" Oct 29 23:13:58.979618 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:13:58.979582 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33c616c6c870e9cdb8b292ddb58ea93e2a3b5fc5ddfe4730f454fddcf90b3b6d\": container with ID starting with 33c616c6c870e9cdb8b292ddb58ea93e2a3b5fc5ddfe4730f454fddcf90b3b6d not found: ID does not exist" containerID="33c616c6c870e9cdb8b292ddb58ea93e2a3b5fc5ddfe4730f454fddcf90b3b6d" Oct 29 23:13:58.979745 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:13:58.979723 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33c616c6c870e9cdb8b292ddb58ea93e2a3b5fc5ddfe4730f454fddcf90b3b6d"} err="failed to get container status \"33c616c6c870e9cdb8b292ddb58ea93e2a3b5fc5ddfe4730f454fddcf90b3b6d\": rpc error: code = NotFound desc = could not find container \"33c616c6c870e9cdb8b292ddb58ea93e2a3b5fc5ddfe4730f454fddcf90b3b6d\": container with ID starting with 33c616c6c870e9cdb8b292ddb58ea93e2a3b5fc5ddfe4730f454fddcf90b3b6d not found: ID does not exist" Oct 29 23:13:58.979820 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:13:58.979807 5057 scope.go:122] "RemoveContainer" containerID="24a12ac377b8c97646d53e585ae796bbd85ff23428c199ae66e10675142c9907" Oct 29 23:13:58.991564 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:13:58.991544 5057 scope.go:122] "RemoveContainer" containerID="24a12ac377b8c97646d53e585ae796bbd85ff23428c199ae66e10675142c9907" Oct 29 23:13:58.991837 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:13:58.991816 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24a12ac377b8c97646d53e585ae796bbd85ff23428c199ae66e10675142c9907\": container with ID starting with 24a12ac377b8c97646d53e585ae796bbd85ff23428c199ae66e10675142c9907 not found: ID does not exist" containerID="24a12ac377b8c97646d53e585ae796bbd85ff23428c199ae66e10675142c9907" Oct 29 23:13:58.991906 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:13:58.991843 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24a12ac377b8c97646d53e585ae796bbd85ff23428c199ae66e10675142c9907"} err="failed to get container status \"24a12ac377b8c97646d53e585ae796bbd85ff23428c199ae66e10675142c9907\": rpc error: code = NotFound desc = could not find container \"24a12ac377b8c97646d53e585ae796bbd85ff23428c199ae66e10675142c9907\": container with ID starting with 24a12ac377b8c97646d53e585ae796bbd85ff23428c199ae66e10675142c9907 not found: ID does not exist" Oct 29 23:13:59.037028 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:13:59.036995 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-be0401-6fc87448b6-vxln4"] Oct 29 23:13:59.042661 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:13:59.042632 5057 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3beaf203-5283-45eb-bac3-c094e6cf8212-plugins-config-volume\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:13:59.042661 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:13:59.042663 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x8zxl\" (UniqueName: \"kubernetes.io/projected/3beaf203-5283-45eb-bac3-c094e6cf8212-kube-api-access-x8zxl\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:13:59.042880 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:13:59.042678 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wv88m\" (UniqueName: \"kubernetes.io/projected/832a7cbc-eb85-4516-92f5-aa93400f3618-kube-api-access-wv88m\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:13:59.042880 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:13:59.042692 5057 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/832a7cbc-eb85-4516-92f5-aa93400f3618-plugins-config-volume\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:13:59.044707 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:13:59.044685 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-be0401-6fc87448b6-vxln4"] Oct 29 23:13:59.077225 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:13:59.077201 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-be0401-79b7645f8c-xjktr"] Oct 29 23:13:59.081962 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:13:59.081936 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-be0401-79b7645f8c-xjktr"] Oct 29 23:13:59.335090 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:13:59.335006 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3beaf203-5283-45eb-bac3-c094e6cf8212" path="/var/lib/kubelet/pods/3beaf203-5283-45eb-bac3-c094e6cf8212/volumes" Oct 29 23:13:59.335574 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:13:59.335557 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="832a7cbc-eb85-4516-92f5-aa93400f3618" path="/var/lib/kubelet/pods/832a7cbc-eb85-4516-92f5-aa93400f3618/volumes" Oct 29 23:13:59.385213 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:13:59.385160 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:14:00.912531 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:14:00.912499 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:14:01.928668 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:14:01.928620 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v2tfm"] Oct 29 23:14:03.542957 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:14:03.538182 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:14:06.516688 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:14:06.516627 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:14:07.415954 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:14:07.415855 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:14:07.687687 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:14:07.687583 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:14:07.916212 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:14:07.916178 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:14:08.035355 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:14:08.029290 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:14:08.254751 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:14:08.254712 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:14:08.373905 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:14:08.373810 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:14:08.888322 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:14:08.888286 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:14:09.023100 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:14:09.023068 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:14:09.955777 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:14:09.955744 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:14:10.068002 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:14:10.067961 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:14:10.177717 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:14:10.177641 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:14:12.187897 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:14:12.187858 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:14:12.771881 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:14:12.771839 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:14:12.897290 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:14:12.897248 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:14:14.097178 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:14:14.097140 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:14:14.465060 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:14:14.465019 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:14:14.576588 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:14:14.576508 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:14:14.581443 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:14:14.581417 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-49d87f-bd9d969c8-gmn7p"] Oct 29 23:14:14.583199 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:14:14.583179 5057 cpu_manager.go:414] "containerMap: removing container" podUID="832a7cbc-eb85-4516-92f5-aa93400f3618" containerName="payload-pre-processing" Oct 29 23:14:14.583199 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:14:14.583199 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="832a7cbc-eb85-4516-92f5-aa93400f3618" containerName="payload-pre-processing" Oct 29 23:14:14.583348 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:14:14.583226 5057 cpu_manager.go:414] "containerMap: removing container" podUID="3beaf203-5283-45eb-bac3-c094e6cf8212" containerName="payload-processing" Oct 29 23:14:14.583348 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:14:14.583231 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3beaf203-5283-45eb-bac3-c094e6cf8212" containerName="payload-processing" Oct 29 23:14:14.583440 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:14:14.583433 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="832a7cbc-eb85-4516-92f5-aa93400f3618" containerName="payload-pre-processing" Oct 29 23:14:14.583475 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:14:14.583448 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="3beaf203-5283-45eb-bac3-c094e6cf8212" containerName="payload-processing" Oct 29 23:14:14.587652 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:14:14.587634 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-49d87f-bd9d969c8-gmn7p" Oct 29 23:14:14.603825 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:14:14.603802 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-49d87f-bd9d969c8-gmn7p"] Oct 29 23:14:14.609660 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:14:14.609636 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-49d87f\"" Oct 29 23:14:14.641217 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:14:14.641177 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pn2tx\" (UniqueName: \"kubernetes.io/projected/831c69ad-255b-4115-8e66-5b1b9b928a5d-kube-api-access-pn2tx\") pod \"payload-processing-e2e-inf-b-49d87f-bd9d969c8-gmn7p\" (UID: \"831c69ad-255b-4115-8e66-5b1b9b928a5d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-49d87f-bd9d969c8-gmn7p" Oct 29 23:14:14.641445 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:14:14.641425 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/831c69ad-255b-4115-8e66-5b1b9b928a5d-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-49d87f-bd9d969c8-gmn7p\" (UID: \"831c69ad-255b-4115-8e66-5b1b9b928a5d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-49d87f-bd9d969c8-gmn7p" Oct 29 23:14:14.655686 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:14:14.655653 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-49d87f-597c5cd9bc-q4vk4"] Oct 29 23:14:14.660847 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:14:14.660827 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-49d87f-597c5cd9bc-q4vk4" Oct 29 23:14:14.676608 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:14:14.676583 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-49d87f-dockercfg-nk2pz\"" Oct 29 23:14:14.680001 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:14:14.679973 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-49d87f-597c5cd9bc-q4vk4"] Oct 29 23:14:14.748067 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:14:14.747980 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33f2625d-10eb-432c-8958-f6768c4db656-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-49d87f-597c5cd9bc-q4vk4\" (UID: \"33f2625d-10eb-432c-8958-f6768c4db656\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-49d87f-597c5cd9bc-q4vk4" Oct 29 23:14:14.748623 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:14:14.748592 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pn2tx\" (UniqueName: \"kubernetes.io/projected/831c69ad-255b-4115-8e66-5b1b9b928a5d-kube-api-access-pn2tx\") pod \"payload-processing-e2e-inf-b-49d87f-bd9d969c8-gmn7p\" (UID: \"831c69ad-255b-4115-8e66-5b1b9b928a5d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-49d87f-bd9d969c8-gmn7p" Oct 29 23:14:14.748972 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:14:14.748953 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/831c69ad-255b-4115-8e66-5b1b9b928a5d-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-49d87f-bd9d969c8-gmn7p\" (UID: \"831c69ad-255b-4115-8e66-5b1b9b928a5d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-49d87f-bd9d969c8-gmn7p" Oct 29 23:14:14.749118 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:14:14.749101 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5kr5\" (UniqueName: \"kubernetes.io/projected/33f2625d-10eb-432c-8958-f6768c4db656-kube-api-access-c5kr5\") pod \"payload-pre-processing-e2e-inf-b-49d87f-597c5cd9bc-q4vk4\" (UID: \"33f2625d-10eb-432c-8958-f6768c4db656\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-49d87f-597c5cd9bc-q4vk4" Oct 29 23:14:14.760767 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:14:14.760737 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/831c69ad-255b-4115-8e66-5b1b9b928a5d-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-49d87f-bd9d969c8-gmn7p\" (UID: \"831c69ad-255b-4115-8e66-5b1b9b928a5d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-49d87f-bd9d969c8-gmn7p" Oct 29 23:14:14.810661 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:14:14.810249 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pn2tx\" (UniqueName: \"kubernetes.io/projected/831c69ad-255b-4115-8e66-5b1b9b928a5d-kube-api-access-pn2tx\") pod \"payload-processing-e2e-inf-b-49d87f-bd9d969c8-gmn7p\" (UID: \"831c69ad-255b-4115-8e66-5b1b9b928a5d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-49d87f-bd9d969c8-gmn7p" Oct 29 23:14:14.899409 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:14:14.875081 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5kr5\" (UniqueName: \"kubernetes.io/projected/33f2625d-10eb-432c-8958-f6768c4db656-kube-api-access-c5kr5\") pod \"payload-pre-processing-e2e-inf-b-49d87f-597c5cd9bc-q4vk4\" (UID: \"33f2625d-10eb-432c-8958-f6768c4db656\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-49d87f-597c5cd9bc-q4vk4" Oct 29 23:14:14.899409 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:14:14.875220 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33f2625d-10eb-432c-8958-f6768c4db656-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-49d87f-597c5cd9bc-q4vk4\" (UID: \"33f2625d-10eb-432c-8958-f6768c4db656\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-49d87f-597c5cd9bc-q4vk4" Oct 29 23:14:14.902114 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:14:14.900227 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33f2625d-10eb-432c-8958-f6768c4db656-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-49d87f-597c5cd9bc-q4vk4\" (UID: \"33f2625d-10eb-432c-8958-f6768c4db656\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-49d87f-597c5cd9bc-q4vk4" Oct 29 23:14:14.905434 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:14:14.905407 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-49d87f-bd9d969c8-gmn7p" Oct 29 23:14:14.947468 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:14:14.947438 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5kr5\" (UniqueName: \"kubernetes.io/projected/33f2625d-10eb-432c-8958-f6768c4db656-kube-api-access-c5kr5\") pod \"payload-pre-processing-e2e-inf-b-49d87f-597c5cd9bc-q4vk4\" (UID: \"33f2625d-10eb-432c-8958-f6768c4db656\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-49d87f-597c5cd9bc-q4vk4" Oct 29 23:14:15.007487 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:14:14.998587 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-49d87f-597c5cd9bc-q4vk4" Oct 29 23:14:15.174825 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:14:15.174687 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:14:15.303571 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:14:15.303451 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:14:15.377527 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:14:15.377492 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-49d87f-597c5cd9bc-q4vk4"] Oct 29 23:14:15.393123 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:14:15.393094 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-49d87f-bd9d969c8-gmn7p"] Oct 29 23:14:15.398028 ip-10-0-140-74 kubenswrapper[5057]: W1029 23:14:15.398001 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod831c69ad_255b_4115_8e66_5b1b9b928a5d.slice/crio-cc8765b0dd21da040cebcaaa5317668433ca0000da1fe2f1a939833b2adbe2ef WatchSource:0}: Error finding container cc8765b0dd21da040cebcaaa5317668433ca0000da1fe2f1a939833b2adbe2ef: Status 404 returned error can't find the container with id cc8765b0dd21da040cebcaaa5317668433ca0000da1fe2f1a939833b2adbe2ef Oct 29 23:14:15.509416 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:14:15.509384 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:14:16.270484 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:14:16.270443 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-49d87f-bd9d969c8-gmn7p" event={"ID":"831c69ad-255b-4115-8e66-5b1b9b928a5d","Type":"ContainerStarted","Data":"5624298438e9776429f7a0944bf085b561245e688610cd64e19818db63aa4593"} Oct 29 23:14:16.270927 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:14:16.270538 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-49d87f-bd9d969c8-gmn7p" event={"ID":"831c69ad-255b-4115-8e66-5b1b9b928a5d","Type":"ContainerStarted","Data":"cc8765b0dd21da040cebcaaa5317668433ca0000da1fe2f1a939833b2adbe2ef"} Oct 29 23:14:16.270927 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:14:16.270661 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-49d87f-bd9d969c8-gmn7p" Oct 29 23:14:16.272581 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:14:16.272554 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-49d87f-597c5cd9bc-q4vk4" event={"ID":"33f2625d-10eb-432c-8958-f6768c4db656","Type":"ContainerStarted","Data":"d11a3d9eebeceeb7ce772b050dc2198bffc5ec253fcedc6f9c72f3b5c4e2f7fd"} Oct 29 23:14:16.272691 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:14:16.272595 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-49d87f-597c5cd9bc-q4vk4" event={"ID":"33f2625d-10eb-432c-8958-f6768c4db656","Type":"ContainerStarted","Data":"deecf0e1236e84f103f206261f6210b16a924cf4b3fc0c064e84bceca82b4b5d"} Oct 29 23:14:16.272757 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:14:16.272701 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-49d87f-597c5cd9bc-q4vk4" Oct 29 23:14:16.304816 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:14:16.304779 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:14:16.313371 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:14:16.313275 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-49d87f-bd9d969c8-gmn7p" podStartSLOduration=2.313257498 podStartE2EDuration="2.313257498s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:14:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:14:16.309500982 +0000 UTC m=+2437.092714420" watchObservedRunningTime="2026-10-29 23:14:16.313257498 +0000 UTC m=+2437.096470924" Oct 29 23:14:16.352190 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:14:16.352139 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-49d87f-597c5cd9bc-q4vk4" podStartSLOduration=2.352124429 podStartE2EDuration="2.352124429s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:14:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:14:16.349483921 +0000 UTC m=+2437.132697346" watchObservedRunningTime="2026-10-29 23:14:16.352124429 +0000 UTC m=+2437.135337855" Oct 29 23:14:16.419173 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:14:16.419128 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:14:16.820028 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:14:16.819988 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:14:17.149222 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:14:17.149123 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:14:18.146758 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:14:18.146720 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:14:19.030593 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:14:19.030556 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d2dnn" containerName="extractor" Oct 29 23:14:20.070924 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:14:20.070882 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:14:20.770768 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:14:20.770734 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:14:20.891152 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:14:20.891105 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:14:22.042010 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:14:22.041974 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:14:22.896567 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:14:22.896529 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:14:23.014521 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:14:23.014477 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-59eefa-bfbc68664-trphs"] Oct 29 23:14:23.016921 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:14:23.016891 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:14:23.020471 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:14:23.020450 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-59eefa-bfbc68664-trphs" Oct 29 23:14:23.038974 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:14:23.038948 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-59eefa-bfbc68664-trphs"] Oct 29 23:14:23.043838 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:14:23.043818 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w1-59eefa\"" Oct 29 23:14:23.064563 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:14:23.064534 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w1-59eefa-dockercfg-z7vlr\"" Oct 29 23:14:23.080542 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:14:23.080515 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5000261-99fe-4f46-8056-ffe2298d8610-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-59eefa-bfbc68664-trphs\" (UID: \"e5000261-99fe-4f46-8056-ffe2298d8610\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-59eefa-bfbc68664-trphs" Oct 29 23:14:23.080679 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:14:23.080579 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnd4v\" (UniqueName: \"kubernetes.io/projected/e5000261-99fe-4f46-8056-ffe2298d8610-kube-api-access-xnd4v\") pod \"payload-processing-e2e-shared-a-w1-59eefa-bfbc68664-trphs\" (UID: \"e5000261-99fe-4f46-8056-ffe2298d8610\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-59eefa-bfbc68664-trphs" Oct 29 23:14:23.084464 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:14:23.084426 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-59eefa-6867f64dbc-5fxv2"] Oct 29 23:14:23.101925 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:14:23.101887 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-59eefa-6867f64dbc-5fxv2"] Oct 29 23:14:23.102083 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:14:23.102025 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-59eefa-6867f64dbc-5fxv2" Oct 29 23:14:23.200467 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:14:23.187882 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ffe9a7c2-5370-4aaf-99b8-d1d4bdad7f0f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-59eefa-6867f64dbc-5fxv2\" (UID: \"ffe9a7c2-5370-4aaf-99b8-d1d4bdad7f0f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-59eefa-6867f64dbc-5fxv2" Oct 29 23:14:23.200467 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:14:23.188045 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5000261-99fe-4f46-8056-ffe2298d8610-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-59eefa-bfbc68664-trphs\" (UID: \"e5000261-99fe-4f46-8056-ffe2298d8610\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-59eefa-bfbc68664-trphs" Oct 29 23:14:23.200467 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:14:23.188082 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4tp8\" (UniqueName: \"kubernetes.io/projected/ffe9a7c2-5370-4aaf-99b8-d1d4bdad7f0f-kube-api-access-p4tp8\") pod \"payload-pre-processing-e2e-shared-a-w1-59eefa-6867f64dbc-5fxv2\" (UID: \"ffe9a7c2-5370-4aaf-99b8-d1d4bdad7f0f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-59eefa-6867f64dbc-5fxv2" Oct 29 23:14:23.200467 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:14:23.188147 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnd4v\" (UniqueName: \"kubernetes.io/projected/e5000261-99fe-4f46-8056-ffe2298d8610-kube-api-access-xnd4v\") pod \"payload-processing-e2e-shared-a-w1-59eefa-bfbc68664-trphs\" (UID: \"e5000261-99fe-4f46-8056-ffe2298d8610\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-59eefa-bfbc68664-trphs" Oct 29 23:14:23.200467 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:14:23.189518 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5000261-99fe-4f46-8056-ffe2298d8610-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-59eefa-bfbc68664-trphs\" (UID: \"e5000261-99fe-4f46-8056-ffe2298d8610\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-59eefa-bfbc68664-trphs" Oct 29 23:14:23.222749 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:14:23.222714 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnd4v\" (UniqueName: \"kubernetes.io/projected/e5000261-99fe-4f46-8056-ffe2298d8610-kube-api-access-xnd4v\") pod \"payload-processing-e2e-shared-a-w1-59eefa-bfbc68664-trphs\" (UID: \"e5000261-99fe-4f46-8056-ffe2298d8610\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-59eefa-bfbc68664-trphs" Oct 29 23:14:23.289284 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:14:23.289241 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ffe9a7c2-5370-4aaf-99b8-d1d4bdad7f0f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-59eefa-6867f64dbc-5fxv2\" (UID: \"ffe9a7c2-5370-4aaf-99b8-d1d4bdad7f0f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-59eefa-6867f64dbc-5fxv2" Oct 29 23:14:23.289508 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:14:23.289378 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4tp8\" (UniqueName: \"kubernetes.io/projected/ffe9a7c2-5370-4aaf-99b8-d1d4bdad7f0f-kube-api-access-p4tp8\") pod \"payload-pre-processing-e2e-shared-a-w1-59eefa-6867f64dbc-5fxv2\" (UID: \"ffe9a7c2-5370-4aaf-99b8-d1d4bdad7f0f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-59eefa-6867f64dbc-5fxv2" Oct 29 23:14:23.290087 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:14:23.290059 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ffe9a7c2-5370-4aaf-99b8-d1d4bdad7f0f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-59eefa-6867f64dbc-5fxv2\" (UID: \"ffe9a7c2-5370-4aaf-99b8-d1d4bdad7f0f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-59eefa-6867f64dbc-5fxv2" Oct 29 23:14:23.314924 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:14:23.314895 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4tp8\" (UniqueName: \"kubernetes.io/projected/ffe9a7c2-5370-4aaf-99b8-d1d4bdad7f0f-kube-api-access-p4tp8\") pod \"payload-pre-processing-e2e-shared-a-w1-59eefa-6867f64dbc-5fxv2\" (UID: \"ffe9a7c2-5370-4aaf-99b8-d1d4bdad7f0f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-59eefa-6867f64dbc-5fxv2" Oct 29 23:14:23.343516 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:14:23.342174 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-59eefa-bfbc68664-trphs" Oct 29 23:14:23.446549 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:14:23.437566 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-59eefa-6867f64dbc-5fxv2" Oct 29 23:14:23.740569 ip-10-0-140-74 kubenswrapper[5057]: W1029 23:14:23.740538 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5000261_99fe_4f46_8056_ffe2298d8610.slice/crio-38eae6eeca1efa1ec4b9322f56aebb8a2730f9c0b6eb3025b1a2b182a2b8649f WatchSource:0}: Error finding container 38eae6eeca1efa1ec4b9322f56aebb8a2730f9c0b6eb3025b1a2b182a2b8649f: Status 404 returned error can't find the container with id 38eae6eeca1efa1ec4b9322f56aebb8a2730f9c0b6eb3025b1a2b182a2b8649f Oct 29 23:14:23.742569 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:14:23.742540 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-59eefa-bfbc68664-trphs"] Oct 29 23:14:23.947969 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:14:23.947937 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-59eefa-6867f64dbc-5fxv2"] Oct 29 23:14:23.948729 ip-10-0-140-74 kubenswrapper[5057]: W1029 23:14:23.948704 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffe9a7c2_5370_4aaf_99b8_d1d4bdad7f0f.slice/crio-d64867445d9189f9dd9860236712022bcc7533fad987f7ccf59051915b6a5001 WatchSource:0}: Error finding container d64867445d9189f9dd9860236712022bcc7533fad987f7ccf59051915b6a5001: Status 404 returned error can't find the container with id d64867445d9189f9dd9860236712022bcc7533fad987f7ccf59051915b6a5001 Oct 29 23:14:24.236956 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:14:24.236917 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:14:24.396446 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:14:24.396341 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-59eefa-bfbc68664-trphs" event={"ID":"e5000261-99fe-4f46-8056-ffe2298d8610","Type":"ContainerStarted","Data":"ee3303b0e063d310fb05ed74624103eeaa7606cdf2d137b533207e6a4a3b39a2"} Oct 29 23:14:24.396446 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:14:24.396416 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-59eefa-bfbc68664-trphs" Oct 29 23:14:24.396446 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:14:24.396440 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-59eefa-bfbc68664-trphs" event={"ID":"e5000261-99fe-4f46-8056-ffe2298d8610","Type":"ContainerStarted","Data":"38eae6eeca1efa1ec4b9322f56aebb8a2730f9c0b6eb3025b1a2b182a2b8649f"} Oct 29 23:14:24.398474 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:14:24.398445 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-59eefa-6867f64dbc-5fxv2" event={"ID":"ffe9a7c2-5370-4aaf-99b8-d1d4bdad7f0f","Type":"ContainerStarted","Data":"c42625b42e3ba7dbb9afe9682f109076edf814c1cf1417439ad9f19c7b54d5d1"} Oct 29 23:14:24.398630 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:14:24.398498 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-59eefa-6867f64dbc-5fxv2" event={"ID":"ffe9a7c2-5370-4aaf-99b8-d1d4bdad7f0f","Type":"ContainerStarted","Data":"d64867445d9189f9dd9860236712022bcc7533fad987f7ccf59051915b6a5001"} Oct 29 23:14:24.398630 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:14:24.398532 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-59eefa-6867f64dbc-5fxv2" Oct 29 23:14:24.439104 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:14:24.439040 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-59eefa-bfbc68664-trphs" podStartSLOduration=2.439022396 podStartE2EDuration="2.439022396s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:14:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:14:24.436502428 +0000 UTC m=+2445.219715879" watchObservedRunningTime="2026-10-29 23:14:24.439022396 +0000 UTC m=+2445.222235823" Oct 29 23:14:24.484451 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:14:24.484397 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-59eefa-6867f64dbc-5fxv2" podStartSLOduration=1.484381717 podStartE2EDuration="1.484381717s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:14:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:14:24.476666756 +0000 UTC m=+2445.259880196" watchObservedRunningTime="2026-10-29 23:14:24.484381717 +0000 UTC m=+2445.267595144" Oct 29 23:14:25.849676 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:14:25.849638 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:14:27.308522 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:14:27.308489 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-49d87f-597c5cd9bc-q4vk4" Oct 29 23:14:27.308905 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:14:27.308615 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-49d87f-bd9d969c8-gmn7p" Oct 29 23:14:27.429208 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:14:27.429169 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:14:28.999083 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:14:28.999049 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:14:30.527950 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:14:30.527909 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:14:32.234156 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:14:32.234115 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:14:33.807806 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:14:33.807765 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:14:34.339159 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:14:34.339116 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:14:35.412824 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:14:35.412788 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-59eefa-6867f64dbc-5fxv2" Oct 29 23:14:35.413363 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:14:35.412904 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-59eefa-bfbc68664-trphs" Oct 29 23:14:35.529892 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:14:35.529851 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:14:37.219748 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:14:37.219713 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:14:38.800110 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:14:38.800073 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:14:40.315800 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:14:40.315760 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:14:41.921734 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:14:41.921696 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:14:43.604618 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:14:43.604572 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:14:45.161881 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:14:45.161838 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:14:46.782915 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:14:46.782878 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:14:48.788827 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:14:48.788793 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:14:48.911076 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:14:48.911027 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:14:49.038927 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:14:49.038841 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d2dnn" containerName="extractor" Oct 29 23:14:51.024991 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:14:51.024900 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:14:52.391452 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:14:52.391411 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:14:53.576682 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:14:53.576642 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:14:53.821943 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:14:53.821905 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:14:54.248363 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:14:54.247033 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:14:54.613937 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:14:54.613822 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:14:56.043366 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:14:56.043319 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:14:56.408311 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:14:56.408229 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e764937-4x99x"] Oct 29 23:14:56.415130 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:14:56.415111 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e764937-4x99x" Oct 29 23:14:56.432877 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:14:56.432850 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e764937-4x99x"] Oct 29 23:14:56.437171 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:14:56.437150 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:14:56.437285 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:14:56.437207 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dmx57\"" Oct 29 23:14:56.494761 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:14:56.494715 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37fea902-3b62-4168-8cee-e4b40c29f140-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e764937-4x99x\" (UID: \"37fea902-3b62-4168-8cee-e4b40c29f140\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e764937-4x99x" Oct 29 23:14:56.495007 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:14:56.494787 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37fea902-3b62-4168-8cee-e4b40c29f140-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e764937-4x99x\" (UID: \"37fea902-3b62-4168-8cee-e4b40c29f140\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e764937-4x99x" Oct 29 23:14:56.495007 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:14:56.494828 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnwr4\" (UniqueName: \"kubernetes.io/projected/37fea902-3b62-4168-8cee-e4b40c29f140-kube-api-access-xnwr4\") pod \"maas-api-revoke-sub-3e764937-4x99x\" (UID: \"37fea902-3b62-4168-8cee-e4b40c29f140\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e764937-4x99x" Oct 29 23:14:56.596430 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:14:56.596398 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37fea902-3b62-4168-8cee-e4b40c29f140-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e764937-4x99x\" (UID: \"37fea902-3b62-4168-8cee-e4b40c29f140\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e764937-4x99x" Oct 29 23:14:56.596638 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:14:56.596451 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37fea902-3b62-4168-8cee-e4b40c29f140-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e764937-4x99x\" (UID: \"37fea902-3b62-4168-8cee-e4b40c29f140\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e764937-4x99x" Oct 29 23:14:56.596638 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:14:56.596604 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnwr4\" (UniqueName: \"kubernetes.io/projected/37fea902-3b62-4168-8cee-e4b40c29f140-kube-api-access-xnwr4\") pod \"maas-api-revoke-sub-3e764937-4x99x\" (UID: \"37fea902-3b62-4168-8cee-e4b40c29f140\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e764937-4x99x" Oct 29 23:14:56.597137 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:14:56.597115 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37fea902-3b62-4168-8cee-e4b40c29f140-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e764937-4x99x\" (UID: \"37fea902-3b62-4168-8cee-e4b40c29f140\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e764937-4x99x" Oct 29 23:14:56.599238 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:14:56.599213 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37fea902-3b62-4168-8cee-e4b40c29f140-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e764937-4x99x\" (UID: \"37fea902-3b62-4168-8cee-e4b40c29f140\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e764937-4x99x" Oct 29 23:14:56.632496 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:14:56.632452 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnwr4\" (UniqueName: \"kubernetes.io/projected/37fea902-3b62-4168-8cee-e4b40c29f140-kube-api-access-xnwr4\") pod \"maas-api-revoke-sub-3e764937-4x99x\" (UID: \"37fea902-3b62-4168-8cee-e4b40c29f140\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e764937-4x99x" Oct 29 23:14:56.733675 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:14:56.733647 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e764937-4x99x" Oct 29 23:14:57.053819 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:14:57.053786 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e764937-4x99x"] Oct 29 23:14:57.054475 ip-10-0-140-74 kubenswrapper[5057]: W1029 23:14:57.054444 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37fea902_3b62_4168_8cee_e4b40c29f140.slice/crio-2ca4ac1235e77e77c9b12b4f2daf4203c87515e04844bc1b17d6bbb9038300ec WatchSource:0}: Error finding container 2ca4ac1235e77e77c9b12b4f2daf4203c87515e04844bc1b17d6bbb9038300ec: Status 404 returned error can't find the container with id 2ca4ac1235e77e77c9b12b4f2daf4203c87515e04844bc1b17d6bbb9038300ec Oct 29 23:14:57.833942 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:14:57.833902 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e764937-4x99x" event={"ID":"37fea902-3b62-4168-8cee-e4b40c29f140","Type":"ContainerStarted","Data":"d510894b22f912db5a0119141c30dc848c0a216c90906143f4d9e835f21a5af7"} Oct 29 23:14:57.834144 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:14:57.833967 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e764937-4x99x" event={"ID":"37fea902-3b62-4168-8cee-e4b40c29f140","Type":"ContainerStarted","Data":"2ca4ac1235e77e77c9b12b4f2daf4203c87515e04844bc1b17d6bbb9038300ec"} Oct 29 23:14:57.877218 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:14:57.877111 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e764937-4x99x" podStartSLOduration=1.877098097 podStartE2EDuration="1.877098097s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:14:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:14:57.87298238 +0000 UTC m=+2478.656195808" watchObservedRunningTime="2026-10-29 23:14:57.877098097 +0000 UTC m=+2478.660311523" Oct 29 23:14:57.969483 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:14:57.969448 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:14:58.694792 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:14:58.694706 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:14:58.812931 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:14:58.812888 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:14:58.843868 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:14:58.843834 5057 generic.go:356] "Generic (PLEG): container finished" podID="37fea902-3b62-4168-8cee-e4b40c29f140" containerID="d510894b22f912db5a0119141c30dc848c0a216c90906143f4d9e835f21a5af7" exitCode=0 Oct 29 23:14:58.844034 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:14:58.843914 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e764937-4x99x" event={"ID":"37fea902-3b62-4168-8cee-e4b40c29f140","Type":"ContainerDied","Data":"d510894b22f912db5a0119141c30dc848c0a216c90906143f4d9e835f21a5af7"} Oct 29 23:14:59.900091 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:14:59.899990 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:15:00.037464 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:00.035832 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:15:00.132097 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:00.132023 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e764937-4x99x" Oct 29 23:15:00.148507 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:00.148472 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-fba83e-29888595-8wtvz"] Oct 29 23:15:00.149836 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:00.149812 5057 cpu_manager.go:414] "containerMap: removing container" podUID="37fea902-3b62-4168-8cee-e4b40c29f140" containerName="revoke-keys" Oct 29 23:15:00.149836 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:00.149835 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37fea902-3b62-4168-8cee-e4b40c29f140" containerName="revoke-keys" Oct 29 23:15:00.150106 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:00.150080 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="37fea902-3b62-4168-8cee-e4b40c29f140" containerName="revoke-keys" Oct 29 23:15:00.154505 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:00.154480 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-fba83e-29888595-8wtvz" Oct 29 23:15:00.156787 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:00.156763 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-c880c7-29888595-rmnjw"] Oct 29 23:15:00.164524 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:00.164472 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-c880c7-29888595-rmnjw" Oct 29 23:15:00.258013 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:00.257436 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-af237d-29888595-zkzf9"] Oct 29 23:15:00.265089 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:00.265054 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-fba83e-29888595-8wtvz"] Oct 29 23:15:00.265262 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:00.265166 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-af237d-29888595-zkzf9" Oct 29 23:15:00.274797 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:00.269427 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/37fea902-3b62-4168-8cee-e4b40c29f140-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37fea902-3b62-4168-8cee-e4b40c29f140-maas-api-openshift-ca\") pod \"37fea902-3b62-4168-8cee-e4b40c29f140\" (UID: \"37fea902-3b62-4168-8cee-e4b40c29f140\") " Oct 29 23:15:00.274797 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:00.269534 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37fea902-3b62-4168-8cee-e4b40c29f140-kube-api-access-xnwr4\" (UniqueName: \"kubernetes.io/projected/37fea902-3b62-4168-8cee-e4b40c29f140-kube-api-access-xnwr4\") pod \"37fea902-3b62-4168-8cee-e4b40c29f140\" (UID: \"37fea902-3b62-4168-8cee-e4b40c29f140\") " Oct 29 23:15:00.274797 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:00.269607 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/37fea902-3b62-4168-8cee-e4b40c29f140-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37fea902-3b62-4168-8cee-e4b40c29f140-maas-api-xks-ca\") pod \"37fea902-3b62-4168-8cee-e4b40c29f140\" (UID: \"37fea902-3b62-4168-8cee-e4b40c29f140\") " Oct 29 23:15:00.274797 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:00.269979 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37fea902-3b62-4168-8cee-e4b40c29f140-maas-api-openshift-ca" pod "37fea902-3b62-4168-8cee-e4b40c29f140" (UID: "37fea902-3b62-4168-8cee-e4b40c29f140"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:15:00.274797 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:00.270211 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pk86q\" (UniqueName: \"kubernetes.io/projected/56b2d3d2-3c8f-4b6f-a32a-e882dc660fa4-kube-api-access-pk86q\") pod \"maas-api-key-cleanup-e2e-inf-a-fba83e-29888595-8wtvz\" (UID: \"56b2d3d2-3c8f-4b6f-a32a-e882dc660fa4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-fba83e-29888595-8wtvz" Oct 29 23:15:00.274797 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:00.270364 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/709fa6cf-8954-48fc-ac8a-b893fb663bd9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-c880c7-29888595-rmnjw\" (UID: \"709fa6cf-8954-48fc-ac8a-b893fb663bd9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-c880c7-29888595-rmnjw" Oct 29 23:15:00.274797 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:00.270499 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7t9hm\" (UniqueName: \"kubernetes.io/projected/709fa6cf-8954-48fc-ac8a-b893fb663bd9-kube-api-access-7t9hm\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-c880c7-29888595-rmnjw\" (UID: \"709fa6cf-8954-48fc-ac8a-b893fb663bd9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-c880c7-29888595-rmnjw" Oct 29 23:15:00.274797 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:00.272865 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37fea902-3b62-4168-8cee-e4b40c29f140-kube-api-access-xnwr4" pod "37fea902-3b62-4168-8cee-e4b40c29f140" (UID: "37fea902-3b62-4168-8cee-e4b40c29f140"). InnerVolumeSpecName "kube-api-access-xnwr4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:15:00.278500 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:00.278417 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-c880c7-29888595-rmnjw"] Oct 29 23:15:00.278633 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:00.278546 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/56b2d3d2-3c8f-4b6f-a32a-e882dc660fa4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-fba83e-29888595-8wtvz\" (UID: \"56b2d3d2-3c8f-4b6f-a32a-e882dc660fa4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-fba83e-29888595-8wtvz" Oct 29 23:15:00.279596 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:00.279554 5057 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37fea902-3b62-4168-8cee-e4b40c29f140-maas-api-openshift-ca\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:15:00.279709 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:00.279605 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xnwr4\" (UniqueName: \"kubernetes.io/projected/37fea902-3b62-4168-8cee-e4b40c29f140-kube-api-access-xnwr4\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:15:00.303569 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:00.282283 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-b8f5c4-29888595-bmqxn"] Oct 29 23:15:00.305357 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:00.304044 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/37fea902-3b62-4168-8cee-e4b40c29f140-maas-api-xks-ca" pod "37fea902-3b62-4168-8cee-e4b40c29f140" (UID: "37fea902-3b62-4168-8cee-e4b40c29f140"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:15:00.358916 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:00.352617 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-af237d-29888595-zkzf9"] Oct 29 23:15:00.358916 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:00.352727 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-b8f5c4-29888595-bmqxn" Oct 29 23:15:00.358916 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:00.352761 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4504f1-29888595-7s2j5"] Oct 29 23:15:00.364511 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:00.364461 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e131af-29888595-9njpr"] Oct 29 23:15:00.365107 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:00.365086 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4504f1-29888595-7s2j5" Oct 29 23:15:00.370732 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:00.370709 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-b8f5c4-29888595-bmqxn"] Oct 29 23:15:00.370885 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:00.370756 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e131af-29888595-9njpr"] Oct 29 23:15:00.370885 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:00.370782 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4504f1-29888595-7s2j5"] Oct 29 23:15:00.370885 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:00.370809 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-575d59-29888595-x6kgd"] Oct 29 23:15:00.375403 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:00.375382 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e131af-29888595-9njpr" Oct 29 23:15:00.383265 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:00.382840 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-063b46-29888595-2hmbf"] Oct 29 23:15:00.388117 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:00.387983 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-575d59-29888595-x6kgd" Oct 29 23:15:00.389858 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:00.389699 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sz9dk\" (UniqueName: \"kubernetes.io/projected/a4b8088b-affb-4734-8c8f-72797838f689-kube-api-access-sz9dk\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-af237d-29888595-zkzf9\" (UID: \"a4b8088b-affb-4734-8c8f-72797838f689\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-af237d-29888595-zkzf9" Oct 29 23:15:00.389858 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:00.389770 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pk86q\" (UniqueName: \"kubernetes.io/projected/56b2d3d2-3c8f-4b6f-a32a-e882dc660fa4-kube-api-access-pk86q\") pod \"maas-api-key-cleanup-e2e-inf-a-fba83e-29888595-8wtvz\" (UID: \"56b2d3d2-3c8f-4b6f-a32a-e882dc660fa4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-fba83e-29888595-8wtvz" Oct 29 23:15:00.390235 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:00.390109 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/709fa6cf-8954-48fc-ac8a-b893fb663bd9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-c880c7-29888595-rmnjw\" (UID: \"709fa6cf-8954-48fc-ac8a-b893fb663bd9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-c880c7-29888595-rmnjw" Oct 29 23:15:00.390429 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:00.390394 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7t9hm\" (UniqueName: \"kubernetes.io/projected/709fa6cf-8954-48fc-ac8a-b893fb663bd9-kube-api-access-7t9hm\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-c880c7-29888595-rmnjw\" (UID: \"709fa6cf-8954-48fc-ac8a-b893fb663bd9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-c880c7-29888595-rmnjw" Oct 29 23:15:00.390754 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:00.390604 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/56b2d3d2-3c8f-4b6f-a32a-e882dc660fa4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-fba83e-29888595-8wtvz\" (UID: \"56b2d3d2-3c8f-4b6f-a32a-e882dc660fa4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-fba83e-29888595-8wtvz" Oct 29 23:15:00.390754 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:00.390704 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4b8088b-affb-4734-8c8f-72797838f689-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-af237d-29888595-zkzf9\" (UID: \"a4b8088b-affb-4734-8c8f-72797838f689\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-af237d-29888595-zkzf9" Oct 29 23:15:00.391270 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:00.391202 5057 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37fea902-3b62-4168-8cee-e4b40c29f140-maas-api-xks-ca\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:15:00.392551 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:00.392502 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-59eefa-29888595-jzqt5"] Oct 29 23:15:00.393996 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:00.393948 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/709fa6cf-8954-48fc-ac8a-b893fb663bd9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-c880c7-29888595-rmnjw\" (UID: \"709fa6cf-8954-48fc-ac8a-b893fb663bd9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-c880c7-29888595-rmnjw" Oct 29 23:15:00.394289 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:00.394175 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-063b46-29888595-2hmbf" Oct 29 23:15:00.394852 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:00.394832 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/56b2d3d2-3c8f-4b6f-a32a-e882dc660fa4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-fba83e-29888595-8wtvz\" (UID: \"56b2d3d2-3c8f-4b6f-a32a-e882dc660fa4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-fba83e-29888595-8wtvz" Oct 29 23:15:00.400487 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:00.398808 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-575d59-29888595-x6kgd"] Oct 29 23:15:00.400487 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:00.398862 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-063b46-29888595-2hmbf"] Oct 29 23:15:00.400487 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:00.398964 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-59eefa-29888595-jzqt5" Oct 29 23:15:00.401946 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:00.401814 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:15:00.402129 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:00.402076 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-59eefa-29888595-jzqt5"] Oct 29 23:15:00.420309 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:00.416375 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7t9hm\" (UniqueName: \"kubernetes.io/projected/709fa6cf-8954-48fc-ac8a-b893fb663bd9-kube-api-access-7t9hm\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-c880c7-29888595-rmnjw\" (UID: \"709fa6cf-8954-48fc-ac8a-b893fb663bd9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-c880c7-29888595-rmnjw" Oct 29 23:15:00.420309 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:00.416375 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pk86q\" (UniqueName: \"kubernetes.io/projected/56b2d3d2-3c8f-4b6f-a32a-e882dc660fa4-kube-api-access-pk86q\") pod \"maas-api-key-cleanup-e2e-inf-a-fba83e-29888595-8wtvz\" (UID: \"56b2d3d2-3c8f-4b6f-a32a-e882dc660fa4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-fba83e-29888595-8wtvz" Oct 29 23:15:00.479692 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:00.479661 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-fba83e-29888595-8wtvz" Oct 29 23:15:00.492245 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:00.492205 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hftj8\" (UniqueName: \"kubernetes.io/projected/4d649d28-2651-4a9a-b73e-05d3407743e5-kube-api-access-hftj8\") pod \"maas-api-key-cleanup-e2e-worker-w2-575d59-29888595-x6kgd\" (UID: \"4d649d28-2651-4a9a-b73e-05d3407743e5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-575d59-29888595-x6kgd" Oct 29 23:15:00.492371 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:00.492248 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dae1295f-4869-46d2-8cfb-0d89f2f3309c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-b8f5c4-29888595-bmqxn\" (UID: \"dae1295f-4869-46d2-8cfb-0d89f2f3309c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-b8f5c4-29888595-bmqxn" Oct 29 23:15:00.492371 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:00.492277 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqbqq\" (UniqueName: \"kubernetes.io/projected/84a5841f-8af1-4840-9019-a4ce56f7ed40-kube-api-access-rqbqq\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-59eefa-29888595-jzqt5\" (UID: \"84a5841f-8af1-4840-9019-a4ce56f7ed40\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-59eefa-29888595-jzqt5" Oct 29 23:15:00.492766 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:00.492726 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxcsh\" (UniqueName: \"kubernetes.io/projected/1dc9bc0e-b528-42cb-86b6-5a4790e3a0c3-kube-api-access-nxcsh\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-4504f1-29888595-7s2j5\" (UID: \"1dc9bc0e-b528-42cb-86b6-5a4790e3a0c3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4504f1-29888595-7s2j5" Oct 29 23:15:00.492850 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:00.492781 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rn9x\" (UniqueName: \"kubernetes.io/projected/653090c9-6e06-4d91-9192-66c79775eded-kube-api-access-4rn9x\") pod \"maas-api-key-cleanup-e2e-worker-w4-063b46-29888595-2hmbf\" (UID: \"653090c9-6e06-4d91-9192-66c79775eded\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-063b46-29888595-2hmbf" Oct 29 23:15:00.492908 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:00.492882 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57s8w\" (UniqueName: \"kubernetes.io/projected/d0e370f0-45f8-4b62-aa7d-52414efa778e-kube-api-access-57s8w\") pod \"maas-api-key-cleanup-e2e-worker-w1-e131af-29888595-9njpr\" (UID: \"d0e370f0-45f8-4b62-aa7d-52414efa778e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e131af-29888595-9njpr" Oct 29 23:15:00.492969 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:00.492930 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1dc9bc0e-b528-42cb-86b6-5a4790e3a0c3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-4504f1-29888595-7s2j5\" (UID: \"1dc9bc0e-b528-42cb-86b6-5a4790e3a0c3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4504f1-29888595-7s2j5" Oct 29 23:15:00.493230 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:00.493206 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4b8088b-affb-4734-8c8f-72797838f689-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-af237d-29888595-zkzf9\" (UID: \"a4b8088b-affb-4734-8c8f-72797838f689\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-af237d-29888595-zkzf9" Oct 29 23:15:00.493299 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:00.493278 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvcw5\" (UniqueName: \"kubernetes.io/projected/dae1295f-4869-46d2-8cfb-0d89f2f3309c-kube-api-access-cvcw5\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-b8f5c4-29888595-bmqxn\" (UID: \"dae1295f-4869-46d2-8cfb-0d89f2f3309c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-b8f5c4-29888595-bmqxn" Oct 29 23:15:00.493376 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:00.493361 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0e370f0-45f8-4b62-aa7d-52414efa778e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-e131af-29888595-9njpr\" (UID: \"d0e370f0-45f8-4b62-aa7d-52414efa778e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e131af-29888595-9njpr" Oct 29 23:15:00.493634 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:00.493610 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d649d28-2651-4a9a-b73e-05d3407743e5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-575d59-29888595-x6kgd\" (UID: \"4d649d28-2651-4a9a-b73e-05d3407743e5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-575d59-29888595-x6kgd" Oct 29 23:15:00.493680 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:00.493665 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/84a5841f-8af1-4840-9019-a4ce56f7ed40-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-59eefa-29888595-jzqt5\" (UID: \"84a5841f-8af1-4840-9019-a4ce56f7ed40\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-59eefa-29888595-jzqt5" Oct 29 23:15:00.493861 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:00.493840 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sz9dk\" (UniqueName: \"kubernetes.io/projected/a4b8088b-affb-4734-8c8f-72797838f689-kube-api-access-sz9dk\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-af237d-29888595-zkzf9\" (UID: \"a4b8088b-affb-4734-8c8f-72797838f689\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-af237d-29888595-zkzf9" Oct 29 23:15:00.493961 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:00.493945 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4b8088b-affb-4734-8c8f-72797838f689-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-af237d-29888595-zkzf9\" (UID: \"a4b8088b-affb-4734-8c8f-72797838f689\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-af237d-29888595-zkzf9" Oct 29 23:15:00.494022 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:00.493953 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/653090c9-6e06-4d91-9192-66c79775eded-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-063b46-29888595-2hmbf\" (UID: \"653090c9-6e06-4d91-9192-66c79775eded\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-063b46-29888595-2hmbf" Oct 29 23:15:00.526308 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:00.526277 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sz9dk\" (UniqueName: \"kubernetes.io/projected/a4b8088b-affb-4734-8c8f-72797838f689-kube-api-access-sz9dk\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-af237d-29888595-zkzf9\" (UID: \"a4b8088b-affb-4734-8c8f-72797838f689\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-af237d-29888595-zkzf9" Oct 29 23:15:00.542869 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:00.542649 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-c880c7-29888595-rmnjw" Oct 29 23:15:00.595837 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:00.595804 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvcw5\" (UniqueName: \"kubernetes.io/projected/dae1295f-4869-46d2-8cfb-0d89f2f3309c-kube-api-access-cvcw5\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-b8f5c4-29888595-bmqxn\" (UID: \"dae1295f-4869-46d2-8cfb-0d89f2f3309c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-b8f5c4-29888595-bmqxn" Oct 29 23:15:00.595987 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:00.595886 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0e370f0-45f8-4b62-aa7d-52414efa778e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-e131af-29888595-9njpr\" (UID: \"d0e370f0-45f8-4b62-aa7d-52414efa778e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e131af-29888595-9njpr" Oct 29 23:15:00.595987 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:00.595968 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d649d28-2651-4a9a-b73e-05d3407743e5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-575d59-29888595-x6kgd\" (UID: \"4d649d28-2651-4a9a-b73e-05d3407743e5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-575d59-29888595-x6kgd" Oct 29 23:15:00.596123 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:00.596009 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/84a5841f-8af1-4840-9019-a4ce56f7ed40-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-59eefa-29888595-jzqt5\" (UID: \"84a5841f-8af1-4840-9019-a4ce56f7ed40\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-59eefa-29888595-jzqt5" Oct 29 23:15:00.596123 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:00.596108 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/653090c9-6e06-4d91-9192-66c79775eded-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-063b46-29888595-2hmbf\" (UID: \"653090c9-6e06-4d91-9192-66c79775eded\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-063b46-29888595-2hmbf" Oct 29 23:15:00.596236 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:00.596203 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hftj8\" (UniqueName: \"kubernetes.io/projected/4d649d28-2651-4a9a-b73e-05d3407743e5-kube-api-access-hftj8\") pod \"maas-api-key-cleanup-e2e-worker-w2-575d59-29888595-x6kgd\" (UID: \"4d649d28-2651-4a9a-b73e-05d3407743e5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-575d59-29888595-x6kgd" Oct 29 23:15:00.596313 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:00.596238 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dae1295f-4869-46d2-8cfb-0d89f2f3309c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-b8f5c4-29888595-bmqxn\" (UID: \"dae1295f-4869-46d2-8cfb-0d89f2f3309c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-b8f5c4-29888595-bmqxn" Oct 29 23:15:00.596313 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:00.596268 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqbqq\" (UniqueName: \"kubernetes.io/projected/84a5841f-8af1-4840-9019-a4ce56f7ed40-kube-api-access-rqbqq\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-59eefa-29888595-jzqt5\" (UID: \"84a5841f-8af1-4840-9019-a4ce56f7ed40\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-59eefa-29888595-jzqt5" Oct 29 23:15:00.596481 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:00.596393 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxcsh\" (UniqueName: \"kubernetes.io/projected/1dc9bc0e-b528-42cb-86b6-5a4790e3a0c3-kube-api-access-nxcsh\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-4504f1-29888595-7s2j5\" (UID: \"1dc9bc0e-b528-42cb-86b6-5a4790e3a0c3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4504f1-29888595-7s2j5" Oct 29 23:15:00.596481 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:00.596434 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rn9x\" (UniqueName: \"kubernetes.io/projected/653090c9-6e06-4d91-9192-66c79775eded-kube-api-access-4rn9x\") pod \"maas-api-key-cleanup-e2e-worker-w4-063b46-29888595-2hmbf\" (UID: \"653090c9-6e06-4d91-9192-66c79775eded\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-063b46-29888595-2hmbf" Oct 29 23:15:00.596596 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:00.596483 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57s8w\" (UniqueName: \"kubernetes.io/projected/d0e370f0-45f8-4b62-aa7d-52414efa778e-kube-api-access-57s8w\") pod \"maas-api-key-cleanup-e2e-worker-w1-e131af-29888595-9njpr\" (UID: \"d0e370f0-45f8-4b62-aa7d-52414efa778e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e131af-29888595-9njpr" Oct 29 23:15:00.596596 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:00.596510 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1dc9bc0e-b528-42cb-86b6-5a4790e3a0c3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-4504f1-29888595-7s2j5\" (UID: \"1dc9bc0e-b528-42cb-86b6-5a4790e3a0c3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4504f1-29888595-7s2j5" Oct 29 23:15:00.597414 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:00.597388 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1dc9bc0e-b528-42cb-86b6-5a4790e3a0c3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-4504f1-29888595-7s2j5\" (UID: \"1dc9bc0e-b528-42cb-86b6-5a4790e3a0c3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4504f1-29888595-7s2j5" Oct 29 23:15:00.598348 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:00.598312 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0e370f0-45f8-4b62-aa7d-52414efa778e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-e131af-29888595-9njpr\" (UID: \"d0e370f0-45f8-4b62-aa7d-52414efa778e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e131af-29888595-9njpr" Oct 29 23:15:00.598456 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:00.598418 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dae1295f-4869-46d2-8cfb-0d89f2f3309c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-b8f5c4-29888595-bmqxn\" (UID: \"dae1295f-4869-46d2-8cfb-0d89f2f3309c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-b8f5c4-29888595-bmqxn" Oct 29 23:15:00.598971 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:00.598950 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d649d28-2651-4a9a-b73e-05d3407743e5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-575d59-29888595-x6kgd\" (UID: \"4d649d28-2651-4a9a-b73e-05d3407743e5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-575d59-29888595-x6kgd" Oct 29 23:15:00.599521 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:00.599500 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/653090c9-6e06-4d91-9192-66c79775eded-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-063b46-29888595-2hmbf\" (UID: \"653090c9-6e06-4d91-9192-66c79775eded\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-063b46-29888595-2hmbf" Oct 29 23:15:00.599628 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:00.599606 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/84a5841f-8af1-4840-9019-a4ce56f7ed40-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-59eefa-29888595-jzqt5\" (UID: \"84a5841f-8af1-4840-9019-a4ce56f7ed40\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-59eefa-29888595-jzqt5" Oct 29 23:15:00.622385 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:00.622350 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvcw5\" (UniqueName: \"kubernetes.io/projected/dae1295f-4869-46d2-8cfb-0d89f2f3309c-kube-api-access-cvcw5\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-b8f5c4-29888595-bmqxn\" (UID: \"dae1295f-4869-46d2-8cfb-0d89f2f3309c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-b8f5c4-29888595-bmqxn" Oct 29 23:15:00.625450 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:00.625418 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hftj8\" (UniqueName: \"kubernetes.io/projected/4d649d28-2651-4a9a-b73e-05d3407743e5-kube-api-access-hftj8\") pod \"maas-api-key-cleanup-e2e-worker-w2-575d59-29888595-x6kgd\" (UID: \"4d649d28-2651-4a9a-b73e-05d3407743e5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-575d59-29888595-x6kgd" Oct 29 23:15:00.627320 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:00.627290 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqbqq\" (UniqueName: \"kubernetes.io/projected/84a5841f-8af1-4840-9019-a4ce56f7ed40-kube-api-access-rqbqq\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-59eefa-29888595-jzqt5\" (UID: \"84a5841f-8af1-4840-9019-a4ce56f7ed40\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-59eefa-29888595-jzqt5" Oct 29 23:15:00.628047 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:00.628000 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-57s8w\" (UniqueName: \"kubernetes.io/projected/d0e370f0-45f8-4b62-aa7d-52414efa778e-kube-api-access-57s8w\") pod \"maas-api-key-cleanup-e2e-worker-w1-e131af-29888595-9njpr\" (UID: \"d0e370f0-45f8-4b62-aa7d-52414efa778e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e131af-29888595-9njpr" Oct 29 23:15:00.629025 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:00.628997 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rn9x\" (UniqueName: \"kubernetes.io/projected/653090c9-6e06-4d91-9192-66c79775eded-kube-api-access-4rn9x\") pod \"maas-api-key-cleanup-e2e-worker-w4-063b46-29888595-2hmbf\" (UID: \"653090c9-6e06-4d91-9192-66c79775eded\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-063b46-29888595-2hmbf" Oct 29 23:15:00.630735 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:00.630684 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxcsh\" (UniqueName: \"kubernetes.io/projected/1dc9bc0e-b528-42cb-86b6-5a4790e3a0c3-kube-api-access-nxcsh\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-4504f1-29888595-7s2j5\" (UID: \"1dc9bc0e-b528-42cb-86b6-5a4790e3a0c3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4504f1-29888595-7s2j5" Oct 29 23:15:00.655238 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:00.655020 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-af237d-29888595-zkzf9" Oct 29 23:15:00.679298 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:00.679237 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-b8f5c4-29888595-bmqxn" Oct 29 23:15:00.719032 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:00.719001 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4504f1-29888595-7s2j5" Oct 29 23:15:00.749563 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:00.749281 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e131af-29888595-9njpr" Oct 29 23:15:00.760674 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:00.760461 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-575d59-29888595-x6kgd" Oct 29 23:15:00.770003 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:00.769820 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-063b46-29888595-2hmbf" Oct 29 23:15:00.777281 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:00.777258 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-59eefa-29888595-jzqt5" Oct 29 23:15:00.817557 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:00.817446 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-fba83e-29888595-8wtvz"] Oct 29 23:15:00.892107 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:00.890177 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e764937-4x99x" event={"ID":"37fea902-3b62-4168-8cee-e4b40c29f140","Type":"ContainerDied","Data":"2ca4ac1235e77e77c9b12b4f2daf4203c87515e04844bc1b17d6bbb9038300ec"} Oct 29 23:15:00.892107 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:00.890239 5057 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2ca4ac1235e77e77c9b12b4f2daf4203c87515e04844bc1b17d6bbb9038300ec" Oct 29 23:15:00.892107 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:00.890349 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e764937-4x99x" Oct 29 23:15:00.910558 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:00.910527 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-c880c7-29888595-rmnjw"] Oct 29 23:15:01.103414 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:01.085058 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:15:01.212985 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:01.212947 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:15:01.245656 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:01.245512 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-af237d-29888595-zkzf9"] Oct 29 23:15:01.249196 ip-10-0-140-74 kubenswrapper[5057]: W1029 23:15:01.249162 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4b8088b_affb_4734_8c8f_72797838f689.slice/crio-ff10e3debbb7697024e324b377500d8e2fd98fccb4ec89900052406aabe4cdf9 WatchSource:0}: Error finding container ff10e3debbb7697024e324b377500d8e2fd98fccb4ec89900052406aabe4cdf9: Status 404 returned error can't find the container with id ff10e3debbb7697024e324b377500d8e2fd98fccb4ec89900052406aabe4cdf9 Oct 29 23:15:01.368784 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:01.368747 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-462fb2-6999659b56-gh9vv"] Oct 29 23:15:01.446632 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:01.446565 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-462fb2-6999659b56-gh9vv"] Oct 29 23:15:01.446916 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:01.446897 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-462fb2-6999659b56-gh9vv" Oct 29 23:15:01.463584 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:01.462621 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-462fb2-dockercfg-mxjvw\"" Oct 29 23:15:01.463584 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:01.462721 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-462fb2\"" Oct 29 23:15:01.553907 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:01.553623 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9zbd\" (UniqueName: \"kubernetes.io/projected/d5c09a5d-ceed-4746-a4e7-31714b345610-kube-api-access-g9zbd\") pod \"payload-processing-e2e-inf-a-462fb2-6999659b56-gh9vv\" (UID: \"d5c09a5d-ceed-4746-a4e7-31714b345610\") " pod="openshift-ingress/payload-processing-e2e-inf-a-462fb2-6999659b56-gh9vv" Oct 29 23:15:01.556570 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:01.554383 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5c09a5d-ceed-4746-a4e7-31714b345610-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-462fb2-6999659b56-gh9vv\" (UID: \"d5c09a5d-ceed-4746-a4e7-31714b345610\") " pod="openshift-ingress/payload-processing-e2e-inf-a-462fb2-6999659b56-gh9vv" Oct 29 23:15:01.562128 ip-10-0-140-74 kubenswrapper[5057]: W1029 23:15:01.562100 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1dc9bc0e_b528_42cb_86b6_5a4790e3a0c3.slice/crio-3589ca23afbf99ecedfea4ebfc6ef1d0005e6641472b83a2a9df56144bb7a1fd WatchSource:0}: Error finding container 3589ca23afbf99ecedfea4ebfc6ef1d0005e6641472b83a2a9df56144bb7a1fd: Status 404 returned error can't find the container with id 3589ca23afbf99ecedfea4ebfc6ef1d0005e6641472b83a2a9df56144bb7a1fd Oct 29 23:15:01.562421 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:01.562368 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4504f1-29888595-7s2j5"] Oct 29 23:15:01.675447 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:01.659411 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9zbd\" (UniqueName: \"kubernetes.io/projected/d5c09a5d-ceed-4746-a4e7-31714b345610-kube-api-access-g9zbd\") pod \"payload-processing-e2e-inf-a-462fb2-6999659b56-gh9vv\" (UID: \"d5c09a5d-ceed-4746-a4e7-31714b345610\") " pod="openshift-ingress/payload-processing-e2e-inf-a-462fb2-6999659b56-gh9vv" Oct 29 23:15:01.675447 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:01.659651 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5c09a5d-ceed-4746-a4e7-31714b345610-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-462fb2-6999659b56-gh9vv\" (UID: \"d5c09a5d-ceed-4746-a4e7-31714b345610\") " pod="openshift-ingress/payload-processing-e2e-inf-a-462fb2-6999659b56-gh9vv" Oct 29 23:15:01.675447 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:01.660711 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5c09a5d-ceed-4746-a4e7-31714b345610-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-462fb2-6999659b56-gh9vv\" (UID: \"d5c09a5d-ceed-4746-a4e7-31714b345610\") " pod="openshift-ingress/payload-processing-e2e-inf-a-462fb2-6999659b56-gh9vv" Oct 29 23:15:01.688243 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:01.688213 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-b8f5c4-29888595-bmqxn"] Oct 29 23:15:01.689633 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:01.689609 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9zbd\" (UniqueName: \"kubernetes.io/projected/d5c09a5d-ceed-4746-a4e7-31714b345610-kube-api-access-g9zbd\") pod \"payload-processing-e2e-inf-a-462fb2-6999659b56-gh9vv\" (UID: \"d5c09a5d-ceed-4746-a4e7-31714b345610\") " pod="openshift-ingress/payload-processing-e2e-inf-a-462fb2-6999659b56-gh9vv" Oct 29 23:15:01.701704 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:01.701620 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e131af-29888595-9njpr"] Oct 29 23:15:01.708061 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:01.708036 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-063b46-29888595-2hmbf"] Oct 29 23:15:01.717266 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:01.717247 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-59eefa-29888595-jzqt5"] Oct 29 23:15:01.721853 ip-10-0-140-74 kubenswrapper[5057]: W1029 23:15:01.721827 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod653090c9_6e06_4d91_9192_66c79775eded.slice/crio-7073be3d0379895d5250f681ad1d70acaf7f633f65f42539af234e04baee5c31 WatchSource:0}: Error finding container 7073be3d0379895d5250f681ad1d70acaf7f633f65f42539af234e04baee5c31: Status 404 returned error can't find the container with id 7073be3d0379895d5250f681ad1d70acaf7f633f65f42539af234e04baee5c31 Oct 29 23:15:01.733427 ip-10-0-140-74 kubenswrapper[5057]: W1029 23:15:01.732695 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84a5841f_8af1_4840_9019_a4ce56f7ed40.slice/crio-63f1edb68ca64a8df362c1b904cced87ff52602cda9633a0bcde55acb08bda15 WatchSource:0}: Error finding container 63f1edb68ca64a8df362c1b904cced87ff52602cda9633a0bcde55acb08bda15: Status 404 returned error can't find the container with id 63f1edb68ca64a8df362c1b904cced87ff52602cda9633a0bcde55acb08bda15 Oct 29 23:15:01.791591 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:01.791562 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-575d59-29888595-x6kgd"] Oct 29 23:15:01.819690 ip-10-0-140-74 kubenswrapper[5057]: W1029 23:15:01.819650 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d649d28_2651_4a9a_b73e_05d3407743e5.slice/crio-fb6849c24410c7fddcf3485f654c8334a62b05e5a18c36a3a16efa9a747324ec WatchSource:0}: Error finding container fb6849c24410c7fddcf3485f654c8334a62b05e5a18c36a3a16efa9a747324ec: Status 404 returned error can't find the container with id fb6849c24410c7fddcf3485f654c8334a62b05e5a18c36a3a16efa9a747324ec Oct 29 23:15:01.855303 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:01.850271 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-462fb2-6999659b56-gh9vv" Oct 29 23:15:01.911534 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:01.911490 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-063b46-29888595-2hmbf" event={"ID":"653090c9-6e06-4d91-9192-66c79775eded","Type":"ContainerStarted","Data":"7073be3d0379895d5250f681ad1d70acaf7f633f65f42539af234e04baee5c31"} Oct 29 23:15:01.915350 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:01.914151 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-c880c7-29888595-rmnjw" event={"ID":"709fa6cf-8954-48fc-ac8a-b893fb663bd9","Type":"ContainerStarted","Data":"a13134cd0fdc98c584aac76fb7dee1cd95ae83a57b9c63f6598cad7eb2de23c7"} Oct 29 23:15:01.915350 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:01.914203 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-c880c7-29888595-rmnjw" event={"ID":"709fa6cf-8954-48fc-ac8a-b893fb663bd9","Type":"ContainerStarted","Data":"83f41edf2d27b54d356a13fac0d747a169e8a6b9de77d0bb3103aa7219e7761c"} Oct 29 23:15:01.917319 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:01.917291 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4504f1-29888595-7s2j5" event={"ID":"1dc9bc0e-b528-42cb-86b6-5a4790e3a0c3","Type":"ContainerStarted","Data":"3589ca23afbf99ecedfea4ebfc6ef1d0005e6641472b83a2a9df56144bb7a1fd"} Oct 29 23:15:01.919128 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:01.919079 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-fba83e-29888595-8wtvz" event={"ID":"56b2d3d2-3c8f-4b6f-a32a-e882dc660fa4","Type":"ContainerStarted","Data":"85738245ffa423ae0d44e52461e3cceca82ed35dcc5350abd3ec8495f6e7a14b"} Oct 29 23:15:01.919236 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:01.919153 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-fba83e-29888595-8wtvz" event={"ID":"56b2d3d2-3c8f-4b6f-a32a-e882dc660fa4","Type":"ContainerStarted","Data":"ff6a76c04e414351bb5efa2c1243ed83159dc5cd89f0770ab83e8676a639f30d"} Oct 29 23:15:01.921422 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:01.921401 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e131af-29888595-9njpr" event={"ID":"d0e370f0-45f8-4b62-aa7d-52414efa778e","Type":"ContainerStarted","Data":"b607c1c14f717f92d0425072e2ad5140f46462bf17f5620e92ddec5a1d392e98"} Oct 29 23:15:01.922911 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:01.922872 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-59eefa-29888595-jzqt5" event={"ID":"84a5841f-8af1-4840-9019-a4ce56f7ed40","Type":"ContainerStarted","Data":"63f1edb68ca64a8df362c1b904cced87ff52602cda9633a0bcde55acb08bda15"} Oct 29 23:15:01.924628 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:01.924258 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-b8f5c4-29888595-bmqxn" event={"ID":"dae1295f-4869-46d2-8cfb-0d89f2f3309c","Type":"ContainerStarted","Data":"4ab65a1857bd5af26ebf8af2821579d95fc2b882b7cab78f4e3fe5c1f9f3fd26"} Oct 29 23:15:01.925809 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:01.925786 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-575d59-29888595-x6kgd" event={"ID":"4d649d28-2651-4a9a-b73e-05d3407743e5","Type":"ContainerStarted","Data":"fb6849c24410c7fddcf3485f654c8334a62b05e5a18c36a3a16efa9a747324ec"} Oct 29 23:15:01.927752 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:01.927730 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-af237d-29888595-zkzf9" event={"ID":"a4b8088b-affb-4734-8c8f-72797838f689","Type":"ContainerStarted","Data":"162b4e39603d1735830dbcead0df45b9d4cbcd3c79e2ad507010dba2ce1f0c94"} Oct 29 23:15:01.927822 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:01.927765 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-af237d-29888595-zkzf9" event={"ID":"a4b8088b-affb-4734-8c8f-72797838f689","Type":"ContainerStarted","Data":"ff10e3debbb7697024e324b377500d8e2fd98fccb4ec89900052406aabe4cdf9"} Oct 29 23:15:01.957177 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:01.957103 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-c880c7-29888595-rmnjw" podStartSLOduration=1.957091212 podStartE2EDuration="1.957091212s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:15:01.95408333 +0000 UTC m=+2482.737296758" watchObservedRunningTime="2026-10-29 23:15:01.957091212 +0000 UTC m=+2482.740304638" Oct 29 23:15:01.994484 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:01.994429 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-af237d-29888595-zkzf9" podStartSLOduration=1.994410363 podStartE2EDuration="1.994410363s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:15:01.992410364 +0000 UTC m=+2482.775623816" watchObservedRunningTime="2026-10-29 23:15:01.994410363 +0000 UTC m=+2482.777623790" Oct 29 23:15:02.037862 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:02.037379 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-fba83e-29888595-8wtvz" podStartSLOduration=2.037359241 podStartE2EDuration="2.037359241s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:15:02.032534266 +0000 UTC m=+2482.815747689" watchObservedRunningTime="2026-10-29 23:15:02.037359241 +0000 UTC m=+2482.820572668" Oct 29 23:15:02.282005 ip-10-0-140-74 kubenswrapper[5057]: W1029 23:15:02.281974 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5c09a5d_ceed_4746_a4e7_31714b345610.slice/crio-80e2db92fd507774dbba858b9e3d1d5cf00c2675dfdeb31779fa6d0074b98d75 WatchSource:0}: Error finding container 80e2db92fd507774dbba858b9e3d1d5cf00c2675dfdeb31779fa6d0074b98d75: Status 404 returned error can't find the container with id 80e2db92fd507774dbba858b9e3d1d5cf00c2675dfdeb31779fa6d0074b98d75 Oct 29 23:15:02.285503 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:02.285406 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-462fb2-6999659b56-gh9vv"] Oct 29 23:15:02.535137 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:02.535072 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:15:02.938097 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:02.938064 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-575d59-29888595-x6kgd" event={"ID":"4d649d28-2651-4a9a-b73e-05d3407743e5","Type":"ContainerStarted","Data":"a0703019c28ae0f6d8aeae2cf7b0f09fd3ab43fd94446cc0c3882f63e1359354"} Oct 29 23:15:02.939952 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:02.939926 5057 generic.go:356] "Generic (PLEG): container finished" podID="a4b8088b-affb-4734-8c8f-72797838f689" containerID="162b4e39603d1735830dbcead0df45b9d4cbcd3c79e2ad507010dba2ce1f0c94" exitCode=60 Oct 29 23:15:02.940087 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:02.940003 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-af237d-29888595-zkzf9" event={"ID":"a4b8088b-affb-4734-8c8f-72797838f689","Type":"ContainerDied","Data":"162b4e39603d1735830dbcead0df45b9d4cbcd3c79e2ad507010dba2ce1f0c94"} Oct 29 23:15:02.940306 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:02.940289 5057 scope.go:122] "RemoveContainer" containerID="162b4e39603d1735830dbcead0df45b9d4cbcd3c79e2ad507010dba2ce1f0c94" Oct 29 23:15:02.942109 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:02.942041 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-063b46-29888595-2hmbf" event={"ID":"653090c9-6e06-4d91-9192-66c79775eded","Type":"ContainerStarted","Data":"54e2f4e5ac88abbb453deee2eea03fed5c08b73f99ae7e8b4e7da2a5dc40c11b"} Oct 29 23:15:02.943953 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:02.943932 5057 generic.go:356] "Generic (PLEG): container finished" podID="709fa6cf-8954-48fc-ac8a-b893fb663bd9" containerID="a13134cd0fdc98c584aac76fb7dee1cd95ae83a57b9c63f6598cad7eb2de23c7" exitCode=60 Oct 29 23:15:02.944062 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:02.944021 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-c880c7-29888595-rmnjw" event={"ID":"709fa6cf-8954-48fc-ac8a-b893fb663bd9","Type":"ContainerDied","Data":"a13134cd0fdc98c584aac76fb7dee1cd95ae83a57b9c63f6598cad7eb2de23c7"} Oct 29 23:15:02.944236 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:02.944221 5057 scope.go:122] "RemoveContainer" containerID="a13134cd0fdc98c584aac76fb7dee1cd95ae83a57b9c63f6598cad7eb2de23c7" Oct 29 23:15:02.945623 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:02.945600 5057 generic.go:356] "Generic (PLEG): container finished" podID="1dc9bc0e-b528-42cb-86b6-5a4790e3a0c3" containerID="ddd459a209873277145e64c2df566e4023c7e159bd84d0f1dd5f8313a99c4a5f" exitCode=60 Oct 29 23:15:02.945714 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:02.945678 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4504f1-29888595-7s2j5" event={"ID":"1dc9bc0e-b528-42cb-86b6-5a4790e3a0c3","Type":"ContainerDied","Data":"ddd459a209873277145e64c2df566e4023c7e159bd84d0f1dd5f8313a99c4a5f"} Oct 29 23:15:02.945868 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:02.945854 5057 scope.go:122] "RemoveContainer" containerID="ddd459a209873277145e64c2df566e4023c7e159bd84d0f1dd5f8313a99c4a5f" Oct 29 23:15:02.947105 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:02.947086 5057 generic.go:356] "Generic (PLEG): container finished" podID="56b2d3d2-3c8f-4b6f-a32a-e882dc660fa4" containerID="85738245ffa423ae0d44e52461e3cceca82ed35dcc5350abd3ec8495f6e7a14b" exitCode=60 Oct 29 23:15:02.947179 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:02.947144 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-fba83e-29888595-8wtvz" event={"ID":"56b2d3d2-3c8f-4b6f-a32a-e882dc660fa4","Type":"ContainerDied","Data":"85738245ffa423ae0d44e52461e3cceca82ed35dcc5350abd3ec8495f6e7a14b"} Oct 29 23:15:02.947420 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:02.947406 5057 scope.go:122] "RemoveContainer" containerID="85738245ffa423ae0d44e52461e3cceca82ed35dcc5350abd3ec8495f6e7a14b" Oct 29 23:15:02.948848 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:02.948829 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e131af-29888595-9njpr" event={"ID":"d0e370f0-45f8-4b62-aa7d-52414efa778e","Type":"ContainerStarted","Data":"654e3516979e2e1ac47f4c7e9e9ce63876d458f7f426577484dde128a3166dc5"} Oct 29 23:15:02.950779 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:02.950758 5057 generic.go:356] "Generic (PLEG): container finished" podID="84a5841f-8af1-4840-9019-a4ce56f7ed40" containerID="6882e917d5fdfb8d7952beacbcb8d84f17a4b3e8f8299df71d5c877d81d309b6" exitCode=7 Oct 29 23:15:02.950865 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:02.950838 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-59eefa-29888595-jzqt5" event={"ID":"84a5841f-8af1-4840-9019-a4ce56f7ed40","Type":"ContainerDied","Data":"6882e917d5fdfb8d7952beacbcb8d84f17a4b3e8f8299df71d5c877d81d309b6"} Oct 29 23:15:02.951011 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:02.950996 5057 scope.go:122] "RemoveContainer" containerID="6882e917d5fdfb8d7952beacbcb8d84f17a4b3e8f8299df71d5c877d81d309b6" Oct 29 23:15:02.952593 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:02.952566 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-b8f5c4-29888595-bmqxn" event={"ID":"dae1295f-4869-46d2-8cfb-0d89f2f3309c","Type":"ContainerStarted","Data":"4b33442ad88b7ad27a0a6dadf6eb4f1cfc18e4d85a0e881b2ab1582ad2c0cf42"} Oct 29 23:15:02.955821 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:02.955797 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-462fb2-6999659b56-gh9vv" event={"ID":"d5c09a5d-ceed-4746-a4e7-31714b345610","Type":"ContainerStarted","Data":"342ced5cabd04c31d69529dc652f207c19c8dec0fcc7db5ef94b46a48fc3084b"} Oct 29 23:15:02.955904 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:02.955844 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-462fb2-6999659b56-gh9vv" event={"ID":"d5c09a5d-ceed-4746-a4e7-31714b345610","Type":"ContainerStarted","Data":"80e2db92fd507774dbba858b9e3d1d5cf00c2675dfdeb31779fa6d0074b98d75"} Oct 29 23:15:02.956699 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:02.956676 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-462fb2-6999659b56-gh9vv" Oct 29 23:15:02.985741 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:02.985450 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-575d59-29888595-x6kgd" podStartSLOduration=2.985432244 podStartE2EDuration="2.985432244s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:15:02.981704497 +0000 UTC m=+2483.764917923" watchObservedRunningTime="2026-10-29 23:15:02.985432244 +0000 UTC m=+2483.768645671" Oct 29 23:15:03.157482 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:03.157395 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-b8f5c4-29888595-bmqxn" podStartSLOduration=3.157382568 podStartE2EDuration="3.157382568s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:15:03.151986751 +0000 UTC m=+2483.935200178" watchObservedRunningTime="2026-10-29 23:15:03.157382568 +0000 UTC m=+2483.940595994" Oct 29 23:15:03.234218 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:03.234161 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-063b46-29888595-2hmbf" podStartSLOduration=3.234148966 podStartE2EDuration="3.234148966s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:15:03.229700178 +0000 UTC m=+2484.012913604" watchObservedRunningTime="2026-10-29 23:15:03.234148966 +0000 UTC m=+2484.017362391" Oct 29 23:15:03.272455 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:03.272397 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-462fb2-6999659b56-gh9vv" podStartSLOduration=2.272377808 podStartE2EDuration="2.272377808s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:15:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:15:03.27078155 +0000 UTC m=+2484.053994978" watchObservedRunningTime="2026-10-29 23:15:03.272377808 +0000 UTC m=+2484.055591249" Oct 29 23:15:03.316879 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:03.316832 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e131af-29888595-9njpr" podStartSLOduration=3.3168147660000002 podStartE2EDuration="3.316814766s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:15:03.31155392 +0000 UTC m=+2484.094767345" watchObservedRunningTime="2026-10-29 23:15:03.316814766 +0000 UTC m=+2484.100028188" Oct 29 23:15:03.970400 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:03.970359 5057 generic.go:356] "Generic (PLEG): container finished" podID="653090c9-6e06-4d91-9192-66c79775eded" containerID="54e2f4e5ac88abbb453deee2eea03fed5c08b73f99ae7e8b4e7da2a5dc40c11b" exitCode=60 Oct 29 23:15:03.970881 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:03.970470 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-063b46-29888595-2hmbf" event={"ID":"653090c9-6e06-4d91-9192-66c79775eded","Type":"ContainerDied","Data":"54e2f4e5ac88abbb453deee2eea03fed5c08b73f99ae7e8b4e7da2a5dc40c11b"} Oct 29 23:15:03.970881 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:03.970832 5057 scope.go:122] "RemoveContainer" containerID="54e2f4e5ac88abbb453deee2eea03fed5c08b73f99ae7e8b4e7da2a5dc40c11b" Oct 29 23:15:03.985964 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:03.985937 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-c880c7-29888595-rmnjw" event={"ID":"709fa6cf-8954-48fc-ac8a-b893fb663bd9","Type":"ContainerStarted","Data":"33c5c5114bab248b053ab6b234d242dce3de079681464b4f9824e111df5ac891"} Oct 29 23:15:03.988738 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:03.988710 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4504f1-29888595-7s2j5" event={"ID":"1dc9bc0e-b528-42cb-86b6-5a4790e3a0c3","Type":"ContainerStarted","Data":"4366129ce2b9dbfa0141c968a47e268065c5db6d0319cbb15a07cbf18f0f5f1d"} Oct 29 23:15:03.992614 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:03.992587 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-fba83e-29888595-8wtvz" event={"ID":"56b2d3d2-3c8f-4b6f-a32a-e882dc660fa4","Type":"ContainerStarted","Data":"4536b5650526a949996e5882f989c57f5da2409ed93c7f44aa669078a6188613"} Oct 29 23:15:04.002570 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:04.002547 5057 generic.go:356] "Generic (PLEG): container finished" podID="d0e370f0-45f8-4b62-aa7d-52414efa778e" containerID="654e3516979e2e1ac47f4c7e9e9ce63876d458f7f426577484dde128a3166dc5" exitCode=60 Oct 29 23:15:04.002693 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:04.002670 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e131af-29888595-9njpr" event={"ID":"d0e370f0-45f8-4b62-aa7d-52414efa778e","Type":"ContainerDied","Data":"654e3516979e2e1ac47f4c7e9e9ce63876d458f7f426577484dde128a3166dc5"} Oct 29 23:15:04.002964 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:04.002947 5057 scope.go:122] "RemoveContainer" containerID="654e3516979e2e1ac47f4c7e9e9ce63876d458f7f426577484dde128a3166dc5" Oct 29 23:15:04.005066 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:04.004986 5057 generic.go:356] "Generic (PLEG): container finished" podID="dae1295f-4869-46d2-8cfb-0d89f2f3309c" containerID="4b33442ad88b7ad27a0a6dadf6eb4f1cfc18e4d85a0e881b2ab1582ad2c0cf42" exitCode=60 Oct 29 23:15:04.005128 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:04.005090 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-b8f5c4-29888595-bmqxn" event={"ID":"dae1295f-4869-46d2-8cfb-0d89f2f3309c","Type":"ContainerDied","Data":"4b33442ad88b7ad27a0a6dadf6eb4f1cfc18e4d85a0e881b2ab1582ad2c0cf42"} Oct 29 23:15:04.005318 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:04.005300 5057 scope.go:122] "RemoveContainer" containerID="4b33442ad88b7ad27a0a6dadf6eb4f1cfc18e4d85a0e881b2ab1582ad2c0cf42" Oct 29 23:15:04.013822 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:04.013801 5057 generic.go:356] "Generic (PLEG): container finished" podID="4d649d28-2651-4a9a-b73e-05d3407743e5" containerID="a0703019c28ae0f6d8aeae2cf7b0f09fd3ab43fd94446cc0c3882f63e1359354" exitCode=60 Oct 29 23:15:04.014019 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:04.013988 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-575d59-29888595-x6kgd" event={"ID":"4d649d28-2651-4a9a-b73e-05d3407743e5","Type":"ContainerDied","Data":"a0703019c28ae0f6d8aeae2cf7b0f09fd3ab43fd94446cc0c3882f63e1359354"} Oct 29 23:15:04.014185 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:04.014168 5057 scope.go:122] "RemoveContainer" containerID="a0703019c28ae0f6d8aeae2cf7b0f09fd3ab43fd94446cc0c3882f63e1359354" Oct 29 23:15:04.017814 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:04.017790 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-af237d-29888595-zkzf9" event={"ID":"a4b8088b-affb-4734-8c8f-72797838f689","Type":"ContainerStarted","Data":"bdbd92d1f886698011da4776ee5e38083c94343224fe2cdfa096ca6821f0b55a"} Oct 29 23:15:04.146752 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:04.146699 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4504f1-29888595-7s2j5" podStartSLOduration=4.14668684 podStartE2EDuration="4.14668684s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:15:04.146281559 +0000 UTC m=+2484.929494986" watchObservedRunningTime="2026-10-29 23:15:04.14668684 +0000 UTC m=+2484.929900266" Oct 29 23:15:04.344686 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:04.344651 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:15:05.032928 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:05.032896 5057 generic.go:356] "Generic (PLEG): container finished" podID="a4b8088b-affb-4734-8c8f-72797838f689" containerID="bdbd92d1f886698011da4776ee5e38083c94343224fe2cdfa096ca6821f0b55a" exitCode=60 Oct 29 23:15:05.033451 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:05.032978 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-af237d-29888595-zkzf9" event={"ID":"a4b8088b-affb-4734-8c8f-72797838f689","Type":"ContainerDied","Data":"bdbd92d1f886698011da4776ee5e38083c94343224fe2cdfa096ca6821f0b55a"} Oct 29 23:15:05.033451 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:05.033085 5057 scope.go:122] "RemoveContainer" containerID="162b4e39603d1735830dbcead0df45b9d4cbcd3c79e2ad507010dba2ce1f0c94" Oct 29 23:15:05.033601 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:05.033542 5057 scope.go:122] "RemoveContainer" containerID="bdbd92d1f886698011da4776ee5e38083c94343224fe2cdfa096ca6821f0b55a" Oct 29 23:15:05.033993 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:15:05.033971 5057 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-af237d-29888595-zkzf9_odh-ai-gateway-infra(a4b8088b-affb-4734-8c8f-72797838f689)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-af237d-29888595-zkzf9" podUID="a4b8088b-affb-4734-8c8f-72797838f689" Oct 29 23:15:05.035876 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:05.035840 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-063b46-29888595-2hmbf" event={"ID":"653090c9-6e06-4d91-9192-66c79775eded","Type":"ContainerStarted","Data":"e8e8c19f5abfeffd39553372e503d286e4609ee202d08fe0b91fd1b515316288"} Oct 29 23:15:05.038008 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:05.037987 5057 generic.go:356] "Generic (PLEG): container finished" podID="709fa6cf-8954-48fc-ac8a-b893fb663bd9" containerID="33c5c5114bab248b053ab6b234d242dce3de079681464b4f9824e111df5ac891" exitCode=60 Oct 29 23:15:05.038107 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:05.038067 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-c880c7-29888595-rmnjw" event={"ID":"709fa6cf-8954-48fc-ac8a-b893fb663bd9","Type":"ContainerDied","Data":"33c5c5114bab248b053ab6b234d242dce3de079681464b4f9824e111df5ac891"} Oct 29 23:15:05.038409 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:05.038374 5057 scope.go:122] "RemoveContainer" containerID="33c5c5114bab248b053ab6b234d242dce3de079681464b4f9824e111df5ac891" Oct 29 23:15:05.038835 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:15:05.038812 5057 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w5-c880c7-29888595-rmnjw_odh-ai-gateway-infra(709fa6cf-8954-48fc-ac8a-b893fb663bd9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-c880c7-29888595-rmnjw" podUID="709fa6cf-8954-48fc-ac8a-b893fb663bd9" Oct 29 23:15:05.040187 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:05.040169 5057 generic.go:356] "Generic (PLEG): container finished" podID="1dc9bc0e-b528-42cb-86b6-5a4790e3a0c3" containerID="4366129ce2b9dbfa0141c968a47e268065c5db6d0319cbb15a07cbf18f0f5f1d" exitCode=60 Oct 29 23:15:05.040267 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:05.040238 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4504f1-29888595-7s2j5" event={"ID":"1dc9bc0e-b528-42cb-86b6-5a4790e3a0c3","Type":"ContainerDied","Data":"4366129ce2b9dbfa0141c968a47e268065c5db6d0319cbb15a07cbf18f0f5f1d"} Oct 29 23:15:05.040497 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:05.040483 5057 scope.go:122] "RemoveContainer" containerID="4366129ce2b9dbfa0141c968a47e268065c5db6d0319cbb15a07cbf18f0f5f1d" Oct 29 23:15:05.040860 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:15:05.040831 5057 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-4504f1-29888595-7s2j5_odh-ai-gateway-infra(1dc9bc0e-b528-42cb-86b6-5a4790e3a0c3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4504f1-29888595-7s2j5" podUID="1dc9bc0e-b528-42cb-86b6-5a4790e3a0c3" Oct 29 23:15:05.042532 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:05.042510 5057 generic.go:356] "Generic (PLEG): container finished" podID="56b2d3d2-3c8f-4b6f-a32a-e882dc660fa4" containerID="4536b5650526a949996e5882f989c57f5da2409ed93c7f44aa669078a6188613" exitCode=60 Oct 29 23:15:05.042703 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:05.042619 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-fba83e-29888595-8wtvz" event={"ID":"56b2d3d2-3c8f-4b6f-a32a-e882dc660fa4","Type":"ContainerDied","Data":"4536b5650526a949996e5882f989c57f5da2409ed93c7f44aa669078a6188613"} Oct 29 23:15:05.042879 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:05.042857 5057 scope.go:122] "RemoveContainer" containerID="4536b5650526a949996e5882f989c57f5da2409ed93c7f44aa669078a6188613" Oct 29 23:15:05.043257 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:15:05.043233 5057 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-fba83e-29888595-8wtvz_odh-ai-gateway-infra(56b2d3d2-3c8f-4b6f-a32a-e882dc660fa4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-fba83e-29888595-8wtvz" podUID="56b2d3d2-3c8f-4b6f-a32a-e882dc660fa4" Oct 29 23:15:05.045012 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:05.044968 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e131af-29888595-9njpr" event={"ID":"d0e370f0-45f8-4b62-aa7d-52414efa778e","Type":"ContainerStarted","Data":"78264a3a1b18b6a06c796130337e61d5339dec3d9b518ceb62df0970e2c0296a"} Oct 29 23:15:05.047199 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:05.047164 5057 generic.go:356] "Generic (PLEG): container finished" podID="84a5841f-8af1-4840-9019-a4ce56f7ed40" containerID="a24ccc14be1082fe0d99f9e8fde556e359349c0c345b6ac0e3fc7ac9b94c6e2b" exitCode=7 Oct 29 23:15:05.047280 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:05.047251 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-59eefa-29888595-jzqt5" event={"ID":"84a5841f-8af1-4840-9019-a4ce56f7ed40","Type":"ContainerDied","Data":"a24ccc14be1082fe0d99f9e8fde556e359349c0c345b6ac0e3fc7ac9b94c6e2b"} Oct 29 23:15:05.047571 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:05.047535 5057 scope.go:122] "RemoveContainer" containerID="a24ccc14be1082fe0d99f9e8fde556e359349c0c345b6ac0e3fc7ac9b94c6e2b" Oct 29 23:15:05.047946 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:15:05.047919 5057 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-59eefa-29888595-jzqt5_odh-ai-gateway-infra(84a5841f-8af1-4840-9019-a4ce56f7ed40)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-59eefa-29888595-jzqt5" podUID="84a5841f-8af1-4840-9019-a4ce56f7ed40" Oct 29 23:15:05.049483 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:05.049462 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-b8f5c4-29888595-bmqxn" event={"ID":"dae1295f-4869-46d2-8cfb-0d89f2f3309c","Type":"ContainerStarted","Data":"f23f74aec8e0d400ef28dcc53833539a8945c1530db73c988dc39bef918fa11d"} Oct 29 23:15:05.051164 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:05.051150 5057 scope.go:122] "RemoveContainer" containerID="a13134cd0fdc98c584aac76fb7dee1cd95ae83a57b9c63f6598cad7eb2de23c7" Oct 29 23:15:05.051341 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:05.051314 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-575d59-29888595-x6kgd" event={"ID":"4d649d28-2651-4a9a-b73e-05d3407743e5","Type":"ContainerStarted","Data":"be204b81f8e589335ce9417b31cfad83a5551e87c402b50be31b6d62648c2430"} Oct 29 23:15:05.084149 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:05.084071 5057 scope.go:122] "RemoveContainer" containerID="ddd459a209873277145e64c2df566e4023c7e159bd84d0f1dd5f8313a99c4a5f" Oct 29 23:15:05.106172 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:05.106119 5057 scope.go:122] "RemoveContainer" containerID="85738245ffa423ae0d44e52461e3cceca82ed35dcc5350abd3ec8495f6e7a14b" Oct 29 23:15:05.126222 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:05.126180 5057 scope.go:122] "RemoveContainer" containerID="6882e917d5fdfb8d7952beacbcb8d84f17a4b3e8f8299df71d5c877d81d309b6" Oct 29 23:15:06.069818 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:06.069780 5057 generic.go:356] "Generic (PLEG): container finished" podID="653090c9-6e06-4d91-9192-66c79775eded" containerID="e8e8c19f5abfeffd39553372e503d286e4609ee202d08fe0b91fd1b515316288" exitCode=60 Oct 29 23:15:06.070436 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:06.070402 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-063b46-29888595-2hmbf" event={"ID":"653090c9-6e06-4d91-9192-66c79775eded","Type":"ContainerDied","Data":"e8e8c19f5abfeffd39553372e503d286e4609ee202d08fe0b91fd1b515316288"} Oct 29 23:15:06.070622 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:06.070606 5057 scope.go:122] "RemoveContainer" containerID="54e2f4e5ac88abbb453deee2eea03fed5c08b73f99ae7e8b4e7da2a5dc40c11b" Oct 29 23:15:06.071317 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:06.071294 5057 scope.go:122] "RemoveContainer" containerID="e8e8c19f5abfeffd39553372e503d286e4609ee202d08fe0b91fd1b515316288" Oct 29 23:15:06.071953 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:15:06.071924 5057 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-063b46-29888595-2hmbf_odh-ai-gateway-infra(653090c9-6e06-4d91-9192-66c79775eded)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-063b46-29888595-2hmbf" podUID="653090c9-6e06-4d91-9192-66c79775eded" Oct 29 23:15:06.083264 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:06.083237 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:15:06.083694 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:06.083664 5057 scope.go:122] "RemoveContainer" containerID="33c5c5114bab248b053ab6b234d242dce3de079681464b4f9824e111df5ac891" Oct 29 23:15:06.084062 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:15:06.084029 5057 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w5-c880c7-29888595-rmnjw_odh-ai-gateway-infra(709fa6cf-8954-48fc-ac8a-b893fb663bd9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-c880c7-29888595-rmnjw" podUID="709fa6cf-8954-48fc-ac8a-b893fb663bd9" Oct 29 23:15:06.087590 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:06.087566 5057 scope.go:122] "RemoveContainer" containerID="4366129ce2b9dbfa0141c968a47e268065c5db6d0319cbb15a07cbf18f0f5f1d" Oct 29 23:15:06.088310 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:15:06.087974 5057 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-4504f1-29888595-7s2j5_odh-ai-gateway-infra(1dc9bc0e-b528-42cb-86b6-5a4790e3a0c3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4504f1-29888595-7s2j5" podUID="1dc9bc0e-b528-42cb-86b6-5a4790e3a0c3" Oct 29 23:15:06.089687 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:06.089657 5057 scope.go:122] "RemoveContainer" containerID="4536b5650526a949996e5882f989c57f5da2409ed93c7f44aa669078a6188613" Oct 29 23:15:06.089991 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:15:06.089966 5057 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-fba83e-29888595-8wtvz_odh-ai-gateway-infra(56b2d3d2-3c8f-4b6f-a32a-e882dc660fa4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-fba83e-29888595-8wtvz" podUID="56b2d3d2-3c8f-4b6f-a32a-e882dc660fa4" Oct 29 23:15:06.092007 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:06.091982 5057 generic.go:356] "Generic (PLEG): container finished" podID="d0e370f0-45f8-4b62-aa7d-52414efa778e" containerID="78264a3a1b18b6a06c796130337e61d5339dec3d9b518ceb62df0970e2c0296a" exitCode=60 Oct 29 23:15:06.092089 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:06.092076 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e131af-29888595-9njpr" event={"ID":"d0e370f0-45f8-4b62-aa7d-52414efa778e","Type":"ContainerDied","Data":"78264a3a1b18b6a06c796130337e61d5339dec3d9b518ceb62df0970e2c0296a"} Oct 29 23:15:06.092463 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:06.092442 5057 scope.go:122] "RemoveContainer" containerID="78264a3a1b18b6a06c796130337e61d5339dec3d9b518ceb62df0970e2c0296a" Oct 29 23:15:06.092776 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:15:06.092751 5057 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-e131af-29888595-9njpr_odh-ai-gateway-infra(d0e370f0-45f8-4b62-aa7d-52414efa778e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e131af-29888595-9njpr" podUID="d0e370f0-45f8-4b62-aa7d-52414efa778e" Oct 29 23:15:06.094584 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:06.094566 5057 scope.go:122] "RemoveContainer" containerID="a24ccc14be1082fe0d99f9e8fde556e359349c0c345b6ac0e3fc7ac9b94c6e2b" Oct 29 23:15:06.094886 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:15:06.094818 5057 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-59eefa-29888595-jzqt5_odh-ai-gateway-infra(84a5841f-8af1-4840-9019-a4ce56f7ed40)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-59eefa-29888595-jzqt5" podUID="84a5841f-8af1-4840-9019-a4ce56f7ed40" Oct 29 23:15:06.096660 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:06.096639 5057 generic.go:356] "Generic (PLEG): container finished" podID="dae1295f-4869-46d2-8cfb-0d89f2f3309c" containerID="f23f74aec8e0d400ef28dcc53833539a8945c1530db73c988dc39bef918fa11d" exitCode=60 Oct 29 23:15:06.096780 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:06.096740 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-b8f5c4-29888595-bmqxn" event={"ID":"dae1295f-4869-46d2-8cfb-0d89f2f3309c","Type":"ContainerDied","Data":"f23f74aec8e0d400ef28dcc53833539a8945c1530db73c988dc39bef918fa11d"} Oct 29 23:15:06.097026 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:06.096977 5057 scope.go:122] "RemoveContainer" containerID="f23f74aec8e0d400ef28dcc53833539a8945c1530db73c988dc39bef918fa11d" Oct 29 23:15:06.097325 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:15:06.097297 5057 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w6-b8f5c4-29888595-bmqxn_odh-ai-gateway-infra(dae1295f-4869-46d2-8cfb-0d89f2f3309c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-b8f5c4-29888595-bmqxn" podUID="dae1295f-4869-46d2-8cfb-0d89f2f3309c" Oct 29 23:15:06.104266 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:06.104243 5057 generic.go:356] "Generic (PLEG): container finished" podID="4d649d28-2651-4a9a-b73e-05d3407743e5" containerID="be204b81f8e589335ce9417b31cfad83a5551e87c402b50be31b6d62648c2430" exitCode=60 Oct 29 23:15:06.104426 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:06.104370 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-575d59-29888595-x6kgd" event={"ID":"4d649d28-2651-4a9a-b73e-05d3407743e5","Type":"ContainerDied","Data":"be204b81f8e589335ce9417b31cfad83a5551e87c402b50be31b6d62648c2430"} Oct 29 23:15:06.104722 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:06.104703 5057 scope.go:122] "RemoveContainer" containerID="be204b81f8e589335ce9417b31cfad83a5551e87c402b50be31b6d62648c2430" Oct 29 23:15:06.105026 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:15:06.104997 5057 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-575d59-29888595-x6kgd_odh-ai-gateway-infra(4d649d28-2651-4a9a-b73e-05d3407743e5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-575d59-29888595-x6kgd" podUID="4d649d28-2651-4a9a-b73e-05d3407743e5" Oct 29 23:15:06.105520 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:06.105487 5057 scope.go:122] "RemoveContainer" containerID="654e3516979e2e1ac47f4c7e9e9ce63876d458f7f426577484dde128a3166dc5" Oct 29 23:15:06.107447 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:06.107426 5057 scope.go:122] "RemoveContainer" containerID="bdbd92d1f886698011da4776ee5e38083c94343224fe2cdfa096ca6821f0b55a" Oct 29 23:15:06.107757 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:15:06.107731 5057 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-af237d-29888595-zkzf9_odh-ai-gateway-infra(a4b8088b-affb-4734-8c8f-72797838f689)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-af237d-29888595-zkzf9" podUID="a4b8088b-affb-4734-8c8f-72797838f689" Oct 29 23:15:06.152110 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:06.152085 5057 scope.go:122] "RemoveContainer" containerID="4b33442ad88b7ad27a0a6dadf6eb4f1cfc18e4d85a0e881b2ab1582ad2c0cf42" Oct 29 23:15:06.193617 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:06.186436 5057 scope.go:122] "RemoveContainer" containerID="a0703019c28ae0f6d8aeae2cf7b0f09fd3ab43fd94446cc0c3882f63e1359354" Oct 29 23:15:07.121655 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:07.121624 5057 scope.go:122] "RemoveContainer" containerID="e8e8c19f5abfeffd39553372e503d286e4609ee202d08fe0b91fd1b515316288" Oct 29 23:15:07.122121 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:15:07.122062 5057 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-063b46-29888595-2hmbf_odh-ai-gateway-infra(653090c9-6e06-4d91-9192-66c79775eded)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-063b46-29888595-2hmbf" podUID="653090c9-6e06-4d91-9192-66c79775eded" Oct 29 23:15:07.123530 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:07.123508 5057 scope.go:122] "RemoveContainer" containerID="78264a3a1b18b6a06c796130337e61d5339dec3d9b518ceb62df0970e2c0296a" Oct 29 23:15:07.123930 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:15:07.123899 5057 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-e131af-29888595-9njpr_odh-ai-gateway-infra(d0e370f0-45f8-4b62-aa7d-52414efa778e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e131af-29888595-9njpr" podUID="d0e370f0-45f8-4b62-aa7d-52414efa778e" Oct 29 23:15:07.125999 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:07.125978 5057 scope.go:122] "RemoveContainer" containerID="f23f74aec8e0d400ef28dcc53833539a8945c1530db73c988dc39bef918fa11d" Oct 29 23:15:07.126326 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:15:07.126307 5057 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w6-b8f5c4-29888595-bmqxn_odh-ai-gateway-infra(dae1295f-4869-46d2-8cfb-0d89f2f3309c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-b8f5c4-29888595-bmqxn" podUID="dae1295f-4869-46d2-8cfb-0d89f2f3309c" Oct 29 23:15:07.127760 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:07.127741 5057 scope.go:122] "RemoveContainer" containerID="be204b81f8e589335ce9417b31cfad83a5551e87c402b50be31b6d62648c2430" Oct 29 23:15:07.128036 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:15:07.128017 5057 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-575d59-29888595-x6kgd_odh-ai-gateway-infra(4d649d28-2651-4a9a-b73e-05d3407743e5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-575d59-29888595-x6kgd" podUID="4d649d28-2651-4a9a-b73e-05d3407743e5" Oct 29 23:15:07.897754 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:07.897715 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:15:09.732197 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:09.732157 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:15:11.439801 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:11.439767 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:15:13.219568 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:13.219524 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:15:14.022822 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:14.022786 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-462fb2-6999659b56-gh9vv" Oct 29 23:15:14.879921 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:14.879880 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:15:15.320532 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:15.320503 5057 scope.go:122] "RemoveContainer" containerID="bdbd92d1f886698011da4776ee5e38083c94343224fe2cdfa096ca6821f0b55a" Oct 29 23:15:15.320734 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:15.320695 5057 scope.go:122] "RemoveContainer" containerID="33c5c5114bab248b053ab6b234d242dce3de079681464b4f9824e111df5ac891" Oct 29 23:15:16.222648 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:16.222609 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-af237d-29888595-zkzf9" event={"ID":"a4b8088b-affb-4734-8c8f-72797838f689","Type":"ContainerStarted","Data":"fcb55bc7677a2e5745677206f83a579672ab4e3e1f7d7b3277c31e3c50b1a4b8"} Oct 29 23:15:16.225176 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:16.225146 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-c880c7-29888595-rmnjw" event={"ID":"709fa6cf-8954-48fc-ac8a-b893fb663bd9","Type":"ContainerStarted","Data":"cbeda32246c9c493e6ed98645f392dbe07d7cea993be24fc8720502a3bd4a023"} Oct 29 23:15:16.320986 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:16.320957 5057 scope.go:122] "RemoveContainer" containerID="4366129ce2b9dbfa0141c968a47e268065c5db6d0319cbb15a07cbf18f0f5f1d" Oct 29 23:15:16.399059 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:16.399021 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-4504f1/auth-test-model-4504f1-kserve-7694bcddfc-dt6vg"] Oct 29 23:15:16.441589 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:16.441552 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:15:16.600352 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:16.596453 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:15:17.239318 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:17.239279 5057 generic.go:356] "Generic (PLEG): container finished" podID="709fa6cf-8954-48fc-ac8a-b893fb663bd9" containerID="cbeda32246c9c493e6ed98645f392dbe07d7cea993be24fc8720502a3bd4a023" exitCode=60 Oct 29 23:15:17.239935 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:17.239381 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-c880c7-29888595-rmnjw" event={"ID":"709fa6cf-8954-48fc-ac8a-b893fb663bd9","Type":"ContainerDied","Data":"cbeda32246c9c493e6ed98645f392dbe07d7cea993be24fc8720502a3bd4a023"} Oct 29 23:15:17.239935 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:17.239519 5057 scope.go:122] "RemoveContainer" containerID="33c5c5114bab248b053ab6b234d242dce3de079681464b4f9824e111df5ac891" Oct 29 23:15:17.240230 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:17.240208 5057 scope.go:122] "RemoveContainer" containerID="cbeda32246c9c493e6ed98645f392dbe07d7cea993be24fc8720502a3bd4a023" Oct 29 23:15:17.240923 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:15:17.240671 5057 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w5-c880c7-29888595-rmnjw_odh-ai-gateway-infra(709fa6cf-8954-48fc-ac8a-b893fb663bd9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-c880c7-29888595-rmnjw" podUID="709fa6cf-8954-48fc-ac8a-b893fb663bd9" Oct 29 23:15:17.243429 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:17.243400 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4504f1-29888595-7s2j5" event={"ID":"1dc9bc0e-b528-42cb-86b6-5a4790e3a0c3","Type":"ContainerStarted","Data":"4a8c4fc4417e6008a1b472b33341a8d7fbc81bd1f7675bdbe0debd88bd7aa7f8"} Oct 29 23:15:17.252912 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:17.252614 5057 generic.go:356] "Generic (PLEG): container finished" podID="a4b8088b-affb-4734-8c8f-72797838f689" containerID="fcb55bc7677a2e5745677206f83a579672ab4e3e1f7d7b3277c31e3c50b1a4b8" exitCode=60 Oct 29 23:15:17.252912 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:17.252683 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-af237d-29888595-zkzf9" event={"ID":"a4b8088b-affb-4734-8c8f-72797838f689","Type":"ContainerDied","Data":"fcb55bc7677a2e5745677206f83a579672ab4e3e1f7d7b3277c31e3c50b1a4b8"} Oct 29 23:15:17.253810 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:17.253788 5057 scope.go:122] "RemoveContainer" containerID="fcb55bc7677a2e5745677206f83a579672ab4e3e1f7d7b3277c31e3c50b1a4b8" Oct 29 23:15:17.254207 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:15:17.254184 5057 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-af237d-29888595-zkzf9_odh-ai-gateway-infra(a4b8088b-affb-4734-8c8f-72797838f689)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-af237d-29888595-zkzf9" podUID="a4b8088b-affb-4734-8c8f-72797838f689" Oct 29 23:15:17.263524 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:17.263503 5057 scope.go:122] "RemoveContainer" containerID="bdbd92d1f886698011da4776ee5e38083c94343224fe2cdfa096ca6821f0b55a" Oct 29 23:15:17.277641 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:17.277611 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-af237d-29888595-zkzf9"] Oct 29 23:15:17.320601 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:17.320566 5057 scope.go:122] "RemoveContainer" containerID="4536b5650526a949996e5882f989c57f5da2409ed93c7f44aa669078a6188613" Oct 29 23:15:17.346580 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:17.346550 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-c880c7-29888595-rmnjw"] Oct 29 23:15:18.211258 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:18.211222 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:15:18.306541 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:18.301376 5057 generic.go:356] "Generic (PLEG): container finished" podID="1dc9bc0e-b528-42cb-86b6-5a4790e3a0c3" containerID="4a8c4fc4417e6008a1b472b33341a8d7fbc81bd1f7675bdbe0debd88bd7aa7f8" exitCode=60 Oct 29 23:15:18.306541 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:18.301475 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4504f1-29888595-7s2j5" event={"ID":"1dc9bc0e-b528-42cb-86b6-5a4790e3a0c3","Type":"ContainerDied","Data":"4a8c4fc4417e6008a1b472b33341a8d7fbc81bd1f7675bdbe0debd88bd7aa7f8"} Oct 29 23:15:18.306541 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:18.301535 5057 scope.go:122] "RemoveContainer" containerID="4366129ce2b9dbfa0141c968a47e268065c5db6d0319cbb15a07cbf18f0f5f1d" Oct 29 23:15:18.306541 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:18.302139 5057 scope.go:122] "RemoveContainer" containerID="4a8c4fc4417e6008a1b472b33341a8d7fbc81bd1f7675bdbe0debd88bd7aa7f8" Oct 29 23:15:18.306541 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:15:18.302649 5057 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-4504f1-29888595-7s2j5_odh-ai-gateway-infra(1dc9bc0e-b528-42cb-86b6-5a4790e3a0c3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4504f1-29888595-7s2j5" podUID="1dc9bc0e-b528-42cb-86b6-5a4790e3a0c3" Oct 29 23:15:18.309718 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:18.309030 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-fba83e-29888595-8wtvz" event={"ID":"56b2d3d2-3c8f-4b6f-a32a-e882dc660fa4","Type":"ContainerStarted","Data":"81716a85c2c4481397355db8c1fc54703c33cf3d327833461adf891af0bf8c63"} Oct 29 23:15:18.330425 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:18.320202 5057 scope.go:122] "RemoveContainer" containerID="be204b81f8e589335ce9417b31cfad83a5551e87c402b50be31b6d62648c2430" Oct 29 23:15:18.330425 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:18.321223 5057 scope.go:122] "RemoveContainer" containerID="a24ccc14be1082fe0d99f9e8fde556e359349c0c345b6ac0e3fc7ac9b94c6e2b" Oct 29 23:15:18.404684 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:18.404270 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4504f1-29888595-7s2j5"] Oct 29 23:15:18.796709 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:18.796684 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-af237d-29888595-zkzf9" Oct 29 23:15:18.799631 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:18.799611 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-c880c7-29888595-rmnjw" Oct 29 23:15:18.866934 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:18.866902 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/709fa6cf-8954-48fc-ac8a-b893fb663bd9-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/709fa6cf-8954-48fc-ac8a-b893fb663bd9-maas-api-service-ca\") pod \"709fa6cf-8954-48fc-ac8a-b893fb663bd9\" (UID: \"709fa6cf-8954-48fc-ac8a-b893fb663bd9\") " Oct 29 23:15:18.867240 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:18.867215 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4b8088b-affb-4734-8c8f-72797838f689-kube-api-access-sz9dk\" (UniqueName: \"kubernetes.io/projected/a4b8088b-affb-4734-8c8f-72797838f689-kube-api-access-sz9dk\") pod \"a4b8088b-affb-4734-8c8f-72797838f689\" (UID: \"a4b8088b-affb-4734-8c8f-72797838f689\") " Oct 29 23:15:18.867383 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:18.867285 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/709fa6cf-8954-48fc-ac8a-b893fb663bd9-maas-api-service-ca" pod "709fa6cf-8954-48fc-ac8a-b893fb663bd9" (UID: "709fa6cf-8954-48fc-ac8a-b893fb663bd9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:15:18.867383 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:18.867318 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/709fa6cf-8954-48fc-ac8a-b893fb663bd9-kube-api-access-7t9hm\" (UniqueName: \"kubernetes.io/projected/709fa6cf-8954-48fc-ac8a-b893fb663bd9-kube-api-access-7t9hm\") pod \"709fa6cf-8954-48fc-ac8a-b893fb663bd9\" (UID: \"709fa6cf-8954-48fc-ac8a-b893fb663bd9\") " Oct 29 23:15:18.867383 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:18.867367 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a4b8088b-affb-4734-8c8f-72797838f689-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4b8088b-affb-4734-8c8f-72797838f689-maas-api-service-ca\") pod \"a4b8088b-affb-4734-8c8f-72797838f689\" (UID: \"a4b8088b-affb-4734-8c8f-72797838f689\") " Oct 29 23:15:18.867758 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:18.867732 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4b8088b-affb-4734-8c8f-72797838f689-maas-api-service-ca" pod "a4b8088b-affb-4734-8c8f-72797838f689" (UID: "a4b8088b-affb-4734-8c8f-72797838f689"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:15:18.868338 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:18.868290 5057 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4b8088b-affb-4734-8c8f-72797838f689-maas-api-service-ca\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:15:18.868439 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:18.868323 5057 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/709fa6cf-8954-48fc-ac8a-b893fb663bd9-maas-api-service-ca\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:15:18.870069 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:18.870039 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4b8088b-affb-4734-8c8f-72797838f689-kube-api-access-sz9dk" pod "a4b8088b-affb-4734-8c8f-72797838f689" (UID: "a4b8088b-affb-4734-8c8f-72797838f689"). InnerVolumeSpecName "kube-api-access-sz9dk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:15:18.870183 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:18.870158 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/709fa6cf-8954-48fc-ac8a-b893fb663bd9-kube-api-access-7t9hm" pod "709fa6cf-8954-48fc-ac8a-b893fb663bd9" (UID: "709fa6cf-8954-48fc-ac8a-b893fb663bd9"). InnerVolumeSpecName "kube-api-access-7t9hm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:15:18.970029 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:18.969992 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sz9dk\" (UniqueName: \"kubernetes.io/projected/a4b8088b-affb-4734-8c8f-72797838f689-kube-api-access-sz9dk\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:15:18.970029 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:18.970029 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7t9hm\" (UniqueName: \"kubernetes.io/projected/709fa6cf-8954-48fc-ac8a-b893fb663bd9-kube-api-access-7t9hm\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:15:19.030940 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:15:19.030909 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d2dnn" containerName="extractor" Oct 29 23:15:19.331245 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:19.330958 5057 scope.go:122] "RemoveContainer" containerID="f23f74aec8e0d400ef28dcc53833539a8945c1530db73c988dc39bef918fa11d" Oct 29 23:15:19.331245 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:19.331106 5057 scope.go:122] "RemoveContainer" containerID="e8e8c19f5abfeffd39553372e503d286e4609ee202d08fe0b91fd1b515316288" Oct 29 23:15:19.336661 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:19.336630 5057 generic.go:356] "Generic (PLEG): container finished" podID="56b2d3d2-3c8f-4b6f-a32a-e882dc660fa4" containerID="81716a85c2c4481397355db8c1fc54703c33cf3d327833461adf891af0bf8c63" exitCode=60 Oct 29 23:15:19.337531 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:19.337504 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-fba83e-29888595-8wtvz" event={"ID":"56b2d3d2-3c8f-4b6f-a32a-e882dc660fa4","Type":"ContainerDied","Data":"81716a85c2c4481397355db8c1fc54703c33cf3d327833461adf891af0bf8c63"} Oct 29 23:15:19.337653 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:19.337564 5057 scope.go:122] "RemoveContainer" containerID="4536b5650526a949996e5882f989c57f5da2409ed93c7f44aa669078a6188613" Oct 29 23:15:19.337964 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:19.337941 5057 scope.go:122] "RemoveContainer" containerID="81716a85c2c4481397355db8c1fc54703c33cf3d327833461adf891af0bf8c63" Oct 29 23:15:19.338293 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:15:19.338233 5057 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-fba83e-29888595-8wtvz_odh-ai-gateway-infra(56b2d3d2-3c8f-4b6f-a32a-e882dc660fa4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-fba83e-29888595-8wtvz" podUID="56b2d3d2-3c8f-4b6f-a32a-e882dc660fa4" Oct 29 23:15:19.340267 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:19.340234 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-59eefa-29888595-jzqt5" event={"ID":"84a5841f-8af1-4840-9019-a4ce56f7ed40","Type":"ContainerStarted","Data":"6edf405b0256dea38c194f72b48a85f7a9e87b9c589a582d639026af92117581"} Oct 29 23:15:19.343257 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:19.343231 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-575d59-29888595-x6kgd" event={"ID":"4d649d28-2651-4a9a-b73e-05d3407743e5","Type":"ContainerStarted","Data":"068701df9ec0e0686bb8bfa0422aa676bdab942229de8457c7e8052ff83757f1"} Oct 29 23:15:19.344872 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:19.344851 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-af237d-29888595-zkzf9" Oct 29 23:15:19.344988 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:19.344897 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-af237d-29888595-zkzf9" event={"ID":"a4b8088b-affb-4734-8c8f-72797838f689","Type":"ContainerDied","Data":"ff10e3debbb7697024e324b377500d8e2fd98fccb4ec89900052406aabe4cdf9"} Oct 29 23:15:19.346777 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:19.346653 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-c880c7-29888595-rmnjw" event={"ID":"709fa6cf-8954-48fc-ac8a-b893fb663bd9","Type":"ContainerDied","Data":"83f41edf2d27b54d356a13fac0d747a169e8a6b9de77d0bb3103aa7219e7761c"} Oct 29 23:15:19.346777 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:19.346732 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-c880c7-29888595-rmnjw" Oct 29 23:15:19.375154 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:19.375114 5057 scope.go:122] "RemoveContainer" containerID="fcb55bc7677a2e5745677206f83a579672ab4e3e1f7d7b3277c31e3c50b1a4b8" Oct 29 23:15:19.412790 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:19.412761 5057 scope.go:122] "RemoveContainer" containerID="cbeda32246c9c493e6ed98645f392dbe07d7cea993be24fc8720502a3bd4a023" Oct 29 23:15:19.476016 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:19.475878 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-fba83e-29888595-8wtvz"] Oct 29 23:15:19.479870 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:19.478612 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-59eefa-29888595-jzqt5" podStartSLOduration=19.478583779 podStartE2EDuration="19.478583779s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:15:19.474851299 +0000 UTC m=+2500.258064720" watchObservedRunningTime="2026-10-29 23:15:19.478583779 +0000 UTC m=+2500.261797207" Oct 29 23:15:19.568889 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:19.568857 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-af237d-29888595-zkzf9"] Oct 29 23:15:19.585600 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:19.585182 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-af237d-29888595-zkzf9"] Oct 29 23:15:19.622665 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:19.616050 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-c880c7-29888595-rmnjw"] Oct 29 23:15:19.627844 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:19.627808 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-c880c7-29888595-rmnjw"] Oct 29 23:15:19.655793 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:19.655767 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4504f1-29888595-7s2j5" Oct 29 23:15:19.808010 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:19.807954 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1dc9bc0e-b528-42cb-86b6-5a4790e3a0c3-kube-api-access-nxcsh\" (UniqueName: \"kubernetes.io/projected/1dc9bc0e-b528-42cb-86b6-5a4790e3a0c3-kube-api-access-nxcsh\") pod \"1dc9bc0e-b528-42cb-86b6-5a4790e3a0c3\" (UID: \"1dc9bc0e-b528-42cb-86b6-5a4790e3a0c3\") " Oct 29 23:15:19.808155 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:19.808138 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1dc9bc0e-b528-42cb-86b6-5a4790e3a0c3-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1dc9bc0e-b528-42cb-86b6-5a4790e3a0c3-maas-api-service-ca\") pod \"1dc9bc0e-b528-42cb-86b6-5a4790e3a0c3\" (UID: \"1dc9bc0e-b528-42cb-86b6-5a4790e3a0c3\") " Oct 29 23:15:19.808559 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:19.808531 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1dc9bc0e-b528-42cb-86b6-5a4790e3a0c3-maas-api-service-ca" pod "1dc9bc0e-b528-42cb-86b6-5a4790e3a0c3" (UID: "1dc9bc0e-b528-42cb-86b6-5a4790e3a0c3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:15:19.809072 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:19.809050 5057 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1dc9bc0e-b528-42cb-86b6-5a4790e3a0c3-maas-api-service-ca\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:15:19.810500 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:19.810472 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1dc9bc0e-b528-42cb-86b6-5a4790e3a0c3-kube-api-access-nxcsh" pod "1dc9bc0e-b528-42cb-86b6-5a4790e3a0c3" (UID: "1dc9bc0e-b528-42cb-86b6-5a4790e3a0c3"). InnerVolumeSpecName "kube-api-access-nxcsh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:15:19.911686 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:19.911635 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nxcsh\" (UniqueName: \"kubernetes.io/projected/1dc9bc0e-b528-42cb-86b6-5a4790e3a0c3-kube-api-access-nxcsh\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:15:20.110702 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:20.110596 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:15:20.320929 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:20.320896 5057 scope.go:122] "RemoveContainer" containerID="78264a3a1b18b6a06c796130337e61d5339dec3d9b518ceb62df0970e2c0296a" Oct 29 23:15:20.363000 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:20.362931 5057 generic.go:356] "Generic (PLEG): container finished" podID="4d649d28-2651-4a9a-b73e-05d3407743e5" containerID="068701df9ec0e0686bb8bfa0422aa676bdab942229de8457c7e8052ff83757f1" exitCode=60 Oct 29 23:15:20.363410 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:20.363005 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-575d59-29888595-x6kgd" event={"ID":"4d649d28-2651-4a9a-b73e-05d3407743e5","Type":"ContainerDied","Data":"068701df9ec0e0686bb8bfa0422aa676bdab942229de8457c7e8052ff83757f1"} Oct 29 23:15:20.363410 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:20.363060 5057 scope.go:122] "RemoveContainer" containerID="be204b81f8e589335ce9417b31cfad83a5551e87c402b50be31b6d62648c2430" Oct 29 23:15:20.363595 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:20.363493 5057 scope.go:122] "RemoveContainer" containerID="068701df9ec0e0686bb8bfa0422aa676bdab942229de8457c7e8052ff83757f1" Oct 29 23:15:20.364012 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:15:20.363981 5057 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-575d59-29888595-x6kgd_odh-ai-gateway-infra(4d649d28-2651-4a9a-b73e-05d3407743e5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-575d59-29888595-x6kgd" podUID="4d649d28-2651-4a9a-b73e-05d3407743e5" Oct 29 23:15:20.367382 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:20.367354 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-063b46-29888595-2hmbf" event={"ID":"653090c9-6e06-4d91-9192-66c79775eded","Type":"ContainerStarted","Data":"d5d822b8e93a255aa60ad9050d06fa93c5cca1b07eeac001d9b061df5b6d4323"} Oct 29 23:15:20.370137 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:20.370102 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4504f1-29888595-7s2j5" event={"ID":"1dc9bc0e-b528-42cb-86b6-5a4790e3a0c3","Type":"ContainerDied","Data":"3589ca23afbf99ecedfea4ebfc6ef1d0005e6641472b83a2a9df56144bb7a1fd"} Oct 29 23:15:20.370276 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:20.370109 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4504f1-29888595-7s2j5" Oct 29 23:15:20.375119 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:20.375091 5057 generic.go:356] "Generic (PLEG): container finished" podID="84a5841f-8af1-4840-9019-a4ce56f7ed40" containerID="6edf405b0256dea38c194f72b48a85f7a9e87b9c589a582d639026af92117581" exitCode=60 Oct 29 23:15:20.375300 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:20.375184 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-59eefa-29888595-jzqt5" event={"ID":"84a5841f-8af1-4840-9019-a4ce56f7ed40","Type":"ContainerDied","Data":"6edf405b0256dea38c194f72b48a85f7a9e87b9c589a582d639026af92117581"} Oct 29 23:15:20.375564 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:20.375545 5057 scope.go:122] "RemoveContainer" containerID="6edf405b0256dea38c194f72b48a85f7a9e87b9c589a582d639026af92117581" Oct 29 23:15:20.375825 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:15:20.375804 5057 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-59eefa-29888595-jzqt5_odh-ai-gateway-infra(84a5841f-8af1-4840-9019-a4ce56f7ed40)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-59eefa-29888595-jzqt5" podUID="84a5841f-8af1-4840-9019-a4ce56f7ed40" Oct 29 23:15:20.383766 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:20.382890 5057 scope.go:122] "RemoveContainer" containerID="4a8c4fc4417e6008a1b472b33341a8d7fbc81bd1f7675bdbe0debd88bd7aa7f8" Oct 29 23:15:20.384416 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:20.384255 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-b8f5c4-29888595-bmqxn" event={"ID":"dae1295f-4869-46d2-8cfb-0d89f2f3309c","Type":"ContainerStarted","Data":"b7ca2d3cc164c83e131398bff05a2f1500f7799d000ee8af496f7717a608f516"} Oct 29 23:15:20.396991 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:20.396971 5057 scope.go:122] "RemoveContainer" containerID="a24ccc14be1082fe0d99f9e8fde556e359349c0c345b6ac0e3fc7ac9b94c6e2b" Oct 29 23:15:20.446424 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:20.446386 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-575d59-29888595-x6kgd"] Oct 29 23:15:20.492042 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:20.492006 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4504f1-29888595-7s2j5"] Oct 29 23:15:20.500811 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:20.500775 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4504f1-29888595-7s2j5"] Oct 29 23:15:20.510908 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:20.510863 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-59eefa-29888595-jzqt5"] Oct 29 23:15:20.908949 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:20.908921 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-fba83e-29888595-8wtvz" Oct 29 23:15:20.933687 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:20.933618 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56b2d3d2-3c8f-4b6f-a32a-e882dc660fa4-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/56b2d3d2-3c8f-4b6f-a32a-e882dc660fa4-maas-api-service-ca\") pod \"56b2d3d2-3c8f-4b6f-a32a-e882dc660fa4\" (UID: \"56b2d3d2-3c8f-4b6f-a32a-e882dc660fa4\") " Oct 29 23:15:20.933870 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:20.933848 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56b2d3d2-3c8f-4b6f-a32a-e882dc660fa4-kube-api-access-pk86q\" (UniqueName: \"kubernetes.io/projected/56b2d3d2-3c8f-4b6f-a32a-e882dc660fa4-kube-api-access-pk86q\") pod \"56b2d3d2-3c8f-4b6f-a32a-e882dc660fa4\" (UID: \"56b2d3d2-3c8f-4b6f-a32a-e882dc660fa4\") " Oct 29 23:15:20.934012 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:20.933990 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56b2d3d2-3c8f-4b6f-a32a-e882dc660fa4-maas-api-service-ca" pod "56b2d3d2-3c8f-4b6f-a32a-e882dc660fa4" (UID: "56b2d3d2-3c8f-4b6f-a32a-e882dc660fa4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:15:20.934553 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:20.934534 5057 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/56b2d3d2-3c8f-4b6f-a32a-e882dc660fa4-maas-api-service-ca\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:15:20.936061 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:20.936031 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56b2d3d2-3c8f-4b6f-a32a-e882dc660fa4-kube-api-access-pk86q" pod "56b2d3d2-3c8f-4b6f-a32a-e882dc660fa4" (UID: "56b2d3d2-3c8f-4b6f-a32a-e882dc660fa4"). InnerVolumeSpecName "kube-api-access-pk86q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:15:21.036794 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:21.036754 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pk86q\" (UniqueName: \"kubernetes.io/projected/56b2d3d2-3c8f-4b6f-a32a-e882dc660fa4-kube-api-access-pk86q\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:15:21.345240 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:21.345207 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1dc9bc0e-b528-42cb-86b6-5a4790e3a0c3" path="/var/lib/kubelet/pods/1dc9bc0e-b528-42cb-86b6-5a4790e3a0c3/volumes" Oct 29 23:15:21.345728 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:21.345706 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="709fa6cf-8954-48fc-ac8a-b893fb663bd9" path="/var/lib/kubelet/pods/709fa6cf-8954-48fc-ac8a-b893fb663bd9/volumes" Oct 29 23:15:21.346196 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:21.346180 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4b8088b-affb-4734-8c8f-72797838f689" path="/var/lib/kubelet/pods/a4b8088b-affb-4734-8c8f-72797838f689/volumes" Oct 29 23:15:21.401222 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:21.401186 5057 generic.go:356] "Generic (PLEG): container finished" podID="653090c9-6e06-4d91-9192-66c79775eded" containerID="d5d822b8e93a255aa60ad9050d06fa93c5cca1b07eeac001d9b061df5b6d4323" exitCode=60 Oct 29 23:15:21.401723 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:21.401252 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-063b46-29888595-2hmbf" event={"ID":"653090c9-6e06-4d91-9192-66c79775eded","Type":"ContainerDied","Data":"d5d822b8e93a255aa60ad9050d06fa93c5cca1b07eeac001d9b061df5b6d4323"} Oct 29 23:15:21.401723 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:21.401314 5057 scope.go:122] "RemoveContainer" containerID="e8e8c19f5abfeffd39553372e503d286e4609ee202d08fe0b91fd1b515316288" Oct 29 23:15:21.401840 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:21.401814 5057 scope.go:122] "RemoveContainer" containerID="d5d822b8e93a255aa60ad9050d06fa93c5cca1b07eeac001d9b061df5b6d4323" Oct 29 23:15:21.402251 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:15:21.402220 5057 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-063b46-29888595-2hmbf_odh-ai-gateway-infra(653090c9-6e06-4d91-9192-66c79775eded)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-063b46-29888595-2hmbf" podUID="653090c9-6e06-4d91-9192-66c79775eded" Oct 29 23:15:21.423796 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:21.423777 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-fba83e-29888595-8wtvz" Oct 29 23:15:21.424257 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:21.423793 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-fba83e-29888595-8wtvz" event={"ID":"56b2d3d2-3c8f-4b6f-a32a-e882dc660fa4","Type":"ContainerDied","Data":"ff6a76c04e414351bb5efa2c1243ed83159dc5cd89f0770ab83e8676a639f30d"} Oct 29 23:15:21.424355 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:21.424319 5057 scope.go:122] "RemoveContainer" containerID="81716a85c2c4481397355db8c1fc54703c33cf3d327833461adf891af0bf8c63" Oct 29 23:15:21.427384 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:21.427316 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e131af-29888595-9njpr" event={"ID":"d0e370f0-45f8-4b62-aa7d-52414efa778e","Type":"ContainerStarted","Data":"cc9440111fb9415f3f77f4107a23c30d99431ddcc6e80b03c1a64483cb3b0cc7"} Oct 29 23:15:21.432125 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:21.432103 5057 generic.go:356] "Generic (PLEG): container finished" podID="dae1295f-4869-46d2-8cfb-0d89f2f3309c" containerID="b7ca2d3cc164c83e131398bff05a2f1500f7799d000ee8af496f7717a608f516" exitCode=60 Oct 29 23:15:21.432227 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:21.432170 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-b8f5c4-29888595-bmqxn" event={"ID":"dae1295f-4869-46d2-8cfb-0d89f2f3309c","Type":"ContainerDied","Data":"b7ca2d3cc164c83e131398bff05a2f1500f7799d000ee8af496f7717a608f516"} Oct 29 23:15:21.432723 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:21.432678 5057 scope.go:122] "RemoveContainer" containerID="b7ca2d3cc164c83e131398bff05a2f1500f7799d000ee8af496f7717a608f516" Oct 29 23:15:21.433519 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:15:21.433418 5057 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w6-b8f5c4-29888595-bmqxn_odh-ai-gateway-infra(dae1295f-4869-46d2-8cfb-0d89f2f3309c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-b8f5c4-29888595-bmqxn" podUID="dae1295f-4869-46d2-8cfb-0d89f2f3309c" Oct 29 23:15:21.444519 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:21.444498 5057 scope.go:122] "RemoveContainer" containerID="f23f74aec8e0d400ef28dcc53833539a8945c1530db73c988dc39bef918fa11d" Oct 29 23:15:21.549386 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:21.542893 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-b8f5c4-29888595-bmqxn"] Oct 29 23:15:21.631784 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:21.631748 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-063b46-29888595-2hmbf"] Oct 29 23:15:21.713647 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:21.713462 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-fba83e-29888595-8wtvz"] Oct 29 23:15:21.719977 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:21.719942 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-fba83e-29888595-8wtvz"] Oct 29 23:15:21.813607 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:21.813572 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-af237d/auth-test-model-af237d-kserve-7444475bdf-5tw7n"] Oct 29 23:15:21.821214 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:21.821178 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:15:21.852231 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:21.852203 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-59eefa-29888595-jzqt5" Oct 29 23:15:21.857898 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:21.854428 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-575d59-29888595-x6kgd" Oct 29 23:15:22.041555 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:22.021159 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:15:22.155477 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:22.155322 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d649d28-2651-4a9a-b73e-05d3407743e5-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d649d28-2651-4a9a-b73e-05d3407743e5-maas-api-service-ca\") pod \"4d649d28-2651-4a9a-b73e-05d3407743e5\" (UID: \"4d649d28-2651-4a9a-b73e-05d3407743e5\") " Oct 29 23:15:22.155650 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:22.155603 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84a5841f-8af1-4840-9019-a4ce56f7ed40-kube-api-access-rqbqq\" (UniqueName: \"kubernetes.io/projected/84a5841f-8af1-4840-9019-a4ce56f7ed40-kube-api-access-rqbqq\") pod \"84a5841f-8af1-4840-9019-a4ce56f7ed40\" (UID: \"84a5841f-8af1-4840-9019-a4ce56f7ed40\") " Oct 29 23:15:22.155748 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:22.155726 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84a5841f-8af1-4840-9019-a4ce56f7ed40-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/84a5841f-8af1-4840-9019-a4ce56f7ed40-maas-api-service-ca\") pod \"84a5841f-8af1-4840-9019-a4ce56f7ed40\" (UID: \"84a5841f-8af1-4840-9019-a4ce56f7ed40\") " Oct 29 23:15:22.156009 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:22.155987 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d649d28-2651-4a9a-b73e-05d3407743e5-kube-api-access-hftj8\" (UniqueName: \"kubernetes.io/projected/4d649d28-2651-4a9a-b73e-05d3407743e5-kube-api-access-hftj8\") pod \"4d649d28-2651-4a9a-b73e-05d3407743e5\" (UID: \"4d649d28-2651-4a9a-b73e-05d3407743e5\") " Oct 29 23:15:22.156089 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:22.156001 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d649d28-2651-4a9a-b73e-05d3407743e5-maas-api-service-ca" pod "4d649d28-2651-4a9a-b73e-05d3407743e5" (UID: "4d649d28-2651-4a9a-b73e-05d3407743e5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:15:22.157256 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:22.157226 5057 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d649d28-2651-4a9a-b73e-05d3407743e5-maas-api-service-ca\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:15:22.157661 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:22.157636 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84a5841f-8af1-4840-9019-a4ce56f7ed40-maas-api-service-ca" pod "84a5841f-8af1-4840-9019-a4ce56f7ed40" (UID: "84a5841f-8af1-4840-9019-a4ce56f7ed40"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:15:22.175239 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:22.175042 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84a5841f-8af1-4840-9019-a4ce56f7ed40-kube-api-access-rqbqq" pod "84a5841f-8af1-4840-9019-a4ce56f7ed40" (UID: "84a5841f-8af1-4840-9019-a4ce56f7ed40"). InnerVolumeSpecName "kube-api-access-rqbqq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:15:22.181275 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:22.181245 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d649d28-2651-4a9a-b73e-05d3407743e5-kube-api-access-hftj8" pod "4d649d28-2651-4a9a-b73e-05d3407743e5" (UID: "4d649d28-2651-4a9a-b73e-05d3407743e5"). InnerVolumeSpecName "kube-api-access-hftj8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:15:22.259015 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:22.258981 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hftj8\" (UniqueName: \"kubernetes.io/projected/4d649d28-2651-4a9a-b73e-05d3407743e5-kube-api-access-hftj8\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:15:22.259215 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:22.259200 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rqbqq\" (UniqueName: \"kubernetes.io/projected/84a5841f-8af1-4840-9019-a4ce56f7ed40-kube-api-access-rqbqq\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:15:22.259326 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:22.259314 5057 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/84a5841f-8af1-4840-9019-a4ce56f7ed40-maas-api-service-ca\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:15:22.371804 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:22.371766 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:15:22.530103 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:22.528472 5057 generic.go:356] "Generic (PLEG): container finished" podID="d0e370f0-45f8-4b62-aa7d-52414efa778e" containerID="cc9440111fb9415f3f77f4107a23c30d99431ddcc6e80b03c1a64483cb3b0cc7" exitCode=60 Oct 29 23:15:22.530103 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:22.528579 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e131af-29888595-9njpr" event={"ID":"d0e370f0-45f8-4b62-aa7d-52414efa778e","Type":"ContainerDied","Data":"cc9440111fb9415f3f77f4107a23c30d99431ddcc6e80b03c1a64483cb3b0cc7"} Oct 29 23:15:22.530103 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:22.528638 5057 scope.go:122] "RemoveContainer" containerID="78264a3a1b18b6a06c796130337e61d5339dec3d9b518ceb62df0970e2c0296a" Oct 29 23:15:22.530103 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:22.529240 5057 scope.go:122] "RemoveContainer" containerID="cc9440111fb9415f3f77f4107a23c30d99431ddcc6e80b03c1a64483cb3b0cc7" Oct 29 23:15:22.530103 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:15:22.529755 5057 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-e131af-29888595-9njpr_odh-ai-gateway-infra(d0e370f0-45f8-4b62-aa7d-52414efa778e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e131af-29888595-9njpr" podUID="d0e370f0-45f8-4b62-aa7d-52414efa778e" Oct 29 23:15:22.535631 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:22.535602 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-59eefa-29888595-jzqt5" event={"ID":"84a5841f-8af1-4840-9019-a4ce56f7ed40","Type":"ContainerDied","Data":"63f1edb68ca64a8df362c1b904cced87ff52602cda9633a0bcde55acb08bda15"} Oct 29 23:15:22.536399 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:22.535601 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-59eefa-29888595-jzqt5" Oct 29 23:15:22.544264 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:22.544237 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-575d59-29888595-x6kgd" event={"ID":"4d649d28-2651-4a9a-b73e-05d3407743e5","Type":"ContainerDied","Data":"fb6849c24410c7fddcf3485f654c8334a62b05e5a18c36a3a16efa9a747324ec"} Oct 29 23:15:22.544531 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:22.544513 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-575d59-29888595-x6kgd" Oct 29 23:15:22.633345 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:22.627432 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e131af-29888595-9njpr"] Oct 29 23:15:22.655428 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:22.653094 5057 scope.go:122] "RemoveContainer" containerID="6edf405b0256dea38c194f72b48a85f7a9e87b9c589a582d639026af92117581" Oct 29 23:15:22.726389 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:22.725273 5057 scope.go:122] "RemoveContainer" containerID="068701df9ec0e0686bb8bfa0422aa676bdab942229de8457c7e8052ff83757f1" Oct 29 23:15:22.779489 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:22.779412 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-575d59-29888595-x6kgd"] Oct 29 23:15:22.781109 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:22.781081 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-575d59-29888595-x6kgd"] Oct 29 23:15:22.836984 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:22.836749 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:15:22.845353 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:22.842525 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-59eefa-29888595-jzqt5"] Oct 29 23:15:22.849633 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:22.848729 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-59eefa-29888595-jzqt5"] Oct 29 23:15:22.921840 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:22.921253 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-063b46-29888595-2hmbf" Oct 29 23:15:22.995634 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:22.995601 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/653090c9-6e06-4d91-9192-66c79775eded-kube-api-access-4rn9x\" (UniqueName: \"kubernetes.io/projected/653090c9-6e06-4d91-9192-66c79775eded-kube-api-access-4rn9x\") pod \"653090c9-6e06-4d91-9192-66c79775eded\" (UID: \"653090c9-6e06-4d91-9192-66c79775eded\") " Oct 29 23:15:22.997153 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:22.997119 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/653090c9-6e06-4d91-9192-66c79775eded-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/653090c9-6e06-4d91-9192-66c79775eded-maas-api-service-ca\") pod \"653090c9-6e06-4d91-9192-66c79775eded\" (UID: \"653090c9-6e06-4d91-9192-66c79775eded\") " Oct 29 23:15:22.997841 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:22.997817 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/653090c9-6e06-4d91-9192-66c79775eded-maas-api-service-ca" pod "653090c9-6e06-4d91-9192-66c79775eded" (UID: "653090c9-6e06-4d91-9192-66c79775eded"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:15:23.000407 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:23.000216 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/653090c9-6e06-4d91-9192-66c79775eded-kube-api-access-4rn9x" pod "653090c9-6e06-4d91-9192-66c79775eded" (UID: "653090c9-6e06-4d91-9192-66c79775eded"). InnerVolumeSpecName "kube-api-access-4rn9x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:15:23.000513 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:23.000402 5057 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/653090c9-6e06-4d91-9192-66c79775eded-maas-api-service-ca\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:15:23.000513 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:23.000425 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4rn9x\" (UniqueName: \"kubernetes.io/projected/653090c9-6e06-4d91-9192-66c79775eded-kube-api-access-4rn9x\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:15:23.011853 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:23.011832 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-b8f5c4-29888595-bmqxn" Oct 29 23:15:23.101745 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:23.101671 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dae1295f-4869-46d2-8cfb-0d89f2f3309c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dae1295f-4869-46d2-8cfb-0d89f2f3309c-maas-api-service-ca\") pod \"dae1295f-4869-46d2-8cfb-0d89f2f3309c\" (UID: \"dae1295f-4869-46d2-8cfb-0d89f2f3309c\") " Oct 29 23:15:23.101902 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:23.101754 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dae1295f-4869-46d2-8cfb-0d89f2f3309c-kube-api-access-cvcw5\" (UniqueName: \"kubernetes.io/projected/dae1295f-4869-46d2-8cfb-0d89f2f3309c-kube-api-access-cvcw5\") pod \"dae1295f-4869-46d2-8cfb-0d89f2f3309c\" (UID: \"dae1295f-4869-46d2-8cfb-0d89f2f3309c\") " Oct 29 23:15:23.102722 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:23.102697 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dae1295f-4869-46d2-8cfb-0d89f2f3309c-maas-api-service-ca" pod "dae1295f-4869-46d2-8cfb-0d89f2f3309c" (UID: "dae1295f-4869-46d2-8cfb-0d89f2f3309c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:15:23.104446 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:23.104415 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dae1295f-4869-46d2-8cfb-0d89f2f3309c-kube-api-access-cvcw5" pod "dae1295f-4869-46d2-8cfb-0d89f2f3309c" (UID: "dae1295f-4869-46d2-8cfb-0d89f2f3309c"). InnerVolumeSpecName "kube-api-access-cvcw5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:15:23.204391 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:23.204305 5057 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dae1295f-4869-46d2-8cfb-0d89f2f3309c-maas-api-service-ca\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:15:23.204391 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:23.204352 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cvcw5\" (UniqueName: \"kubernetes.io/projected/dae1295f-4869-46d2-8cfb-0d89f2f3309c-kube-api-access-cvcw5\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:15:23.518352 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:23.495325 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d649d28-2651-4a9a-b73e-05d3407743e5" path="/var/lib/kubelet/pods/4d649d28-2651-4a9a-b73e-05d3407743e5/volumes" Oct 29 23:15:23.518352 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:23.495949 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56b2d3d2-3c8f-4b6f-a32a-e882dc660fa4" path="/var/lib/kubelet/pods/56b2d3d2-3c8f-4b6f-a32a-e882dc660fa4/volumes" Oct 29 23:15:23.520542 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:23.518749 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84a5841f-8af1-4840-9019-a4ce56f7ed40" path="/var/lib/kubelet/pods/84a5841f-8af1-4840-9019-a4ce56f7ed40/volumes" Oct 29 23:15:23.571798 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:23.571769 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-b8f5c4-29888595-bmqxn" event={"ID":"dae1295f-4869-46d2-8cfb-0d89f2f3309c","Type":"ContainerDied","Data":"4ab65a1857bd5af26ebf8af2821579d95fc2b882b7cab78f4e3fe5c1f9f3fd26"} Oct 29 23:15:23.572098 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:23.571827 5057 scope.go:122] "RemoveContainer" containerID="b7ca2d3cc164c83e131398bff05a2f1500f7799d000ee8af496f7717a608f516" Oct 29 23:15:23.572098 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:23.571920 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-b8f5c4-29888595-bmqxn" Oct 29 23:15:23.576661 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:23.576585 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-063b46-29888595-2hmbf" event={"ID":"653090c9-6e06-4d91-9192-66c79775eded","Type":"ContainerDied","Data":"7073be3d0379895d5250f681ad1d70acaf7f633f65f42539af234e04baee5c31"} Oct 29 23:15:23.576783 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:23.576674 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-063b46-29888595-2hmbf" Oct 29 23:15:23.607266 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:23.606280 5057 scope.go:122] "RemoveContainer" containerID="d5d822b8e93a255aa60ad9050d06fa93c5cca1b07eeac001d9b061df5b6d4323" Oct 29 23:15:23.648610 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:23.648582 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-063b46-29888595-2hmbf"] Oct 29 23:15:23.653948 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:23.653924 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-063b46-29888595-2hmbf"] Oct 29 23:15:23.694125 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:23.691990 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-b8f5c4-29888595-bmqxn"] Oct 29 23:15:23.701808 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:23.701359 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-b8f5c4-29888595-bmqxn"] Oct 29 23:15:23.828484 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:23.828460 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e131af-29888595-9njpr" Oct 29 23:15:23.900206 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:23.898899 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:15:23.992916 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:23.992254 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0e370f0-45f8-4b62-aa7d-52414efa778e-kube-api-access-57s8w\" (UniqueName: \"kubernetes.io/projected/d0e370f0-45f8-4b62-aa7d-52414efa778e-kube-api-access-57s8w\") pod \"d0e370f0-45f8-4b62-aa7d-52414efa778e\" (UID: \"d0e370f0-45f8-4b62-aa7d-52414efa778e\") " Oct 29 23:15:23.993150 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:23.993099 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0e370f0-45f8-4b62-aa7d-52414efa778e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0e370f0-45f8-4b62-aa7d-52414efa778e-maas-api-service-ca\") pod \"d0e370f0-45f8-4b62-aa7d-52414efa778e\" (UID: \"d0e370f0-45f8-4b62-aa7d-52414efa778e\") " Oct 29 23:15:23.996877 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:23.993767 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0e370f0-45f8-4b62-aa7d-52414efa778e-maas-api-service-ca" pod "d0e370f0-45f8-4b62-aa7d-52414efa778e" (UID: "d0e370f0-45f8-4b62-aa7d-52414efa778e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:15:23.996877 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:23.995804 5057 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0e370f0-45f8-4b62-aa7d-52414efa778e-maas-api-service-ca\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:15:24.006727 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:23.998463 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0e370f0-45f8-4b62-aa7d-52414efa778e-kube-api-access-57s8w" pod "d0e370f0-45f8-4b62-aa7d-52414efa778e" (UID: "d0e370f0-45f8-4b62-aa7d-52414efa778e"). InnerVolumeSpecName "kube-api-access-57s8w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:15:24.097641 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:24.097549 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-57s8w\" (UniqueName: \"kubernetes.io/projected/d0e370f0-45f8-4b62-aa7d-52414efa778e-kube-api-access-57s8w\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:15:24.210352 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:24.209993 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:15:24.418148 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:24.418062 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:15:24.588650 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:24.588617 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e131af-29888595-9njpr" Oct 29 23:15:24.588650 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:24.588633 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e131af-29888595-9njpr" event={"ID":"d0e370f0-45f8-4b62-aa7d-52414efa778e","Type":"ContainerDied","Data":"b607c1c14f717f92d0425072e2ad5140f46462bf17f5620e92ddec5a1d392e98"} Oct 29 23:15:24.589277 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:24.588708 5057 scope.go:122] "RemoveContainer" containerID="cc9440111fb9415f3f77f4107a23c30d99431ddcc6e80b03c1a64483cb3b0cc7" Oct 29 23:15:24.615185 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:24.615154 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-67363b-openshift-default-745d499d87-z5d9k"] Oct 29 23:15:24.616714 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:24.616691 5057 cpu_manager.go:414] "containerMap: removing container" podUID="709fa6cf-8954-48fc-ac8a-b893fb663bd9" containerName="cleanup" Oct 29 23:15:24.616828 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:24.616715 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="709fa6cf-8954-48fc-ac8a-b893fb663bd9" containerName="cleanup" Oct 29 23:15:24.616828 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:24.616731 5057 cpu_manager.go:414] "containerMap: removing container" podUID="56b2d3d2-3c8f-4b6f-a32a-e882dc660fa4" containerName="cleanup" Oct 29 23:15:24.616828 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:24.616738 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56b2d3d2-3c8f-4b6f-a32a-e882dc660fa4" containerName="cleanup" Oct 29 23:15:24.616828 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:24.616750 5057 cpu_manager.go:414] "containerMap: removing container" podUID="653090c9-6e06-4d91-9192-66c79775eded" containerName="cleanup" Oct 29 23:15:24.616828 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:24.616759 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="653090c9-6e06-4d91-9192-66c79775eded" containerName="cleanup" Oct 29 23:15:24.616828 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:24.616776 5057 cpu_manager.go:414] "containerMap: removing container" podUID="84a5841f-8af1-4840-9019-a4ce56f7ed40" containerName="cleanup" Oct 29 23:15:24.616828 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:24.616783 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84a5841f-8af1-4840-9019-a4ce56f7ed40" containerName="cleanup" Oct 29 23:15:24.616828 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:24.616799 5057 cpu_manager.go:414] "containerMap: removing container" podUID="56b2d3d2-3c8f-4b6f-a32a-e882dc660fa4" containerName="cleanup" Oct 29 23:15:24.616828 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:24.616807 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56b2d3d2-3c8f-4b6f-a32a-e882dc660fa4" containerName="cleanup" Oct 29 23:15:24.616828 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:24.616820 5057 cpu_manager.go:414] "containerMap: removing container" podUID="d0e370f0-45f8-4b62-aa7d-52414efa778e" containerName="cleanup" Oct 29 23:15:24.616828 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:24.616827 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0e370f0-45f8-4b62-aa7d-52414efa778e" containerName="cleanup" Oct 29 23:15:24.617239 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:24.616863 5057 cpu_manager.go:414] "containerMap: removing container" podUID="d0e370f0-45f8-4b62-aa7d-52414efa778e" containerName="cleanup" Oct 29 23:15:24.617239 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:24.616871 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0e370f0-45f8-4b62-aa7d-52414efa778e" containerName="cleanup" Oct 29 23:15:24.617239 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:24.616883 5057 cpu_manager.go:414] "containerMap: removing container" podUID="dae1295f-4869-46d2-8cfb-0d89f2f3309c" containerName="cleanup" Oct 29 23:15:24.617239 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:24.616890 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dae1295f-4869-46d2-8cfb-0d89f2f3309c" containerName="cleanup" Oct 29 23:15:24.617239 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:24.616909 5057 cpu_manager.go:414] "containerMap: removing container" podUID="4d649d28-2651-4a9a-b73e-05d3407743e5" containerName="cleanup" Oct 29 23:15:24.617239 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:24.616916 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d649d28-2651-4a9a-b73e-05d3407743e5" containerName="cleanup" Oct 29 23:15:24.617239 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:24.616933 5057 cpu_manager.go:414] "containerMap: removing container" podUID="a4b8088b-affb-4734-8c8f-72797838f689" containerName="cleanup" Oct 29 23:15:24.617239 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:24.616939 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4b8088b-affb-4734-8c8f-72797838f689" containerName="cleanup" Oct 29 23:15:24.617239 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:24.616954 5057 cpu_manager.go:414] "containerMap: removing container" podUID="1dc9bc0e-b528-42cb-86b6-5a4790e3a0c3" containerName="cleanup" Oct 29 23:15:24.617239 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:24.616961 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1dc9bc0e-b528-42cb-86b6-5a4790e3a0c3" containerName="cleanup" Oct 29 23:15:24.617239 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:24.616973 5057 cpu_manager.go:414] "containerMap: removing container" podUID="dae1295f-4869-46d2-8cfb-0d89f2f3309c" containerName="cleanup" Oct 29 23:15:24.617239 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:24.616981 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dae1295f-4869-46d2-8cfb-0d89f2f3309c" containerName="cleanup" Oct 29 23:15:24.617239 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:24.616994 5057 cpu_manager.go:414] "containerMap: removing container" podUID="84a5841f-8af1-4840-9019-a4ce56f7ed40" containerName="cleanup" Oct 29 23:15:24.617239 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:24.617000 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84a5841f-8af1-4840-9019-a4ce56f7ed40" containerName="cleanup" Oct 29 23:15:24.617239 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:24.617023 5057 cpu_manager.go:414] "containerMap: removing container" podUID="a4b8088b-affb-4734-8c8f-72797838f689" containerName="cleanup" Oct 29 23:15:24.617239 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:24.617030 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4b8088b-affb-4734-8c8f-72797838f689" containerName="cleanup" Oct 29 23:15:24.617239 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:24.617041 5057 cpu_manager.go:414] "containerMap: removing container" podUID="84a5841f-8af1-4840-9019-a4ce56f7ed40" containerName="cleanup" Oct 29 23:15:24.617239 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:24.617049 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84a5841f-8af1-4840-9019-a4ce56f7ed40" containerName="cleanup" Oct 29 23:15:24.617239 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:24.617065 5057 cpu_manager.go:414] "containerMap: removing container" podUID="653090c9-6e06-4d91-9192-66c79775eded" containerName="cleanup" Oct 29 23:15:24.617239 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:24.617072 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="653090c9-6e06-4d91-9192-66c79775eded" containerName="cleanup" Oct 29 23:15:24.617239 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:24.617086 5057 cpu_manager.go:414] "containerMap: removing container" podUID="a4b8088b-affb-4734-8c8f-72797838f689" containerName="cleanup" Oct 29 23:15:24.617239 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:24.617093 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4b8088b-affb-4734-8c8f-72797838f689" containerName="cleanup" Oct 29 23:15:24.617239 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:24.617106 5057 cpu_manager.go:414] "containerMap: removing container" podUID="709fa6cf-8954-48fc-ac8a-b893fb663bd9" containerName="cleanup" Oct 29 23:15:24.617239 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:24.617113 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="709fa6cf-8954-48fc-ac8a-b893fb663bd9" containerName="cleanup" Oct 29 23:15:24.617239 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:24.617140 5057 cpu_manager.go:414] "containerMap: removing container" podUID="653090c9-6e06-4d91-9192-66c79775eded" containerName="cleanup" Oct 29 23:15:24.617239 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:24.617147 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="653090c9-6e06-4d91-9192-66c79775eded" containerName="cleanup" Oct 29 23:15:24.617239 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:24.617159 5057 cpu_manager.go:414] "containerMap: removing container" podUID="1dc9bc0e-b528-42cb-86b6-5a4790e3a0c3" containerName="cleanup" Oct 29 23:15:24.617239 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:24.617166 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1dc9bc0e-b528-42cb-86b6-5a4790e3a0c3" containerName="cleanup" Oct 29 23:15:24.617239 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:24.617178 5057 cpu_manager.go:414] "containerMap: removing container" podUID="4d649d28-2651-4a9a-b73e-05d3407743e5" containerName="cleanup" Oct 29 23:15:24.617239 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:24.617185 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d649d28-2651-4a9a-b73e-05d3407743e5" containerName="cleanup" Oct 29 23:15:24.618450 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:24.617436 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="709fa6cf-8954-48fc-ac8a-b893fb663bd9" containerName="cleanup" Oct 29 23:15:24.618450 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:24.617450 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="56b2d3d2-3c8f-4b6f-a32a-e882dc660fa4" containerName="cleanup" Oct 29 23:15:24.618450 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:24.617465 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="653090c9-6e06-4d91-9192-66c79775eded" containerName="cleanup" Oct 29 23:15:24.618450 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:24.617475 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="84a5841f-8af1-4840-9019-a4ce56f7ed40" containerName="cleanup" Oct 29 23:15:24.618450 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:24.617484 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="56b2d3d2-3c8f-4b6f-a32a-e882dc660fa4" containerName="cleanup" Oct 29 23:15:24.618450 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:24.617492 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0e370f0-45f8-4b62-aa7d-52414efa778e" containerName="cleanup" Oct 29 23:15:24.618450 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:24.617504 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="709fa6cf-8954-48fc-ac8a-b893fb663bd9" containerName="cleanup" Oct 29 23:15:24.618450 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:24.617516 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="dae1295f-4869-46d2-8cfb-0d89f2f3309c" containerName="cleanup" Oct 29 23:15:24.618450 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:24.617525 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0e370f0-45f8-4b62-aa7d-52414efa778e" containerName="cleanup" Oct 29 23:15:24.618450 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:24.617534 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="dae1295f-4869-46d2-8cfb-0d89f2f3309c" containerName="cleanup" Oct 29 23:15:24.618450 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:24.617544 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d649d28-2651-4a9a-b73e-05d3407743e5" containerName="cleanup" Oct 29 23:15:24.618450 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:24.617554 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4b8088b-affb-4734-8c8f-72797838f689" containerName="cleanup" Oct 29 23:15:24.618450 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:24.617564 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d649d28-2651-4a9a-b73e-05d3407743e5" containerName="cleanup" Oct 29 23:15:24.618450 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:24.617572 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="1dc9bc0e-b528-42cb-86b6-5a4790e3a0c3" containerName="cleanup" Oct 29 23:15:24.618450 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:24.617583 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="84a5841f-8af1-4840-9019-a4ce56f7ed40" containerName="cleanup" Oct 29 23:15:24.618450 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:24.617596 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4b8088b-affb-4734-8c8f-72797838f689" containerName="cleanup" Oct 29 23:15:24.618450 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:24.617602 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="84a5841f-8af1-4840-9019-a4ce56f7ed40" containerName="cleanup" Oct 29 23:15:24.618450 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:24.617615 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4b8088b-affb-4734-8c8f-72797838f689" containerName="cleanup" Oct 29 23:15:24.618450 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:24.617624 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="1dc9bc0e-b528-42cb-86b6-5a4790e3a0c3" containerName="cleanup" Oct 29 23:15:24.618450 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:24.617636 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="653090c9-6e06-4d91-9192-66c79775eded" containerName="cleanup" Oct 29 23:15:24.618450 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:24.617645 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="1dc9bc0e-b528-42cb-86b6-5a4790e3a0c3" containerName="cleanup" Oct 29 23:15:24.618450 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:24.617653 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d649d28-2651-4a9a-b73e-05d3407743e5" containerName="cleanup" Oct 29 23:15:24.624388 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:24.624365 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-67363b-openshift-default-745d499d87-z5d9k" Oct 29 23:15:24.640612 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:24.640584 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w0-67363b-openshift-default-dockercfg-kr7r9\"" Oct 29 23:15:24.643159 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:24.643129 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-67363b-openshift-default-745d499d87-z5d9k"] Oct 29 23:15:24.650695 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:24.650675 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e131af-29888595-9njpr"] Oct 29 23:15:24.656000 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:24.655976 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e131af-29888595-9njpr"] Oct 29 23:15:24.717007 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:24.716966 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76be25ad-e827-41d0-975d-a6bf7a1dcdee-workload-certs\") pod \"e2e-shared-b-w0-67363b-openshift-default-745d499d87-z5d9k\" (UID: \"76be25ad-e827-41d0-975d-a6bf7a1dcdee\") " pod="openshift-ingress/e2e-shared-b-w0-67363b-openshift-default-745d499d87-z5d9k" Oct 29 23:15:24.717179 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:24.717020 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76be25ad-e827-41d0-975d-a6bf7a1dcdee-istio-podinfo\") pod \"e2e-shared-b-w0-67363b-openshift-default-745d499d87-z5d9k\" (UID: \"76be25ad-e827-41d0-975d-a6bf7a1dcdee\") " pod="openshift-ingress/e2e-shared-b-w0-67363b-openshift-default-745d499d87-z5d9k" Oct 29 23:15:24.717179 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:24.717146 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76be25ad-e827-41d0-975d-a6bf7a1dcdee-istio-token\") pod \"e2e-shared-b-w0-67363b-openshift-default-745d499d87-z5d9k\" (UID: \"76be25ad-e827-41d0-975d-a6bf7a1dcdee\") " pod="openshift-ingress/e2e-shared-b-w0-67363b-openshift-default-745d499d87-z5d9k" Oct 29 23:15:24.717310 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:24.717206 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76be25ad-e827-41d0-975d-a6bf7a1dcdee-istio-envoy\") pod \"e2e-shared-b-w0-67363b-openshift-default-745d499d87-z5d9k\" (UID: \"76be25ad-e827-41d0-975d-a6bf7a1dcdee\") " pod="openshift-ingress/e2e-shared-b-w0-67363b-openshift-default-745d499d87-z5d9k" Oct 29 23:15:24.717310 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:24.717256 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76be25ad-e827-41d0-975d-a6bf7a1dcdee-istiod-ca-cert\") pod \"e2e-shared-b-w0-67363b-openshift-default-745d499d87-z5d9k\" (UID: \"76be25ad-e827-41d0-975d-a6bf7a1dcdee\") " pod="openshift-ingress/e2e-shared-b-w0-67363b-openshift-default-745d499d87-z5d9k" Oct 29 23:15:24.717440 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:24.717421 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22llm\" (UniqueName: \"kubernetes.io/projected/76be25ad-e827-41d0-975d-a6bf7a1dcdee-kube-api-access-22llm\") pod \"e2e-shared-b-w0-67363b-openshift-default-745d499d87-z5d9k\" (UID: \"76be25ad-e827-41d0-975d-a6bf7a1dcdee\") " pod="openshift-ingress/e2e-shared-b-w0-67363b-openshift-default-745d499d87-z5d9k" Oct 29 23:15:24.717734 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:24.717702 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76be25ad-e827-41d0-975d-a6bf7a1dcdee-workload-socket\") pod \"e2e-shared-b-w0-67363b-openshift-default-745d499d87-z5d9k\" (UID: \"76be25ad-e827-41d0-975d-a6bf7a1dcdee\") " pod="openshift-ingress/e2e-shared-b-w0-67363b-openshift-default-745d499d87-z5d9k" Oct 29 23:15:24.717831 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:24.717812 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76be25ad-e827-41d0-975d-a6bf7a1dcdee-credential-socket\") pod \"e2e-shared-b-w0-67363b-openshift-default-745d499d87-z5d9k\" (UID: \"76be25ad-e827-41d0-975d-a6bf7a1dcdee\") " pod="openshift-ingress/e2e-shared-b-w0-67363b-openshift-default-745d499d87-z5d9k" Oct 29 23:15:24.717888 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:24.717845 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76be25ad-e827-41d0-975d-a6bf7a1dcdee-istio-data\") pod \"e2e-shared-b-w0-67363b-openshift-default-745d499d87-z5d9k\" (UID: \"76be25ad-e827-41d0-975d-a6bf7a1dcdee\") " pod="openshift-ingress/e2e-shared-b-w0-67363b-openshift-default-745d499d87-z5d9k" Oct 29 23:15:24.819300 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:24.819261 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76be25ad-e827-41d0-975d-a6bf7a1dcdee-workload-socket\") pod \"e2e-shared-b-w0-67363b-openshift-default-745d499d87-z5d9k\" (UID: \"76be25ad-e827-41d0-975d-a6bf7a1dcdee\") " pod="openshift-ingress/e2e-shared-b-w0-67363b-openshift-default-745d499d87-z5d9k" Oct 29 23:15:24.819542 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:24.819362 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76be25ad-e827-41d0-975d-a6bf7a1dcdee-credential-socket\") pod \"e2e-shared-b-w0-67363b-openshift-default-745d499d87-z5d9k\" (UID: \"76be25ad-e827-41d0-975d-a6bf7a1dcdee\") " pod="openshift-ingress/e2e-shared-b-w0-67363b-openshift-default-745d499d87-z5d9k" Oct 29 23:15:24.819542 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:24.819383 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76be25ad-e827-41d0-975d-a6bf7a1dcdee-istio-data\") pod \"e2e-shared-b-w0-67363b-openshift-default-745d499d87-z5d9k\" (UID: \"76be25ad-e827-41d0-975d-a6bf7a1dcdee\") " pod="openshift-ingress/e2e-shared-b-w0-67363b-openshift-default-745d499d87-z5d9k" Oct 29 23:15:24.819542 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:24.819419 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76be25ad-e827-41d0-975d-a6bf7a1dcdee-workload-certs\") pod \"e2e-shared-b-w0-67363b-openshift-default-745d499d87-z5d9k\" (UID: \"76be25ad-e827-41d0-975d-a6bf7a1dcdee\") " pod="openshift-ingress/e2e-shared-b-w0-67363b-openshift-default-745d499d87-z5d9k" Oct 29 23:15:24.819542 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:24.819444 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76be25ad-e827-41d0-975d-a6bf7a1dcdee-istio-podinfo\") pod \"e2e-shared-b-w0-67363b-openshift-default-745d499d87-z5d9k\" (UID: \"76be25ad-e827-41d0-975d-a6bf7a1dcdee\") " pod="openshift-ingress/e2e-shared-b-w0-67363b-openshift-default-745d499d87-z5d9k" Oct 29 23:15:24.819542 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:24.819476 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76be25ad-e827-41d0-975d-a6bf7a1dcdee-istio-token\") pod \"e2e-shared-b-w0-67363b-openshift-default-745d499d87-z5d9k\" (UID: \"76be25ad-e827-41d0-975d-a6bf7a1dcdee\") " pod="openshift-ingress/e2e-shared-b-w0-67363b-openshift-default-745d499d87-z5d9k" Oct 29 23:15:24.819542 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:24.819508 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76be25ad-e827-41d0-975d-a6bf7a1dcdee-istio-envoy\") pod \"e2e-shared-b-w0-67363b-openshift-default-745d499d87-z5d9k\" (UID: \"76be25ad-e827-41d0-975d-a6bf7a1dcdee\") " pod="openshift-ingress/e2e-shared-b-w0-67363b-openshift-default-745d499d87-z5d9k" Oct 29 23:15:24.819949 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:24.819832 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76be25ad-e827-41d0-975d-a6bf7a1dcdee-istiod-ca-cert\") pod \"e2e-shared-b-w0-67363b-openshift-default-745d499d87-z5d9k\" (UID: \"76be25ad-e827-41d0-975d-a6bf7a1dcdee\") " pod="openshift-ingress/e2e-shared-b-w0-67363b-openshift-default-745d499d87-z5d9k" Oct 29 23:15:24.819949 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:24.819837 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76be25ad-e827-41d0-975d-a6bf7a1dcdee-credential-socket\") pod \"e2e-shared-b-w0-67363b-openshift-default-745d499d87-z5d9k\" (UID: \"76be25ad-e827-41d0-975d-a6bf7a1dcdee\") " pod="openshift-ingress/e2e-shared-b-w0-67363b-openshift-default-745d499d87-z5d9k" Oct 29 23:15:24.819949 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:24.819849 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76be25ad-e827-41d0-975d-a6bf7a1dcdee-istio-data\") pod \"e2e-shared-b-w0-67363b-openshift-default-745d499d87-z5d9k\" (UID: \"76be25ad-e827-41d0-975d-a6bf7a1dcdee\") " pod="openshift-ingress/e2e-shared-b-w0-67363b-openshift-default-745d499d87-z5d9k" Oct 29 23:15:24.820115 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:24.819960 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22llm\" (UniqueName: \"kubernetes.io/projected/76be25ad-e827-41d0-975d-a6bf7a1dcdee-kube-api-access-22llm\") pod \"e2e-shared-b-w0-67363b-openshift-default-745d499d87-z5d9k\" (UID: \"76be25ad-e827-41d0-975d-a6bf7a1dcdee\") " pod="openshift-ingress/e2e-shared-b-w0-67363b-openshift-default-745d499d87-z5d9k" Oct 29 23:15:24.820181 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:24.820128 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76be25ad-e827-41d0-975d-a6bf7a1dcdee-workload-certs\") pod \"e2e-shared-b-w0-67363b-openshift-default-745d499d87-z5d9k\" (UID: \"76be25ad-e827-41d0-975d-a6bf7a1dcdee\") " pod="openshift-ingress/e2e-shared-b-w0-67363b-openshift-default-745d499d87-z5d9k" Oct 29 23:15:24.820442 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:24.820400 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76be25ad-e827-41d0-975d-a6bf7a1dcdee-workload-socket\") pod \"e2e-shared-b-w0-67363b-openshift-default-745d499d87-z5d9k\" (UID: \"76be25ad-e827-41d0-975d-a6bf7a1dcdee\") " pod="openshift-ingress/e2e-shared-b-w0-67363b-openshift-default-745d499d87-z5d9k" Oct 29 23:15:24.820558 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:24.820535 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76be25ad-e827-41d0-975d-a6bf7a1dcdee-istiod-ca-cert\") pod \"e2e-shared-b-w0-67363b-openshift-default-745d499d87-z5d9k\" (UID: \"76be25ad-e827-41d0-975d-a6bf7a1dcdee\") " pod="openshift-ingress/e2e-shared-b-w0-67363b-openshift-default-745d499d87-z5d9k" Oct 29 23:15:24.822612 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:24.822586 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76be25ad-e827-41d0-975d-a6bf7a1dcdee-istio-envoy\") pod \"e2e-shared-b-w0-67363b-openshift-default-745d499d87-z5d9k\" (UID: \"76be25ad-e827-41d0-975d-a6bf7a1dcdee\") " pod="openshift-ingress/e2e-shared-b-w0-67363b-openshift-default-745d499d87-z5d9k" Oct 29 23:15:24.822756 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:24.822648 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76be25ad-e827-41d0-975d-a6bf7a1dcdee-istio-podinfo\") pod \"e2e-shared-b-w0-67363b-openshift-default-745d499d87-z5d9k\" (UID: \"76be25ad-e827-41d0-975d-a6bf7a1dcdee\") " pod="openshift-ingress/e2e-shared-b-w0-67363b-openshift-default-745d499d87-z5d9k" Oct 29 23:15:24.842911 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:24.842886 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76be25ad-e827-41d0-975d-a6bf7a1dcdee-istio-token\") pod \"e2e-shared-b-w0-67363b-openshift-default-745d499d87-z5d9k\" (UID: \"76be25ad-e827-41d0-975d-a6bf7a1dcdee\") " pod="openshift-ingress/e2e-shared-b-w0-67363b-openshift-default-745d499d87-z5d9k" Oct 29 23:15:24.843052 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:24.843035 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-22llm\" (UniqueName: \"kubernetes.io/projected/76be25ad-e827-41d0-975d-a6bf7a1dcdee-kube-api-access-22llm\") pod \"e2e-shared-b-w0-67363b-openshift-default-745d499d87-z5d9k\" (UID: \"76be25ad-e827-41d0-975d-a6bf7a1dcdee\") " pod="openshift-ingress/e2e-shared-b-w0-67363b-openshift-default-745d499d87-z5d9k" Oct 29 23:15:24.940419 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:24.940386 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-67363b-openshift-default-745d499d87-z5d9k" Oct 29 23:15:25.276645 ip-10-0-140-74 kubenswrapper[5057]: W1029 23:15:25.276617 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76be25ad_e827_41d0_975d_a6bf7a1dcdee.slice/crio-968642abf6227f964a6e964b7996dc2a1c620ba1344b9a15244602d30992c67d WatchSource:0}: Error finding container 968642abf6227f964a6e964b7996dc2a1c620ba1344b9a15244602d30992c67d: Status 404 returned error can't find the container with id 968642abf6227f964a6e964b7996dc2a1c620ba1344b9a15244602d30992c67d Oct 29 23:15:25.277223 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:25.277199 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-67363b-openshift-default-745d499d87-z5d9k"] Oct 29 23:15:25.280434 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:25.280390 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 23:15:25.280518 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:25.280481 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 23:15:25.280567 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:25.280521 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 23:15:25.334308 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:25.334283 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="653090c9-6e06-4d91-9192-66c79775eded" path="/var/lib/kubelet/pods/653090c9-6e06-4d91-9192-66c79775eded/volumes" Oct 29 23:15:25.334871 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:25.334847 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0e370f0-45f8-4b62-aa7d-52414efa778e" path="/var/lib/kubelet/pods/d0e370f0-45f8-4b62-aa7d-52414efa778e/volumes" Oct 29 23:15:25.335293 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:25.335279 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dae1295f-4869-46d2-8cfb-0d89f2f3309c" path="/var/lib/kubelet/pods/dae1295f-4869-46d2-8cfb-0d89f2f3309c/volumes" Oct 29 23:15:25.606733 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:25.606628 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-67363b-openshift-default-745d499d87-z5d9k" event={"ID":"76be25ad-e827-41d0-975d-a6bf7a1dcdee","Type":"ContainerStarted","Data":"93d5677e2d0a1bc0250b19d36222d04ede6231167ead1db5731bb56bf2c9bddf"} Oct 29 23:15:25.606733 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:25.606706 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-67363b-openshift-default-745d499d87-z5d9k" event={"ID":"76be25ad-e827-41d0-975d-a6bf7a1dcdee","Type":"ContainerStarted","Data":"968642abf6227f964a6e964b7996dc2a1c620ba1344b9a15244602d30992c67d"} Oct 29 23:15:25.626979 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:25.626945 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:15:25.655917 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:25.655872 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-67363b-openshift-default-745d499d87-z5d9k" podStartSLOduration=1.6558586210000001 podStartE2EDuration="1.655858621s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:15:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:15:25.650123445 +0000 UTC m=+2506.433336871" watchObservedRunningTime="2026-10-29 23:15:25.655858621 +0000 UTC m=+2506.439072047" Oct 29 23:15:25.940878 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:25.940800 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-67363b-openshift-default-745d499d87-z5d9k" Oct 29 23:15:26.942864 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:26.942830 5057 patch_prober.go:29] interesting pod/e2e-shared-b-w0-67363b-openshift-default-745d499d87-z5d9k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" start-of-body= Oct 29 23:15:26.943377 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:26.942883 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-67363b-openshift-default-745d499d87-z5d9k" podUID="76be25ad-e827-41d0-975d-a6bf7a1dcdee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" Oct 29 23:15:27.240670 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:27.240396 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v2tfm"] Oct 29 23:15:27.710482 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:27.710449 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:15:27.941062 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:27.941017 5057 patch_prober.go:29] interesting pod/e2e-shared-b-w0-67363b-openshift-default-745d499d87-z5d9k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" start-of-body= Oct 29 23:15:27.941238 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:27.941091 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-67363b-openshift-default-745d499d87-z5d9k" podUID="76be25ad-e827-41d0-975d-a6bf7a1dcdee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" Oct 29 23:15:28.157012 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:28.156915 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:15:28.271261 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:28.271223 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:15:28.942356 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:28.941386 5057 patch_prober.go:29] interesting pod/e2e-shared-b-w0-67363b-openshift-default-745d499d87-z5d9k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" start-of-body= Oct 29 23:15:28.942356 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:28.941463 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-67363b-openshift-default-745d499d87-z5d9k" podUID="76be25ad-e827-41d0-975d-a6bf7a1dcdee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" Oct 29 23:15:29.941497 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:29.941465 5057 patch_prober.go:29] interesting pod/e2e-shared-b-w0-67363b-openshift-default-745d499d87-z5d9k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" start-of-body= Oct 29 23:15:29.941898 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:29.941532 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-67363b-openshift-default-745d499d87-z5d9k" podUID="76be25ad-e827-41d0-975d-a6bf7a1dcdee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" Oct 29 23:15:30.211631 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:30.211594 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:15:30.327583 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:30.327552 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:15:30.660530 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:30.626757 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:15:30.940979 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:30.940875 5057 patch_prober.go:29] interesting pod/e2e-shared-b-w0-67363b-openshift-default-745d499d87-z5d9k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" start-of-body= Oct 29 23:15:30.940979 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:30.940954 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-67363b-openshift-default-745d499d87-z5d9k" podUID="76be25ad-e827-41d0-975d-a6bf7a1dcdee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" Oct 29 23:15:31.445901 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:31.445861 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w0-4504f1/auth-test-model-4504f1-kserve-7694bcddfc-dt6vg" podUID="8d7303c8-8d98-4ff1-bb7c-2018d73a5033" containerName="main" containerID="cri-o://ffbd0e5175827b44d40f512779ee80119e511fbda77e7c45c2507d89f43c3e2a" gracePeriod=45 Oct 29 23:15:31.834373 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:31.833847 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-4504f1/auth-test-model-4504f1-kserve-7694bcddfc-dt6vg" Oct 29 23:15:31.850710 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:31.850676 5057 generic.go:356] "Generic (PLEG): container finished" podID="8d7303c8-8d98-4ff1-bb7c-2018d73a5033" containerID="ffbd0e5175827b44d40f512779ee80119e511fbda77e7c45c2507d89f43c3e2a" exitCode=0 Oct 29 23:15:31.850895 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:31.850723 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-4504f1/auth-test-model-4504f1-kserve-7694bcddfc-dt6vg" event={"ID":"8d7303c8-8d98-4ff1-bb7c-2018d73a5033","Type":"ContainerDied","Data":"ffbd0e5175827b44d40f512779ee80119e511fbda77e7c45c2507d89f43c3e2a"} Oct 29 23:15:31.850895 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:31.850754 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-4504f1/auth-test-model-4504f1-kserve-7694bcddfc-dt6vg" event={"ID":"8d7303c8-8d98-4ff1-bb7c-2018d73a5033","Type":"ContainerDied","Data":"3cbd17fd945785763b2070ae021600f680edc1a3bf4ddd25fbaa7f67b96663ac"} Oct 29 23:15:31.850895 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:31.850776 5057 scope.go:122] "RemoveContainer" containerID="ffbd0e5175827b44d40f512779ee80119e511fbda77e7c45c2507d89f43c3e2a" Oct 29 23:15:31.850895 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:31.850724 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-4504f1/auth-test-model-4504f1-kserve-7694bcddfc-dt6vg" Oct 29 23:15:31.862379 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:31.862319 5057 scope.go:122] "RemoveContainer" containerID="ffbd0e5175827b44d40f512779ee80119e511fbda77e7c45c2507d89f43c3e2a" Oct 29 23:15:31.862647 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:15:31.862625 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ffbd0e5175827b44d40f512779ee80119e511fbda77e7c45c2507d89f43c3e2a\": container with ID starting with ffbd0e5175827b44d40f512779ee80119e511fbda77e7c45c2507d89f43c3e2a not found: ID does not exist" containerID="ffbd0e5175827b44d40f512779ee80119e511fbda77e7c45c2507d89f43c3e2a" Oct 29 23:15:31.862716 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:31.862656 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ffbd0e5175827b44d40f512779ee80119e511fbda77e7c45c2507d89f43c3e2a"} err="failed to get container status \"ffbd0e5175827b44d40f512779ee80119e511fbda77e7c45c2507d89f43c3e2a\": rpc error: code = NotFound desc = could not find container \"ffbd0e5175827b44d40f512779ee80119e511fbda77e7c45c2507d89f43c3e2a\": container with ID starting with ffbd0e5175827b44d40f512779ee80119e511fbda77e7c45c2507d89f43c3e2a not found: ID does not exist" Oct 29 23:15:31.925153 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:31.925117 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8d7303c8-8d98-4ff1-bb7c-2018d73a5033-tls-certs\" (UniqueName: \"kubernetes.io/secret/8d7303c8-8d98-4ff1-bb7c-2018d73a5033-tls-certs\") pod \"8d7303c8-8d98-4ff1-bb7c-2018d73a5033\" (UID: \"8d7303c8-8d98-4ff1-bb7c-2018d73a5033\") " Oct 29 23:15:31.927400 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:31.925602 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d7303c8-8d98-4ff1-bb7c-2018d73a5033-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8d7303c8-8d98-4ff1-bb7c-2018d73a5033-dshm\") pod \"8d7303c8-8d98-4ff1-bb7c-2018d73a5033\" (UID: \"8d7303c8-8d98-4ff1-bb7c-2018d73a5033\") " Oct 29 23:15:31.927400 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:31.925687 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d7303c8-8d98-4ff1-bb7c-2018d73a5033-home\" (UniqueName: \"kubernetes.io/empty-dir/8d7303c8-8d98-4ff1-bb7c-2018d73a5033-home\") pod \"8d7303c8-8d98-4ff1-bb7c-2018d73a5033\" (UID: \"8d7303c8-8d98-4ff1-bb7c-2018d73a5033\") " Oct 29 23:15:31.927400 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:31.925777 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d7303c8-8d98-4ff1-bb7c-2018d73a5033-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d7303c8-8d98-4ff1-bb7c-2018d73a5033-tmp-dir\") pod \"8d7303c8-8d98-4ff1-bb7c-2018d73a5033\" (UID: \"8d7303c8-8d98-4ff1-bb7c-2018d73a5033\") " Oct 29 23:15:31.927400 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:31.925950 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d7303c8-8d98-4ff1-bb7c-2018d73a5033-kube-api-access-lrfpb\" (UniqueName: \"kubernetes.io/projected/8d7303c8-8d98-4ff1-bb7c-2018d73a5033-kube-api-access-lrfpb\") pod \"8d7303c8-8d98-4ff1-bb7c-2018d73a5033\" (UID: \"8d7303c8-8d98-4ff1-bb7c-2018d73a5033\") " Oct 29 23:15:31.927400 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:31.926083 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d7303c8-8d98-4ff1-bb7c-2018d73a5033-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8d7303c8-8d98-4ff1-bb7c-2018d73a5033-model-cache\") pod \"8d7303c8-8d98-4ff1-bb7c-2018d73a5033\" (UID: \"8d7303c8-8d98-4ff1-bb7c-2018d73a5033\") " Oct 29 23:15:31.927884 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:31.927847 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d7303c8-8d98-4ff1-bb7c-2018d73a5033-model-cache" pod "8d7303c8-8d98-4ff1-bb7c-2018d73a5033" (UID: "8d7303c8-8d98-4ff1-bb7c-2018d73a5033"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:15:31.928076 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:31.928055 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d7303c8-8d98-4ff1-bb7c-2018d73a5033-home" pod "8d7303c8-8d98-4ff1-bb7c-2018d73a5033" (UID: "8d7303c8-8d98-4ff1-bb7c-2018d73a5033"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:15:31.928264 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:31.928244 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d7303c8-8d98-4ff1-bb7c-2018d73a5033-tmp-dir" pod "8d7303c8-8d98-4ff1-bb7c-2018d73a5033" (UID: "8d7303c8-8d98-4ff1-bb7c-2018d73a5033"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:15:31.929994 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:31.929966 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d7303c8-8d98-4ff1-bb7c-2018d73a5033-tls-certs" pod "8d7303c8-8d98-4ff1-bb7c-2018d73a5033" (UID: "8d7303c8-8d98-4ff1-bb7c-2018d73a5033"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:15:31.930448 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:31.930418 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d7303c8-8d98-4ff1-bb7c-2018d73a5033-dshm" pod "8d7303c8-8d98-4ff1-bb7c-2018d73a5033" (UID: "8d7303c8-8d98-4ff1-bb7c-2018d73a5033"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:15:31.931505 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:31.931481 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d7303c8-8d98-4ff1-bb7c-2018d73a5033-kube-api-access-lrfpb" pod "8d7303c8-8d98-4ff1-bb7c-2018d73a5033" (UID: "8d7303c8-8d98-4ff1-bb7c-2018d73a5033"). InnerVolumeSpecName "kube-api-access-lrfpb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:15:31.941722 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:31.941697 5057 patch_prober.go:29] interesting pod/e2e-shared-b-w0-67363b-openshift-default-745d499d87-z5d9k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" start-of-body= Oct 29 23:15:31.941830 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:31.941747 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-67363b-openshift-default-745d499d87-z5d9k" podUID="76be25ad-e827-41d0-975d-a6bf7a1dcdee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" Oct 29 23:15:32.028810 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:32.028774 5057 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8d7303c8-8d98-4ff1-bb7c-2018d73a5033-dshm\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:15:32.028810 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:32.028805 5057 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8d7303c8-8d98-4ff1-bb7c-2018d73a5033-home\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:15:32.028810 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:32.028814 5057 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d7303c8-8d98-4ff1-bb7c-2018d73a5033-tmp-dir\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:15:32.029124 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:32.028823 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lrfpb\" (UniqueName: \"kubernetes.io/projected/8d7303c8-8d98-4ff1-bb7c-2018d73a5033-kube-api-access-lrfpb\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:15:32.029124 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:32.028833 5057 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8d7303c8-8d98-4ff1-bb7c-2018d73a5033-model-cache\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:15:32.029124 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:32.028842 5057 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d7303c8-8d98-4ff1-bb7c-2018d73a5033-tls-certs\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:15:32.204210 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:32.204178 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-4504f1/auth-test-model-4504f1-kserve-7694bcddfc-dt6vg"] Oct 29 23:15:32.217914 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:32.217883 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w0-4504f1/auth-test-model-4504f1-kserve-7694bcddfc-dt6vg"] Oct 29 23:15:32.843121 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:32.843080 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:15:32.946494 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:32.946463 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-67363b-openshift-default-745d499d87-z5d9k" Oct 29 23:15:32.946985 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:32.946948 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-67363b-openshift-default-745d499d87-z5d9k" Oct 29 23:15:32.947895 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:32.947875 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-67363b-openshift-default-745d499d87-z5d9k" Oct 29 23:15:33.332431 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:33.332397 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d7303c8-8d98-4ff1-bb7c-2018d73a5033" path="/var/lib/kubelet/pods/8d7303c8-8d98-4ff1-bb7c-2018d73a5033/volumes" Oct 29 23:15:33.664672 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:33.664594 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4265824e-x58pr"] Oct 29 23:15:33.666407 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:33.666263 5057 cpu_manager.go:414] "containerMap: removing container" podUID="8d7303c8-8d98-4ff1-bb7c-2018d73a5033" containerName="main" Oct 29 23:15:33.666573 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:33.666426 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d7303c8-8d98-4ff1-bb7c-2018d73a5033" containerName="main" Oct 29 23:15:33.666573 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:33.666451 5057 cpu_manager.go:414] "containerMap: removing container" podUID="709fa6cf-8954-48fc-ac8a-b893fb663bd9" containerName="cleanup" Oct 29 23:15:33.666573 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:33.666459 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="709fa6cf-8954-48fc-ac8a-b893fb663bd9" containerName="cleanup" Oct 29 23:15:33.666573 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:33.666486 5057 cpu_manager.go:414] "containerMap: removing container" podUID="56b2d3d2-3c8f-4b6f-a32a-e882dc660fa4" containerName="cleanup" Oct 29 23:15:33.666573 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:33.666494 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56b2d3d2-3c8f-4b6f-a32a-e882dc660fa4" containerName="cleanup" Oct 29 23:15:33.666573 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:33.666529 5057 cpu_manager.go:414] "containerMap: removing container" podUID="d0e370f0-45f8-4b62-aa7d-52414efa778e" containerName="cleanup" Oct 29 23:15:33.666573 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:33.666536 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0e370f0-45f8-4b62-aa7d-52414efa778e" containerName="cleanup" Oct 29 23:15:33.666573 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:33.666556 5057 cpu_manager.go:414] "containerMap: removing container" podUID="4d649d28-2651-4a9a-b73e-05d3407743e5" containerName="cleanup" Oct 29 23:15:33.666573 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:33.666563 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d649d28-2651-4a9a-b73e-05d3407743e5" containerName="cleanup" Oct 29 23:15:33.667108 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:33.666589 5057 cpu_manager.go:414] "containerMap: removing container" podUID="dae1295f-4869-46d2-8cfb-0d89f2f3309c" containerName="cleanup" Oct 29 23:15:33.667108 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:33.666596 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dae1295f-4869-46d2-8cfb-0d89f2f3309c" containerName="cleanup" Oct 29 23:15:33.667108 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:33.666651 5057 cpu_manager.go:414] "containerMap: removing container" podUID="1dc9bc0e-b528-42cb-86b6-5a4790e3a0c3" containerName="cleanup" Oct 29 23:15:33.667108 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:33.666657 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1dc9bc0e-b528-42cb-86b6-5a4790e3a0c3" containerName="cleanup" Oct 29 23:15:33.667108 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:33.666934 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d7303c8-8d98-4ff1-bb7c-2018d73a5033" containerName="main" Oct 29 23:15:33.667108 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:33.666963 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="56b2d3d2-3c8f-4b6f-a32a-e882dc660fa4" containerName="cleanup" Oct 29 23:15:33.667108 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:33.667018 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="653090c9-6e06-4d91-9192-66c79775eded" containerName="cleanup" Oct 29 23:15:33.667108 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:33.667031 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="709fa6cf-8954-48fc-ac8a-b893fb663bd9" containerName="cleanup" Oct 29 23:15:33.667108 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:33.667045 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0e370f0-45f8-4b62-aa7d-52414efa778e" containerName="cleanup" Oct 29 23:15:33.667108 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:33.667053 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="dae1295f-4869-46d2-8cfb-0d89f2f3309c" containerName="cleanup" Oct 29 23:15:33.673549 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:33.673527 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4265824e-x58pr" Oct 29 23:15:33.695201 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:33.695174 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4265824e-x58pr"] Oct 29 23:15:33.697151 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:33.697125 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:15:33.697269 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:33.697169 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dmx57\"" Oct 29 23:15:33.759711 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:33.759664 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ae7ef4b-9c55-41c6-bc89-1007e8fd63b1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4265824e-x58pr\" (UID: \"7ae7ef4b-9c55-41c6-bc89-1007e8fd63b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4265824e-x58pr" Oct 29 23:15:33.759904 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:33.759810 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4v477\" (UniqueName: \"kubernetes.io/projected/7ae7ef4b-9c55-41c6-bc89-1007e8fd63b1-kube-api-access-4v477\") pod \"maas-api-revoke-sub-4265824e-x58pr\" (UID: \"7ae7ef4b-9c55-41c6-bc89-1007e8fd63b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4265824e-x58pr" Oct 29 23:15:33.760189 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:33.760158 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ae7ef4b-9c55-41c6-bc89-1007e8fd63b1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4265824e-x58pr\" (UID: \"7ae7ef4b-9c55-41c6-bc89-1007e8fd63b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4265824e-x58pr" Oct 29 23:15:33.862246 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:33.862206 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4v477\" (UniqueName: \"kubernetes.io/projected/7ae7ef4b-9c55-41c6-bc89-1007e8fd63b1-kube-api-access-4v477\") pod \"maas-api-revoke-sub-4265824e-x58pr\" (UID: \"7ae7ef4b-9c55-41c6-bc89-1007e8fd63b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4265824e-x58pr" Oct 29 23:15:33.862708 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:33.862461 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ae7ef4b-9c55-41c6-bc89-1007e8fd63b1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4265824e-x58pr\" (UID: \"7ae7ef4b-9c55-41c6-bc89-1007e8fd63b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4265824e-x58pr" Oct 29 23:15:33.862708 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:33.862547 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ae7ef4b-9c55-41c6-bc89-1007e8fd63b1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4265824e-x58pr\" (UID: \"7ae7ef4b-9c55-41c6-bc89-1007e8fd63b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4265824e-x58pr" Oct 29 23:15:33.863225 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:33.863203 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ae7ef4b-9c55-41c6-bc89-1007e8fd63b1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4265824e-x58pr\" (UID: \"7ae7ef4b-9c55-41c6-bc89-1007e8fd63b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4265824e-x58pr" Oct 29 23:15:33.864909 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:33.864883 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ae7ef4b-9c55-41c6-bc89-1007e8fd63b1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4265824e-x58pr\" (UID: \"7ae7ef4b-9c55-41c6-bc89-1007e8fd63b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4265824e-x58pr" Oct 29 23:15:33.886087 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:33.886058 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4v477\" (UniqueName: \"kubernetes.io/projected/7ae7ef4b-9c55-41c6-bc89-1007e8fd63b1-kube-api-access-4v477\") pod \"maas-api-revoke-sub-4265824e-x58pr\" (UID: \"7ae7ef4b-9c55-41c6-bc89-1007e8fd63b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4265824e-x58pr" Oct 29 23:15:33.997499 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:33.997467 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4265824e-x58pr" Oct 29 23:15:34.318436 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:34.318195 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4265824e-x58pr"] Oct 29 23:15:34.318658 ip-10-0-140-74 kubenswrapper[5057]: W1029 23:15:34.318623 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ae7ef4b_9c55_41c6_bc89_1007e8fd63b1.slice/crio-6cb4f6f16f23c3de98e64b27408756c21ed87f0b7dce9019a3e468ee5a966749 WatchSource:0}: Error finding container 6cb4f6f16f23c3de98e64b27408756c21ed87f0b7dce9019a3e468ee5a966749: Status 404 returned error can't find the container with id 6cb4f6f16f23c3de98e64b27408756c21ed87f0b7dce9019a3e468ee5a966749 Oct 29 23:15:34.452472 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:34.452438 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:15:34.582527 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:34.582455 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:15:34.883758 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:34.883648 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4265824e-x58pr" event={"ID":"7ae7ef4b-9c55-41c6-bc89-1007e8fd63b1","Type":"ContainerStarted","Data":"cb9df601f3b58829063cb9ae2a55f6af59ad9c9cda67697f7b84636564b43690"} Oct 29 23:15:34.883758 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:34.883732 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4265824e-x58pr" event={"ID":"7ae7ef4b-9c55-41c6-bc89-1007e8fd63b1","Type":"ContainerStarted","Data":"6cb4f6f16f23c3de98e64b27408756c21ed87f0b7dce9019a3e468ee5a966749"} Oct 29 23:15:34.924098 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:34.924052 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4265824e-x58pr" podStartSLOduration=1.924037614 podStartE2EDuration="1.924037614s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:15:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:15:34.922827608 +0000 UTC m=+2515.706041034" watchObservedRunningTime="2026-10-29 23:15:34.924037614 +0000 UTC m=+2515.707251039" Oct 29 23:15:35.897777 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:35.897738 5057 generic.go:356] "Generic (PLEG): container finished" podID="7ae7ef4b-9c55-41c6-bc89-1007e8fd63b1" containerID="cb9df601f3b58829063cb9ae2a55f6af59ad9c9cda67697f7b84636564b43690" exitCode=0 Oct 29 23:15:35.898174 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:35.897813 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4265824e-x58pr" event={"ID":"7ae7ef4b-9c55-41c6-bc89-1007e8fd63b1","Type":"ContainerDied","Data":"cb9df601f3b58829063cb9ae2a55f6af59ad9c9cda67697f7b84636564b43690"} Oct 29 23:15:36.526399 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:36.526357 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:15:36.871900 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:36.871813 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w4-af237d/auth-test-model-af237d-kserve-7444475bdf-5tw7n" podUID="813724c2-17b1-4473-9c1a-3ee7d092ecd5" containerName="main" containerID="cri-o://e974f95ebb4ff0cc0f75186e1477d7ca31a6d6f2e6f8dfc03025c6671b4a3544" gracePeriod=45 Oct 29 23:15:37.131997 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:37.131930 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4265824e-x58pr" Oct 29 23:15:37.238275 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:37.238244 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7ae7ef4b-9c55-41c6-bc89-1007e8fd63b1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ae7ef4b-9c55-41c6-bc89-1007e8fd63b1-maas-api-xks-ca\") pod \"7ae7ef4b-9c55-41c6-bc89-1007e8fd63b1\" (UID: \"7ae7ef4b-9c55-41c6-bc89-1007e8fd63b1\") " Oct 29 23:15:37.238423 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:37.238299 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ae7ef4b-9c55-41c6-bc89-1007e8fd63b1-kube-api-access-4v477\" (UniqueName: \"kubernetes.io/projected/7ae7ef4b-9c55-41c6-bc89-1007e8fd63b1-kube-api-access-4v477\") pod \"7ae7ef4b-9c55-41c6-bc89-1007e8fd63b1\" (UID: \"7ae7ef4b-9c55-41c6-bc89-1007e8fd63b1\") " Oct 29 23:15:37.238681 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:37.238655 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ae7ef4b-9c55-41c6-bc89-1007e8fd63b1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ae7ef4b-9c55-41c6-bc89-1007e8fd63b1-maas-api-openshift-ca\") pod \"7ae7ef4b-9c55-41c6-bc89-1007e8fd63b1\" (UID: \"7ae7ef4b-9c55-41c6-bc89-1007e8fd63b1\") " Oct 29 23:15:37.239187 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:37.239146 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ae7ef4b-9c55-41c6-bc89-1007e8fd63b1-maas-api-openshift-ca" pod "7ae7ef4b-9c55-41c6-bc89-1007e8fd63b1" (UID: "7ae7ef4b-9c55-41c6-bc89-1007e8fd63b1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:15:37.240162 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:37.240135 5057 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ae7ef4b-9c55-41c6-bc89-1007e8fd63b1-maas-api-openshift-ca\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:15:37.240696 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:37.240671 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ae7ef4b-9c55-41c6-bc89-1007e8fd63b1-kube-api-access-4v477" pod "7ae7ef4b-9c55-41c6-bc89-1007e8fd63b1" (UID: "7ae7ef4b-9c55-41c6-bc89-1007e8fd63b1"). InnerVolumeSpecName "kube-api-access-4v477". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:15:37.241865 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:37.241842 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ae7ef4b-9c55-41c6-bc89-1007e8fd63b1-maas-api-xks-ca" pod "7ae7ef4b-9c55-41c6-bc89-1007e8fd63b1" (UID: "7ae7ef4b-9c55-41c6-bc89-1007e8fd63b1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:15:37.270227 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:37.270205 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-af237d/auth-test-model-af237d-kserve-7444475bdf-5tw7n" Oct 29 23:15:37.340878 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:37.340846 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/813724c2-17b1-4473-9c1a-3ee7d092ecd5-kube-api-access-t52v9\" (UniqueName: \"kubernetes.io/projected/813724c2-17b1-4473-9c1a-3ee7d092ecd5-kube-api-access-t52v9\") pod \"813724c2-17b1-4473-9c1a-3ee7d092ecd5\" (UID: \"813724c2-17b1-4473-9c1a-3ee7d092ecd5\") " Oct 29 23:15:37.341068 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:37.341015 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/813724c2-17b1-4473-9c1a-3ee7d092ecd5-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/813724c2-17b1-4473-9c1a-3ee7d092ecd5-tmp-dir\") pod \"813724c2-17b1-4473-9c1a-3ee7d092ecd5\" (UID: \"813724c2-17b1-4473-9c1a-3ee7d092ecd5\") " Oct 29 23:15:37.341158 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:37.341068 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/813724c2-17b1-4473-9c1a-3ee7d092ecd5-tls-certs\" (UniqueName: \"kubernetes.io/secret/813724c2-17b1-4473-9c1a-3ee7d092ecd5-tls-certs\") pod \"813724c2-17b1-4473-9c1a-3ee7d092ecd5\" (UID: \"813724c2-17b1-4473-9c1a-3ee7d092ecd5\") " Oct 29 23:15:37.341158 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:37.341124 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/813724c2-17b1-4473-9c1a-3ee7d092ecd5-dshm\" (UniqueName: \"kubernetes.io/empty-dir/813724c2-17b1-4473-9c1a-3ee7d092ecd5-dshm\") pod \"813724c2-17b1-4473-9c1a-3ee7d092ecd5\" (UID: \"813724c2-17b1-4473-9c1a-3ee7d092ecd5\") " Oct 29 23:15:37.341258 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:37.341183 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/813724c2-17b1-4473-9c1a-3ee7d092ecd5-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/813724c2-17b1-4473-9c1a-3ee7d092ecd5-model-cache\") pod \"813724c2-17b1-4473-9c1a-3ee7d092ecd5\" (UID: \"813724c2-17b1-4473-9c1a-3ee7d092ecd5\") " Oct 29 23:15:37.341258 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:37.341229 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/813724c2-17b1-4473-9c1a-3ee7d092ecd5-home\" (UniqueName: \"kubernetes.io/empty-dir/813724c2-17b1-4473-9c1a-3ee7d092ecd5-home\") pod \"813724c2-17b1-4473-9c1a-3ee7d092ecd5\" (UID: \"813724c2-17b1-4473-9c1a-3ee7d092ecd5\") " Oct 29 23:15:37.341621 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:37.341578 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/813724c2-17b1-4473-9c1a-3ee7d092ecd5-home" pod "813724c2-17b1-4473-9c1a-3ee7d092ecd5" (UID: "813724c2-17b1-4473-9c1a-3ee7d092ecd5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:15:37.341791 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:37.341770 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/813724c2-17b1-4473-9c1a-3ee7d092ecd5-tmp-dir" pod "813724c2-17b1-4473-9c1a-3ee7d092ecd5" (UID: "813724c2-17b1-4473-9c1a-3ee7d092ecd5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:15:37.342385 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:37.342315 5057 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/813724c2-17b1-4473-9c1a-3ee7d092ecd5-tmp-dir\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:15:37.342385 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:37.342355 5057 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/813724c2-17b1-4473-9c1a-3ee7d092ecd5-home\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:15:37.342385 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:37.342369 5057 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ae7ef4b-9c55-41c6-bc89-1007e8fd63b1-maas-api-xks-ca\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:15:37.342385 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:37.342384 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4v477\" (UniqueName: \"kubernetes.io/projected/7ae7ef4b-9c55-41c6-bc89-1007e8fd63b1-kube-api-access-4v477\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:15:37.343736 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:37.343706 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/813724c2-17b1-4473-9c1a-3ee7d092ecd5-kube-api-access-t52v9" pod "813724c2-17b1-4473-9c1a-3ee7d092ecd5" (UID: "813724c2-17b1-4473-9c1a-3ee7d092ecd5"). InnerVolumeSpecName "kube-api-access-t52v9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:15:37.344001 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:37.343970 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/813724c2-17b1-4473-9c1a-3ee7d092ecd5-model-cache" pod "813724c2-17b1-4473-9c1a-3ee7d092ecd5" (UID: "813724c2-17b1-4473-9c1a-3ee7d092ecd5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:15:37.344846 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:37.344820 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/813724c2-17b1-4473-9c1a-3ee7d092ecd5-tls-certs" pod "813724c2-17b1-4473-9c1a-3ee7d092ecd5" (UID: "813724c2-17b1-4473-9c1a-3ee7d092ecd5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:15:37.345593 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:37.345571 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/813724c2-17b1-4473-9c1a-3ee7d092ecd5-dshm" pod "813724c2-17b1-4473-9c1a-3ee7d092ecd5" (UID: "813724c2-17b1-4473-9c1a-3ee7d092ecd5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:15:37.443722 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:37.443637 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t52v9\" (UniqueName: \"kubernetes.io/projected/813724c2-17b1-4473-9c1a-3ee7d092ecd5-kube-api-access-t52v9\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:15:37.443722 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:37.443663 5057 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/813724c2-17b1-4473-9c1a-3ee7d092ecd5-tls-certs\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:15:37.443722 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:37.443685 5057 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/813724c2-17b1-4473-9c1a-3ee7d092ecd5-dshm\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:15:37.443722 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:37.443694 5057 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/813724c2-17b1-4473-9c1a-3ee7d092ecd5-model-cache\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:15:37.923532 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:37.923498 5057 generic.go:356] "Generic (PLEG): container finished" podID="813724c2-17b1-4473-9c1a-3ee7d092ecd5" containerID="e974f95ebb4ff0cc0f75186e1477d7ca31a6d6f2e6f8dfc03025c6671b4a3544" exitCode=0 Oct 29 23:15:37.923701 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:37.923546 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-af237d/auth-test-model-af237d-kserve-7444475bdf-5tw7n" Oct 29 23:15:37.923701 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:37.923587 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-af237d/auth-test-model-af237d-kserve-7444475bdf-5tw7n" event={"ID":"813724c2-17b1-4473-9c1a-3ee7d092ecd5","Type":"ContainerDied","Data":"e974f95ebb4ff0cc0f75186e1477d7ca31a6d6f2e6f8dfc03025c6671b4a3544"} Oct 29 23:15:37.923701 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:37.923642 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-af237d/auth-test-model-af237d-kserve-7444475bdf-5tw7n" event={"ID":"813724c2-17b1-4473-9c1a-3ee7d092ecd5","Type":"ContainerDied","Data":"cd8354326439486c501664523fc327b9f8628788ba56f0fe834698624893a98b"} Oct 29 23:15:37.923701 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:37.923666 5057 scope.go:122] "RemoveContainer" containerID="e974f95ebb4ff0cc0f75186e1477d7ca31a6d6f2e6f8dfc03025c6671b4a3544" Oct 29 23:15:37.925969 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:37.925944 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4265824e-x58pr" event={"ID":"7ae7ef4b-9c55-41c6-bc89-1007e8fd63b1","Type":"ContainerDied","Data":"6cb4f6f16f23c3de98e64b27408756c21ed87f0b7dce9019a3e468ee5a966749"} Oct 29 23:15:37.926096 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:37.925987 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4265824e-x58pr" Oct 29 23:15:37.926096 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:37.925991 5057 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6cb4f6f16f23c3de98e64b27408756c21ed87f0b7dce9019a3e468ee5a966749" Oct 29 23:15:37.937697 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:37.937678 5057 scope.go:122] "RemoveContainer" containerID="e974f95ebb4ff0cc0f75186e1477d7ca31a6d6f2e6f8dfc03025c6671b4a3544" Oct 29 23:15:37.937986 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:15:37.937966 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e974f95ebb4ff0cc0f75186e1477d7ca31a6d6f2e6f8dfc03025c6671b4a3544\": container with ID starting with e974f95ebb4ff0cc0f75186e1477d7ca31a6d6f2e6f8dfc03025c6671b4a3544 not found: ID does not exist" containerID="e974f95ebb4ff0cc0f75186e1477d7ca31a6d6f2e6f8dfc03025c6671b4a3544" Oct 29 23:15:37.938061 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:37.937991 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e974f95ebb4ff0cc0f75186e1477d7ca31a6d6f2e6f8dfc03025c6671b4a3544"} err="failed to get container status \"e974f95ebb4ff0cc0f75186e1477d7ca31a6d6f2e6f8dfc03025c6671b4a3544\": rpc error: code = NotFound desc = could not find container \"e974f95ebb4ff0cc0f75186e1477d7ca31a6d6f2e6f8dfc03025c6671b4a3544\": container with ID starting with e974f95ebb4ff0cc0f75186e1477d7ca31a6d6f2e6f8dfc03025c6671b4a3544 not found: ID does not exist" Oct 29 23:15:37.978399 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:37.978363 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-af237d/auth-test-model-af237d-kserve-7444475bdf-5tw7n"] Oct 29 23:15:37.986076 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:37.986042 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w4-af237d/auth-test-model-af237d-kserve-7444475bdf-5tw7n"] Oct 29 23:15:38.293545 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:38.293495 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:15:39.332337 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:39.332294 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="813724c2-17b1-4473-9c1a-3ee7d092ecd5" path="/var/lib/kubelet/pods/813724c2-17b1-4473-9c1a-3ee7d092ecd5/volumes" Oct 29 23:15:39.667891 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:39.667596 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:15:39.856209 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:39.856169 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:15:41.328434 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:41.328398 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:15:41.486605 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:41.486562 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:15:41.603844 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:41.603752 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:15:41.720355 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:41.718455 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:15:43.479056 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:43.479019 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:15:44.052050 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:44.052018 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ea7eae2-2bt9w"] Oct 29 23:15:44.053018 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:44.053000 5057 cpu_manager.go:414] "containerMap: removing container" podUID="813724c2-17b1-4473-9c1a-3ee7d092ecd5" containerName="main" Oct 29 23:15:44.053084 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:44.053020 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="813724c2-17b1-4473-9c1a-3ee7d092ecd5" containerName="main" Oct 29 23:15:44.053084 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:44.053050 5057 cpu_manager.go:414] "containerMap: removing container" podUID="7ae7ef4b-9c55-41c6-bc89-1007e8fd63b1" containerName="revoke-keys" Oct 29 23:15:44.053084 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:44.053055 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ae7ef4b-9c55-41c6-bc89-1007e8fd63b1" containerName="revoke-keys" Oct 29 23:15:44.053232 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:44.053220 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="813724c2-17b1-4473-9c1a-3ee7d092ecd5" containerName="main" Oct 29 23:15:44.053282 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:44.053241 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ae7ef4b-9c55-41c6-bc89-1007e8fd63b1" containerName="revoke-keys" Oct 29 23:15:44.059151 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:44.059132 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ea7eae2-2bt9w" Oct 29 23:15:44.077581 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:44.077557 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ea7eae2-2bt9w"] Oct 29 23:15:44.085194 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:44.084341 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dmx57\"" Oct 29 23:15:44.085194 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:44.084925 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:15:44.121217 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:44.121151 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:15:44.173032 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:44.173000 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74c06cbc-5309-4045-9c17-c0fc69d05638-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3ea7eae2-2bt9w\" (UID: \"74c06cbc-5309-4045-9c17-c0fc69d05638\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ea7eae2-2bt9w" Oct 29 23:15:44.173209 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:44.173086 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74c06cbc-5309-4045-9c17-c0fc69d05638-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3ea7eae2-2bt9w\" (UID: \"74c06cbc-5309-4045-9c17-c0fc69d05638\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ea7eae2-2bt9w" Oct 29 23:15:44.173285 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:44.173218 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wm4l\" (UniqueName: \"kubernetes.io/projected/74c06cbc-5309-4045-9c17-c0fc69d05638-kube-api-access-6wm4l\") pod \"maas-api-revoke-sub-3ea7eae2-2bt9w\" (UID: \"74c06cbc-5309-4045-9c17-c0fc69d05638\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ea7eae2-2bt9w" Oct 29 23:15:44.275049 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:44.275016 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74c06cbc-5309-4045-9c17-c0fc69d05638-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3ea7eae2-2bt9w\" (UID: \"74c06cbc-5309-4045-9c17-c0fc69d05638\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ea7eae2-2bt9w" Oct 29 23:15:44.275192 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:44.275114 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74c06cbc-5309-4045-9c17-c0fc69d05638-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3ea7eae2-2bt9w\" (UID: \"74c06cbc-5309-4045-9c17-c0fc69d05638\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ea7eae2-2bt9w" Oct 29 23:15:44.275231 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:44.275193 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wm4l\" (UniqueName: \"kubernetes.io/projected/74c06cbc-5309-4045-9c17-c0fc69d05638-kube-api-access-6wm4l\") pod \"maas-api-revoke-sub-3ea7eae2-2bt9w\" (UID: \"74c06cbc-5309-4045-9c17-c0fc69d05638\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ea7eae2-2bt9w" Oct 29 23:15:44.275737 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:44.275714 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74c06cbc-5309-4045-9c17-c0fc69d05638-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3ea7eae2-2bt9w\" (UID: \"74c06cbc-5309-4045-9c17-c0fc69d05638\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ea7eae2-2bt9w" Oct 29 23:15:44.277554 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:44.277527 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74c06cbc-5309-4045-9c17-c0fc69d05638-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3ea7eae2-2bt9w\" (UID: \"74c06cbc-5309-4045-9c17-c0fc69d05638\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ea7eae2-2bt9w" Oct 29 23:15:44.298510 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:44.298482 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wm4l\" (UniqueName: \"kubernetes.io/projected/74c06cbc-5309-4045-9c17-c0fc69d05638-kube-api-access-6wm4l\") pod \"maas-api-revoke-sub-3ea7eae2-2bt9w\" (UID: \"74c06cbc-5309-4045-9c17-c0fc69d05638\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ea7eae2-2bt9w" Oct 29 23:15:44.375447 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:44.375380 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ea7eae2-2bt9w" Oct 29 23:15:44.690307 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:44.690276 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ea7eae2-2bt9w"] Oct 29 23:15:44.691381 ip-10-0-140-74 kubenswrapper[5057]: W1029 23:15:44.691355 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74c06cbc_5309_4045_9c17_c0fc69d05638.slice/crio-4464c5c531fa35b05f6ef2ccdb8a626d2579f4c113ac019c00337663886b7bd7 WatchSource:0}: Error finding container 4464c5c531fa35b05f6ef2ccdb8a626d2579f4c113ac019c00337663886b7bd7: Status 404 returned error can't find the container with id 4464c5c531fa35b05f6ef2ccdb8a626d2579f4c113ac019c00337663886b7bd7 Oct 29 23:15:45.036016 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:45.035983 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ea7eae2-2bt9w" event={"ID":"74c06cbc-5309-4045-9c17-c0fc69d05638","Type":"ContainerStarted","Data":"4464c5c531fa35b05f6ef2ccdb8a626d2579f4c113ac019c00337663886b7bd7"} Oct 29 23:15:45.337652 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:45.337573 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:15:45.470178 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:45.470132 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:15:46.050803 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:46.050720 5057 generic.go:356] "Generic (PLEG): container finished" podID="74c06cbc-5309-4045-9c17-c0fc69d05638" containerID="154ad77663a91b2fb80ce9d97ee4ab0e1a40096d46078821d41a53aebff654b7" exitCode=0 Oct 29 23:15:46.051209 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:46.050811 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ea7eae2-2bt9w" event={"ID":"74c06cbc-5309-4045-9c17-c0fc69d05638","Type":"ContainerDied","Data":"154ad77663a91b2fb80ce9d97ee4ab0e1a40096d46078821d41a53aebff654b7"} Oct 29 23:15:47.300357 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:47.300316 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:15:47.368641 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:47.368567 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ea7eae2-2bt9w" Oct 29 23:15:47.476384 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:47.472987 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/74c06cbc-5309-4045-9c17-c0fc69d05638-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74c06cbc-5309-4045-9c17-c0fc69d05638-maas-api-xks-ca\") pod \"74c06cbc-5309-4045-9c17-c0fc69d05638\" (UID: \"74c06cbc-5309-4045-9c17-c0fc69d05638\") " Oct 29 23:15:47.476384 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:47.473175 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74c06cbc-5309-4045-9c17-c0fc69d05638-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74c06cbc-5309-4045-9c17-c0fc69d05638-maas-api-openshift-ca\") pod \"74c06cbc-5309-4045-9c17-c0fc69d05638\" (UID: \"74c06cbc-5309-4045-9c17-c0fc69d05638\") " Oct 29 23:15:47.476384 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:47.473713 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74c06cbc-5309-4045-9c17-c0fc69d05638-maas-api-openshift-ca" pod "74c06cbc-5309-4045-9c17-c0fc69d05638" (UID: "74c06cbc-5309-4045-9c17-c0fc69d05638"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:15:47.476384 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:47.473748 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74c06cbc-5309-4045-9c17-c0fc69d05638-kube-api-access-6wm4l\" (UniqueName: \"kubernetes.io/projected/74c06cbc-5309-4045-9c17-c0fc69d05638-kube-api-access-6wm4l\") pod \"74c06cbc-5309-4045-9c17-c0fc69d05638\" (UID: \"74c06cbc-5309-4045-9c17-c0fc69d05638\") " Oct 29 23:15:47.476384 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:47.474715 5057 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74c06cbc-5309-4045-9c17-c0fc69d05638-maas-api-openshift-ca\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:15:47.476384 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:47.476157 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74c06cbc-5309-4045-9c17-c0fc69d05638-kube-api-access-6wm4l" pod "74c06cbc-5309-4045-9c17-c0fc69d05638" (UID: "74c06cbc-5309-4045-9c17-c0fc69d05638"). InnerVolumeSpecName "kube-api-access-6wm4l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:15:47.522377 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:47.522287 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74c06cbc-5309-4045-9c17-c0fc69d05638-maas-api-xks-ca" pod "74c06cbc-5309-4045-9c17-c0fc69d05638" (UID: "74c06cbc-5309-4045-9c17-c0fc69d05638"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:15:47.576091 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:47.576063 5057 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74c06cbc-5309-4045-9c17-c0fc69d05638-maas-api-xks-ca\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:15:47.576091 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:47.576090 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6wm4l\" (UniqueName: \"kubernetes.io/projected/74c06cbc-5309-4045-9c17-c0fc69d05638-kube-api-access-6wm4l\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:15:47.711251 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:47.711209 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:15:48.080163 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:48.080079 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ea7eae2-2bt9w" Oct 29 23:15:48.080163 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:48.080104 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ea7eae2-2bt9w" event={"ID":"74c06cbc-5309-4045-9c17-c0fc69d05638","Type":"ContainerDied","Data":"4464c5c531fa35b05f6ef2ccdb8a626d2579f4c113ac019c00337663886b7bd7"} Oct 29 23:15:48.080163 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:48.080153 5057 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4464c5c531fa35b05f6ef2ccdb8a626d2579f4c113ac019c00337663886b7bd7" Oct 29 23:15:48.226039 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:48.226001 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:15:48.766632 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:48.766585 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:15:48.920367 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:48.920318 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:15:49.064689 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:15:49.064606 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d2dnn" containerName="extractor" Oct 29 23:15:49.572269 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:49.572230 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:15:50.348642 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:50.348602 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:15:50.799130 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:50.799091 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:15:51.295316 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:51.295273 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:15:51.409779 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:51.409743 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:15:51.595406 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:51.594859 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:15:52.689538 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:52.689502 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:15:52.816182 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:52.816146 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:15:53.637586 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:53.637550 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:15:54.390533 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:54.390495 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:15:54.514259 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:54.514227 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:15:55.464373 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:55.464300 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:15:55.646996 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:55.646964 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:15:55.949456 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:55.948411 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:15:56.295277 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:56.295191 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:15:56.700857 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:56.700819 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:15:56.821354 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:56.820892 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:15:57.807395 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:57.807355 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:15:58.150647 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:58.150554 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:15:58.254792 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:58.254757 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-9e7cc9-784694f548-nrc56"] Oct 29 23:15:58.256106 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:58.256079 5057 cpu_manager.go:414] "containerMap: removing container" podUID="74c06cbc-5309-4045-9c17-c0fc69d05638" containerName="revoke-keys" Oct 29 23:15:58.256106 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:58.256107 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74c06cbc-5309-4045-9c17-c0fc69d05638" containerName="revoke-keys" Oct 29 23:15:58.256414 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:58.256376 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="74c06cbc-5309-4045-9c17-c0fc69d05638" containerName="revoke-keys" Oct 29 23:15:58.261111 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:58.261093 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-9e7cc9-784694f548-nrc56" Oct 29 23:15:58.265861 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:58.265835 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:15:58.286364 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:58.275979 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-9e7cc9-784694f548-nrc56"] Oct 29 23:15:58.286364 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:58.283579 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-9e7cc9\"" Oct 29 23:15:58.367560 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:58.367521 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/238bb923-ad70-47a6-8e04-2023e2042006-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-9e7cc9-784694f548-nrc56\" (UID: \"238bb923-ad70-47a6-8e04-2023e2042006\") " pod="openshift-ingress/payload-processing-e2e-inf-a-9e7cc9-784694f548-nrc56" Oct 29 23:15:58.367874 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:58.367844 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-csllq\" (UniqueName: \"kubernetes.io/projected/238bb923-ad70-47a6-8e04-2023e2042006-kube-api-access-csllq\") pod \"payload-processing-e2e-inf-a-9e7cc9-784694f548-nrc56\" (UID: \"238bb923-ad70-47a6-8e04-2023e2042006\") " pod="openshift-ingress/payload-processing-e2e-inf-a-9e7cc9-784694f548-nrc56" Oct 29 23:15:58.506351 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:58.505968 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-csllq\" (UniqueName: \"kubernetes.io/projected/238bb923-ad70-47a6-8e04-2023e2042006-kube-api-access-csllq\") pod \"payload-processing-e2e-inf-a-9e7cc9-784694f548-nrc56\" (UID: \"238bb923-ad70-47a6-8e04-2023e2042006\") " pod="openshift-ingress/payload-processing-e2e-inf-a-9e7cc9-784694f548-nrc56" Oct 29 23:15:58.506351 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:58.506249 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/238bb923-ad70-47a6-8e04-2023e2042006-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-9e7cc9-784694f548-nrc56\" (UID: \"238bb923-ad70-47a6-8e04-2023e2042006\") " pod="openshift-ingress/payload-processing-e2e-inf-a-9e7cc9-784694f548-nrc56" Oct 29 23:15:58.507475 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:58.507448 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/238bb923-ad70-47a6-8e04-2023e2042006-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-9e7cc9-784694f548-nrc56\" (UID: \"238bb923-ad70-47a6-8e04-2023e2042006\") " pod="openshift-ingress/payload-processing-e2e-inf-a-9e7cc9-784694f548-nrc56" Oct 29 23:15:58.566356 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:58.560039 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-csllq\" (UniqueName: \"kubernetes.io/projected/238bb923-ad70-47a6-8e04-2023e2042006-kube-api-access-csllq\") pod \"payload-processing-e2e-inf-a-9e7cc9-784694f548-nrc56\" (UID: \"238bb923-ad70-47a6-8e04-2023e2042006\") " pod="openshift-ingress/payload-processing-e2e-inf-a-9e7cc9-784694f548-nrc56" Oct 29 23:15:58.604357 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:58.603691 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-9e7cc9-784694f548-nrc56" Oct 29 23:15:58.948529 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:58.941017 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-9e7cc9-784694f548-nrc56"] Oct 29 23:15:59.204325 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:59.204289 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-9e7cc9-784694f548-nrc56" event={"ID":"238bb923-ad70-47a6-8e04-2023e2042006","Type":"ContainerStarted","Data":"2c10d09dd3853f21965bee2efd8e9d4a0c2e38c24f0ffde0d1bc8e91564346f1"} Oct 29 23:15:59.204514 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:59.204357 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-9e7cc9-784694f548-nrc56" event={"ID":"238bb923-ad70-47a6-8e04-2023e2042006","Type":"ContainerStarted","Data":"4be2436a9384811d63659b2a73954c0a644c174d3d74449e48017095abfbe69e"} Oct 29 23:15:59.204514 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:59.204381 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-9e7cc9-784694f548-nrc56" Oct 29 23:15:59.247847 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:59.247787 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-9e7cc9-784694f548-nrc56" podStartSLOduration=1.247768535 podStartE2EDuration="1.247768535s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:15:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:15:59.244241494 +0000 UTC m=+2540.027454921" watchObservedRunningTime="2026-10-29 23:15:59.247768535 +0000 UTC m=+2540.030981963" Oct 29 23:15:59.525910 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:15:59.525828 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:16:01.219017 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:16:01.218965 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:16:02.875486 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:16:02.875453 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:16:04.469959 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:16:04.469924 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:16:05.550082 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:16:05.550020 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:16:05.872847 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:16:05.872752 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:16:06.053056 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:16:06.053002 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:16:06.255253 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:16:06.255220 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:16:06.617402 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:16:06.617300 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:16:06.773322 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:16:06.773097 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:16:08.214125 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:16:08.214086 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:16:09.869599 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:16:09.869563 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:16:10.215903 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:16:10.215873 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-9e7cc9-784694f548-nrc56" Oct 29 23:16:10.393591 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:16:10.393550 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:16:10.526167 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:16:10.526082 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:16:11.728710 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:16:11.728673 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:16:12.086616 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:16:12.086526 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:16:12.203950 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:16:12.203904 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:16:12.417485 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:16:12.417397 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-faa4fc-5f7c674df4-mgk4c"] Oct 29 23:16:12.430721 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:16:12.430682 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-faa4fc-5f7c674df4-mgk4c" Oct 29 23:16:12.490308 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:16:12.490265 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-faa4fc-5f7c674df4-mgk4c"] Oct 29 23:16:12.490904 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:16:12.490877 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-faa4fc\"" Oct 29 23:16:12.501705 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:16:12.501609 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-faa4fc-dockercfg-5k98q\"" Oct 29 23:16:12.596369 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:16:12.583420 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jstp\" (UniqueName: \"kubernetes.io/projected/32151649-e3d5-434d-8385-d318db8ade35-kube-api-access-7jstp\") pod \"payload-processing-e2e-inf-a-faa4fc-5f7c674df4-mgk4c\" (UID: \"32151649-e3d5-434d-8385-d318db8ade35\") " pod="openshift-ingress/payload-processing-e2e-inf-a-faa4fc-5f7c674df4-mgk4c" Oct 29 23:16:12.596369 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:16:12.583500 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32151649-e3d5-434d-8385-d318db8ade35-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-faa4fc-5f7c674df4-mgk4c\" (UID: \"32151649-e3d5-434d-8385-d318db8ade35\") " pod="openshift-ingress/payload-processing-e2e-inf-a-faa4fc-5f7c674df4-mgk4c" Oct 29 23:16:12.685498 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:16:12.685409 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jstp\" (UniqueName: \"kubernetes.io/projected/32151649-e3d5-434d-8385-d318db8ade35-kube-api-access-7jstp\") pod \"payload-processing-e2e-inf-a-faa4fc-5f7c674df4-mgk4c\" (UID: \"32151649-e3d5-434d-8385-d318db8ade35\") " pod="openshift-ingress/payload-processing-e2e-inf-a-faa4fc-5f7c674df4-mgk4c" Oct 29 23:16:12.686076 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:16:12.686050 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32151649-e3d5-434d-8385-d318db8ade35-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-faa4fc-5f7c674df4-mgk4c\" (UID: \"32151649-e3d5-434d-8385-d318db8ade35\") " pod="openshift-ingress/payload-processing-e2e-inf-a-faa4fc-5f7c674df4-mgk4c" Oct 29 23:16:12.687240 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:16:12.687102 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32151649-e3d5-434d-8385-d318db8ade35-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-faa4fc-5f7c674df4-mgk4c\" (UID: \"32151649-e3d5-434d-8385-d318db8ade35\") " pod="openshift-ingress/payload-processing-e2e-inf-a-faa4fc-5f7c674df4-mgk4c" Oct 29 23:16:12.710368 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:16:12.709428 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jstp\" (UniqueName: \"kubernetes.io/projected/32151649-e3d5-434d-8385-d318db8ade35-kube-api-access-7jstp\") pod \"payload-processing-e2e-inf-a-faa4fc-5f7c674df4-mgk4c\" (UID: \"32151649-e3d5-434d-8385-d318db8ade35\") " pod="openshift-ingress/payload-processing-e2e-inf-a-faa4fc-5f7c674df4-mgk4c" Oct 29 23:16:12.913902 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:16:12.913872 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-faa4fc-5f7c674df4-mgk4c" Oct 29 23:16:13.223992 ip-10-0-140-74 kubenswrapper[5057]: W1029 23:16:13.223918 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32151649_e3d5_434d_8385_d318db8ade35.slice/crio-09fa602e23669d995e8521c514bcc6e5c51f23d962b9ead4db7e46ce742441e7 WatchSource:0}: Error finding container 09fa602e23669d995e8521c514bcc6e5c51f23d962b9ead4db7e46ce742441e7: Status 404 returned error can't find the container with id 09fa602e23669d995e8521c514bcc6e5c51f23d962b9ead4db7e46ce742441e7 Oct 29 23:16:13.228178 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:16:13.228151 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-faa4fc-5f7c674df4-mgk4c"] Oct 29 23:16:13.395374 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:16:13.393176 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-faa4fc-5f7c674df4-mgk4c" event={"ID":"32151649-e3d5-434d-8385-d318db8ade35","Type":"ContainerStarted","Data":"471095009c90e5135edc3e4bfbc83d9fb155a16bed636ca1b37f53a5e922abf1"} Oct 29 23:16:13.395374 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:16:13.393246 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-faa4fc-5f7c674df4-mgk4c" event={"ID":"32151649-e3d5-434d-8385-d318db8ade35","Type":"ContainerStarted","Data":"09fa602e23669d995e8521c514bcc6e5c51f23d962b9ead4db7e46ce742441e7"} Oct 29 23:16:13.395374 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:16:13.394397 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-faa4fc-5f7c674df4-mgk4c" Oct 29 23:16:13.437962 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:16:13.437908 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-faa4fc-5f7c674df4-mgk4c" podStartSLOduration=1.437893986 podStartE2EDuration="1.437893986s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:16:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:16:13.433856919 +0000 UTC m=+2554.217070367" watchObservedRunningTime="2026-10-29 23:16:13.437893986 +0000 UTC m=+2554.221107412" Oct 29 23:16:13.837751 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:16:13.837720 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:16:13.957202 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:16:13.957164 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:16:14.408901 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:16:14.408867 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:16:15.720485 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:16:15.720440 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:16:17.413180 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:16:17.413140 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:16:19.022265 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:16:19.022226 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:16:19.031406 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:16:19.031374 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d2dnn" containerName="extractor" Oct 29 23:16:20.730003 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:16:20.729913 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:16:22.411027 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:16:22.410982 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:16:22.922593 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:16:22.922556 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v2tfm"] Oct 29 23:16:23.108244 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:16:23.108199 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:16:25.457400 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:16:25.454138 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-faa4fc-5f7c674df4-mgk4c" Oct 29 23:16:25.776506 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:16:25.776422 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:16:27.132952 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:16:27.132916 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:16:27.773060 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:16:27.773019 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:16:28.247420 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:16:28.247294 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:16:28.929667 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:16:28.929623 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:16:29.505075 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:16:29.505038 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:16:29.729385 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:16:29.726374 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:16:30.366740 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:16:30.366693 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:16:31.545636 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:16:31.545602 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:16:31.665446 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:16:31.665404 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:16:31.814178 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:16:31.814093 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:16:32.499722 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:16:32.499687 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:16:34.218760 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:16:34.218721 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:16:34.584737 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:16:34.584648 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:16:35.884851 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:16:35.884812 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:16:36.001852 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:16:36.001822 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:16:36.375055 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:16:36.375016 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:16:36.874679 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:16:36.874638 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:16:36.992374 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:16:36.987229 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:16:37.829493 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:16:37.829454 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:16:39.536695 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:16:39.536658 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:16:40.683806 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:16:40.683769 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:16:40.798532 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:16:40.798496 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:16:41.503531 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:16:41.503496 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:16:41.636357 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:16:41.634815 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:16:42.456876 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:16:42.456841 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:16:42.572294 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:16:42.572250 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:16:43.701353 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:16:43.695296 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:16:45.414290 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:16:45.414247 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:16:45.522722 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:16:45.522682 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:16:45.646785 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:16:45.646714 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:16:47.289197 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:16:47.289160 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:16:48.975485 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:16:48.975445 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:16:49.056507 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:16:49.056465 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d2dnn" containerName="extractor" Oct 29 23:16:49.983297 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:16:49.983252 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-qbx4b"] Oct 29 23:16:49.991348 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:16:49.991313 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-qbx4b" Oct 29 23:16:50.017804 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:16:50.017776 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-ed3bf2-openshift-default-dockercfg-tnc4j\"" Oct 29 23:16:50.024856 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:16:50.024829 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-qbx4b"] Oct 29 23:16:50.030647 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:16:50.030615 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7dbc6f5-3755-4686-b1af-4fb84d22db3a-workload-certs\") pod \"e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-qbx4b\" (UID: \"e7dbc6f5-3755-4686-b1af-4fb84d22db3a\") " pod="openshift-ingress/e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-qbx4b" Oct 29 23:16:50.030763 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:16:50.030695 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7dbc6f5-3755-4686-b1af-4fb84d22db3a-credential-socket\") pod \"e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-qbx4b\" (UID: \"e7dbc6f5-3755-4686-b1af-4fb84d22db3a\") " pod="openshift-ingress/e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-qbx4b" Oct 29 23:16:50.030859 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:16:50.030830 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7dbc6f5-3755-4686-b1af-4fb84d22db3a-istio-podinfo\") pod \"e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-qbx4b\" (UID: \"e7dbc6f5-3755-4686-b1af-4fb84d22db3a\") " pod="openshift-ingress/e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-qbx4b" Oct 29 23:16:50.030961 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:16:50.030944 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7dbc6f5-3755-4686-b1af-4fb84d22db3a-istiod-ca-cert\") pod \"e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-qbx4b\" (UID: \"e7dbc6f5-3755-4686-b1af-4fb84d22db3a\") " pod="openshift-ingress/e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-qbx4b" Oct 29 23:16:50.031142 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:16:50.031123 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7dbc6f5-3755-4686-b1af-4fb84d22db3a-istio-envoy\") pod \"e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-qbx4b\" (UID: \"e7dbc6f5-3755-4686-b1af-4fb84d22db3a\") " pod="openshift-ingress/e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-qbx4b" Oct 29 23:16:50.031217 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:16:50.031196 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7dbc6f5-3755-4686-b1af-4fb84d22db3a-istio-token\") pod \"e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-qbx4b\" (UID: \"e7dbc6f5-3755-4686-b1af-4fb84d22db3a\") " pod="openshift-ingress/e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-qbx4b" Oct 29 23:16:50.031358 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:16:50.031317 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7dbc6f5-3755-4686-b1af-4fb84d22db3a-istio-data\") pod \"e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-qbx4b\" (UID: \"e7dbc6f5-3755-4686-b1af-4fb84d22db3a\") " pod="openshift-ingress/e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-qbx4b" Oct 29 23:16:50.031413 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:16:50.031373 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jbn5\" (UniqueName: \"kubernetes.io/projected/e7dbc6f5-3755-4686-b1af-4fb84d22db3a-kube-api-access-4jbn5\") pod \"e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-qbx4b\" (UID: \"e7dbc6f5-3755-4686-b1af-4fb84d22db3a\") " pod="openshift-ingress/e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-qbx4b" Oct 29 23:16:50.031456 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:16:50.031412 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7dbc6f5-3755-4686-b1af-4fb84d22db3a-workload-socket\") pod \"e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-qbx4b\" (UID: \"e7dbc6f5-3755-4686-b1af-4fb84d22db3a\") " pod="openshift-ingress/e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-qbx4b" Oct 29 23:16:50.133486 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:16:50.133449 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7dbc6f5-3755-4686-b1af-4fb84d22db3a-istio-envoy\") pod \"e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-qbx4b\" (UID: \"e7dbc6f5-3755-4686-b1af-4fb84d22db3a\") " pod="openshift-ingress/e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-qbx4b" Oct 29 23:16:50.133647 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:16:50.133515 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7dbc6f5-3755-4686-b1af-4fb84d22db3a-istio-token\") pod \"e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-qbx4b\" (UID: \"e7dbc6f5-3755-4686-b1af-4fb84d22db3a\") " pod="openshift-ingress/e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-qbx4b" Oct 29 23:16:50.133647 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:16:50.133562 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7dbc6f5-3755-4686-b1af-4fb84d22db3a-istio-data\") pod \"e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-qbx4b\" (UID: \"e7dbc6f5-3755-4686-b1af-4fb84d22db3a\") " pod="openshift-ingress/e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-qbx4b" Oct 29 23:16:50.133647 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:16:50.133580 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jbn5\" (UniqueName: \"kubernetes.io/projected/e7dbc6f5-3755-4686-b1af-4fb84d22db3a-kube-api-access-4jbn5\") pod \"e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-qbx4b\" (UID: \"e7dbc6f5-3755-4686-b1af-4fb84d22db3a\") " pod="openshift-ingress/e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-qbx4b" Oct 29 23:16:50.133647 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:16:50.133604 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7dbc6f5-3755-4686-b1af-4fb84d22db3a-workload-socket\") pod \"e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-qbx4b\" (UID: \"e7dbc6f5-3755-4686-b1af-4fb84d22db3a\") " pod="openshift-ingress/e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-qbx4b" Oct 29 23:16:50.133923 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:16:50.133683 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7dbc6f5-3755-4686-b1af-4fb84d22db3a-workload-certs\") pod \"e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-qbx4b\" (UID: \"e7dbc6f5-3755-4686-b1af-4fb84d22db3a\") " pod="openshift-ingress/e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-qbx4b" Oct 29 23:16:50.133923 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:16:50.133748 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7dbc6f5-3755-4686-b1af-4fb84d22db3a-credential-socket\") pod \"e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-qbx4b\" (UID: \"e7dbc6f5-3755-4686-b1af-4fb84d22db3a\") " pod="openshift-ingress/e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-qbx4b" Oct 29 23:16:50.133923 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:16:50.133826 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7dbc6f5-3755-4686-b1af-4fb84d22db3a-istio-podinfo\") pod \"e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-qbx4b\" (UID: \"e7dbc6f5-3755-4686-b1af-4fb84d22db3a\") " pod="openshift-ingress/e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-qbx4b" Oct 29 23:16:50.133923 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:16:50.133899 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7dbc6f5-3755-4686-b1af-4fb84d22db3a-istiod-ca-cert\") pod \"e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-qbx4b\" (UID: \"e7dbc6f5-3755-4686-b1af-4fb84d22db3a\") " pod="openshift-ingress/e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-qbx4b" Oct 29 23:16:50.134150 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:16:50.134084 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7dbc6f5-3755-4686-b1af-4fb84d22db3a-workload-certs\") pod \"e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-qbx4b\" (UID: \"e7dbc6f5-3755-4686-b1af-4fb84d22db3a\") " pod="openshift-ingress/e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-qbx4b" Oct 29 23:16:50.134150 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:16:50.134110 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7dbc6f5-3755-4686-b1af-4fb84d22db3a-workload-socket\") pod \"e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-qbx4b\" (UID: \"e7dbc6f5-3755-4686-b1af-4fb84d22db3a\") " pod="openshift-ingress/e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-qbx4b" Oct 29 23:16:50.134271 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:16:50.134190 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7dbc6f5-3755-4686-b1af-4fb84d22db3a-credential-socket\") pod \"e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-qbx4b\" (UID: \"e7dbc6f5-3755-4686-b1af-4fb84d22db3a\") " pod="openshift-ingress/e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-qbx4b" Oct 29 23:16:50.134395 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:16:50.134375 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7dbc6f5-3755-4686-b1af-4fb84d22db3a-istio-data\") pod \"e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-qbx4b\" (UID: \"e7dbc6f5-3755-4686-b1af-4fb84d22db3a\") " pod="openshift-ingress/e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-qbx4b" Oct 29 23:16:50.134626 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:16:50.134608 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7dbc6f5-3755-4686-b1af-4fb84d22db3a-istiod-ca-cert\") pod \"e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-qbx4b\" (UID: \"e7dbc6f5-3755-4686-b1af-4fb84d22db3a\") " pod="openshift-ingress/e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-qbx4b" Oct 29 23:16:50.135751 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:16:50.135729 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7dbc6f5-3755-4686-b1af-4fb84d22db3a-istio-envoy\") pod \"e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-qbx4b\" (UID: \"e7dbc6f5-3755-4686-b1af-4fb84d22db3a\") " pod="openshift-ingress/e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-qbx4b" Oct 29 23:16:50.136230 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:16:50.136213 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7dbc6f5-3755-4686-b1af-4fb84d22db3a-istio-podinfo\") pod \"e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-qbx4b\" (UID: \"e7dbc6f5-3755-4686-b1af-4fb84d22db3a\") " pod="openshift-ingress/e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-qbx4b" Oct 29 23:16:50.156305 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:16:50.156280 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7dbc6f5-3755-4686-b1af-4fb84d22db3a-istio-token\") pod \"e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-qbx4b\" (UID: \"e7dbc6f5-3755-4686-b1af-4fb84d22db3a\") " pod="openshift-ingress/e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-qbx4b" Oct 29 23:16:50.157298 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:16:50.157280 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jbn5\" (UniqueName: \"kubernetes.io/projected/e7dbc6f5-3755-4686-b1af-4fb84d22db3a-kube-api-access-4jbn5\") pod \"e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-qbx4b\" (UID: \"e7dbc6f5-3755-4686-b1af-4fb84d22db3a\") " pod="openshift-ingress/e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-qbx4b" Oct 29 23:16:50.310586 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:16:50.310494 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-qbx4b" Oct 29 23:16:50.616095 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:16:50.616010 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:16:50.641422 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:16:50.641392 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-qbx4b"] Oct 29 23:16:50.641672 ip-10-0-140-74 kubenswrapper[5057]: W1029 23:16:50.641648 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7dbc6f5_3755_4686_b1af_4fb84d22db3a.slice/crio-3496d3a5a685fd5d2ebfd5601b8df785d50cf47504f304b362105d3b4a5cb85b WatchSource:0}: Error finding container 3496d3a5a685fd5d2ebfd5601b8df785d50cf47504f304b362105d3b4a5cb85b: Status 404 returned error can't find the container with id 3496d3a5a685fd5d2ebfd5601b8df785d50cf47504f304b362105d3b4a5cb85b Oct 29 23:16:50.644947 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:16:50.644911 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 23:16:50.645052 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:16:50.644979 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 23:16:50.645052 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:16:50.645013 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 23:16:50.955230 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:16:50.955197 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-qbx4b" event={"ID":"e7dbc6f5-3755-4686-b1af-4fb84d22db3a","Type":"ContainerStarted","Data":"84195371efdc4d7172b68a9dc56f842bf74b50f4a8573093a1b0ea4dd9aad2b1"} Oct 29 23:16:50.955441 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:16:50.955262 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-qbx4b" event={"ID":"e7dbc6f5-3755-4686-b1af-4fb84d22db3a","Type":"ContainerStarted","Data":"3496d3a5a685fd5d2ebfd5601b8df785d50cf47504f304b362105d3b4a5cb85b"} Oct 29 23:16:51.009648 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:16:51.009599 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-qbx4b" podStartSLOduration=2.009585255 podStartE2EDuration="2.009585255s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:16:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:16:50.999362485 +0000 UTC m=+2591.782575928" watchObservedRunningTime="2026-10-29 23:16:51.009585255 +0000 UTC m=+2591.792798681" Oct 29 23:16:51.311690 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:16:51.311596 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-qbx4b" Oct 29 23:16:51.313176 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:16:51.313145 5057 patch_prober.go:29] interesting pod/e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-qbx4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" start-of-body= Oct 29 23:16:51.313319 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:16:51.313204 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-qbx4b" podUID="e7dbc6f5-3755-4686-b1af-4fb84d22db3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" Oct 29 23:16:52.314354 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:16:52.312727 5057 patch_prober.go:29] interesting pod/e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-qbx4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" start-of-body= Oct 29 23:16:52.314354 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:16:52.312808 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-qbx4b" podUID="e7dbc6f5-3755-4686-b1af-4fb84d22db3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" Oct 29 23:16:52.552935 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:16:52.552896 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:16:53.311792 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:16:53.311755 5057 patch_prober.go:29] interesting pod/e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-qbx4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" start-of-body= Oct 29 23:16:53.311984 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:16:53.311819 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-qbx4b" podUID="e7dbc6f5-3755-4686-b1af-4fb84d22db3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" Oct 29 23:16:54.316355 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:16:54.315391 5057 patch_prober.go:29] interesting pod/e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-qbx4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" start-of-body= Oct 29 23:16:54.316355 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:16:54.315455 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-qbx4b" podUID="e7dbc6f5-3755-4686-b1af-4fb84d22db3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" Oct 29 23:16:54.843577 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:16:54.843538 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:16:55.310833 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:16:55.310792 5057 patch_prober.go:29] interesting pod/e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-qbx4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" start-of-body= Oct 29 23:16:55.311019 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:16:55.310857 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-qbx4b" podUID="e7dbc6f5-3755-4686-b1af-4fb84d22db3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" Oct 29 23:16:56.311357 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:16:56.311048 5057 patch_prober.go:29] interesting pod/e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-qbx4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" start-of-body= Oct 29 23:16:56.311357 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:16:56.311122 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-qbx4b" podUID="e7dbc6f5-3755-4686-b1af-4fb84d22db3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" Oct 29 23:16:56.711889 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:16:56.711849 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:16:57.311201 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:16:57.311163 5057 patch_prober.go:29] interesting pod/e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-qbx4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" start-of-body= Oct 29 23:16:57.311453 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:16:57.311241 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-qbx4b" podUID="e7dbc6f5-3755-4686-b1af-4fb84d22db3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" Oct 29 23:16:58.317108 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:16:58.317073 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-qbx4b" Oct 29 23:16:58.317664 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:16:58.317644 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-qbx4b" Oct 29 23:16:58.318372 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:16:58.318348 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-qbx4b" Oct 29 23:16:58.426992 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:16:58.426958 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:16:59.280492 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:16:59.280430 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:17:00.268351 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:17:00.268302 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:17:01.975073 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:17:01.975028 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:17:02.100046 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:17:02.100008 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:17:04.007962 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:17:04.007925 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:17:04.898981 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:17:04.898947 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:17:05.063709 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:17:05.063531 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:17:05.477575 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:17:05.477442 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:17:05.585751 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:17:05.585714 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:17:06.098700 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:17:06.098663 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:17:07.850964 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:17:07.850919 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:17:10.019316 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:17:10.019277 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:17:10.608873 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:17:10.608828 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:17:10.721569 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:17:10.721525 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:17:11.045573 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:17:11.045521 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:17:11.514113 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:17:11.514076 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:17:11.701048 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:17:11.701007 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:17:12.511881 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:17:12.511843 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:17:14.230116 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:17:14.230081 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:17:15.217978 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:17:15.217943 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:17:15.329559 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:17:15.329523 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:17:16.019996 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:17:16.019959 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:17:16.868894 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:17:16.868854 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:17:17.008240 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:17:17.008185 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:17:17.928150 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:17:17.928115 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:17:19.030142 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:17:19.030103 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d2dnn" containerName="extractor" Oct 29 23:17:19.625118 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:17:19.625074 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:17:21.254544 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:17:21.254513 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:17:22.816537 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:17:22.816498 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:17:24.403845 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:17:24.403807 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:17:26.032561 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:17:26.032505 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:17:27.653311 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:17:27.653267 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:17:29.423986 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:17:29.423948 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:17:30.993865 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:17:30.993830 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:17:31.125318 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:17:31.125282 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:17:31.236078 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:17:31.236037 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:17:32.791784 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:17:32.791748 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:17:34.404419 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:17:34.404380 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:17:34.724522 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:17:34.724490 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:17:36.044489 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:17:36.044454 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:17:37.560402 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:17:37.558388 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v2tfm"] Oct 29 23:17:38.423577 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:17:38.423544 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:17:40.187795 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:17:40.187763 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:17:41.163845 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:17:41.163805 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:17:42.605495 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:17:42.605456 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:17:43.086929 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:17:43.086891 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:17:43.326287 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:17:43.326232 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:17:43.652680 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:17:43.652638 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:17:43.782357 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:17:43.779862 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:17:45.298053 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:17:45.298013 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:17:47.311037 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:17:47.310998 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:17:47.991598 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:17:47.991557 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:17:48.113034 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:17:48.112999 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:17:48.224217 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:17:48.224177 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:17:48.630966 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:17:48.630929 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:17:49.031836 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:17:49.031791 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d2dnn" containerName="extractor" Oct 29 23:17:49.123215 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:17:49.123179 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:17:49.271345 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:17:49.271280 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:17:49.784948 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:17:49.784860 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:17:49.897424 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:17:49.897386 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:17:49.960292 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:17:49.960258 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-a9ae14-685b89c579-htlrh"] Oct 29 23:17:49.967653 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:17:49.967630 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a9ae14-685b89c579-htlrh" Oct 29 23:17:49.982435 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:17:49.982401 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-a9ae14-685b89c579-htlrh"] Oct 29 23:17:49.983090 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:17:49.983071 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-a9ae14-dockercfg-57n7t\"" Oct 29 23:17:49.987159 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:17:49.987135 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-a9ae14\"" Oct 29 23:17:50.353246 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:17:50.352255 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c68bc52c-5a5a-49ca-bd1a-4dc908b4aee0-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-a9ae14-685b89c579-htlrh\" (UID: \"c68bc52c-5a5a-49ca-bd1a-4dc908b4aee0\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a9ae14-685b89c579-htlrh" Oct 29 23:17:50.353246 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:17:50.352595 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75b54\" (UniqueName: \"kubernetes.io/projected/c68bc52c-5a5a-49ca-bd1a-4dc908b4aee0-kube-api-access-75b54\") pod \"payload-pre-processing-e2e-inf-b-a9ae14-685b89c579-htlrh\" (UID: \"c68bc52c-5a5a-49ca-bd1a-4dc908b4aee0\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a9ae14-685b89c579-htlrh" Oct 29 23:17:50.404473 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:17:50.404438 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-a9ae14-75774d5dcb-ddk62"] Oct 29 23:17:50.427617 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:17:50.427586 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-a9ae14-75774d5dcb-ddk62"] Oct 29 23:17:50.427801 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:17:50.427703 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-a9ae14-75774d5dcb-ddk62" Oct 29 23:17:50.454937 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:17:50.454898 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ef2b071-92d9-44ae-9068-cf14644d05d5-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-a9ae14-75774d5dcb-ddk62\" (UID: \"3ef2b071-92d9-44ae-9068-cf14644d05d5\") " pod="openshift-ingress/payload-processing-e2e-inf-b-a9ae14-75774d5dcb-ddk62" Oct 29 23:17:50.455109 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:17:50.455083 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c68bc52c-5a5a-49ca-bd1a-4dc908b4aee0-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-a9ae14-685b89c579-htlrh\" (UID: \"c68bc52c-5a5a-49ca-bd1a-4dc908b4aee0\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a9ae14-685b89c579-htlrh" Oct 29 23:17:50.455483 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:17:50.455456 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75b54\" (UniqueName: \"kubernetes.io/projected/c68bc52c-5a5a-49ca-bd1a-4dc908b4aee0-kube-api-access-75b54\") pod \"payload-pre-processing-e2e-inf-b-a9ae14-685b89c579-htlrh\" (UID: \"c68bc52c-5a5a-49ca-bd1a-4dc908b4aee0\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a9ae14-685b89c579-htlrh" Oct 29 23:17:50.455586 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:17:50.455559 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s58fz\" (UniqueName: \"kubernetes.io/projected/3ef2b071-92d9-44ae-9068-cf14644d05d5-kube-api-access-s58fz\") pod \"payload-processing-e2e-inf-b-a9ae14-75774d5dcb-ddk62\" (UID: \"3ef2b071-92d9-44ae-9068-cf14644d05d5\") " pod="openshift-ingress/payload-processing-e2e-inf-b-a9ae14-75774d5dcb-ddk62" Oct 29 23:17:50.456424 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:17:50.456401 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c68bc52c-5a5a-49ca-bd1a-4dc908b4aee0-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-a9ae14-685b89c579-htlrh\" (UID: \"c68bc52c-5a5a-49ca-bd1a-4dc908b4aee0\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a9ae14-685b89c579-htlrh" Oct 29 23:17:50.485325 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:17:50.485297 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-75b54\" (UniqueName: \"kubernetes.io/projected/c68bc52c-5a5a-49ca-bd1a-4dc908b4aee0-kube-api-access-75b54\") pod \"payload-pre-processing-e2e-inf-b-a9ae14-685b89c579-htlrh\" (UID: \"c68bc52c-5a5a-49ca-bd1a-4dc908b4aee0\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a9ae14-685b89c579-htlrh" Oct 29 23:17:50.557040 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:17:50.557012 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s58fz\" (UniqueName: \"kubernetes.io/projected/3ef2b071-92d9-44ae-9068-cf14644d05d5-kube-api-access-s58fz\") pod \"payload-processing-e2e-inf-b-a9ae14-75774d5dcb-ddk62\" (UID: \"3ef2b071-92d9-44ae-9068-cf14644d05d5\") " pod="openshift-ingress/payload-processing-e2e-inf-b-a9ae14-75774d5dcb-ddk62" Oct 29 23:17:50.557199 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:17:50.557087 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ef2b071-92d9-44ae-9068-cf14644d05d5-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-a9ae14-75774d5dcb-ddk62\" (UID: \"3ef2b071-92d9-44ae-9068-cf14644d05d5\") " pod="openshift-ingress/payload-processing-e2e-inf-b-a9ae14-75774d5dcb-ddk62" Oct 29 23:17:50.557840 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:17:50.557817 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ef2b071-92d9-44ae-9068-cf14644d05d5-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-a9ae14-75774d5dcb-ddk62\" (UID: \"3ef2b071-92d9-44ae-9068-cf14644d05d5\") " pod="openshift-ingress/payload-processing-e2e-inf-b-a9ae14-75774d5dcb-ddk62" Oct 29 23:17:50.583570 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:17:50.581512 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s58fz\" (UniqueName: \"kubernetes.io/projected/3ef2b071-92d9-44ae-9068-cf14644d05d5-kube-api-access-s58fz\") pod \"payload-processing-e2e-inf-b-a9ae14-75774d5dcb-ddk62\" (UID: \"3ef2b071-92d9-44ae-9068-cf14644d05d5\") " pod="openshift-ingress/payload-processing-e2e-inf-b-a9ae14-75774d5dcb-ddk62" Oct 29 23:17:50.588380 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:17:50.588351 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a9ae14-685b89c579-htlrh" Oct 29 23:17:50.752871 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:17:50.752556 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-a9ae14-75774d5dcb-ddk62" Oct 29 23:17:50.926019 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:17:50.925916 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-a9ae14-685b89c579-htlrh"] Oct 29 23:17:51.080457 ip-10-0-140-74 kubenswrapper[5057]: W1029 23:17:51.080428 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ef2b071_92d9_44ae_9068_cf14644d05d5.slice/crio-519dad2a8ebc5a94746918b2e5ab6dee1a22e4f830f9b921af986c1b1f1dcd1d WatchSource:0}: Error finding container 519dad2a8ebc5a94746918b2e5ab6dee1a22e4f830f9b921af986c1b1f1dcd1d: Status 404 returned error can't find the container with id 519dad2a8ebc5a94746918b2e5ab6dee1a22e4f830f9b921af986c1b1f1dcd1d Oct 29 23:17:51.081669 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:17:51.081639 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-a9ae14-75774d5dcb-ddk62"] Oct 29 23:17:51.398385 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:17:51.396455 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a9ae14-685b89c579-htlrh" event={"ID":"c68bc52c-5a5a-49ca-bd1a-4dc908b4aee0","Type":"ContainerStarted","Data":"3aec26c4039a003051f43c5542694ce94abc5decb7a965a2aac1c00f7f53a339"} Oct 29 23:17:51.398385 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:17:51.396523 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a9ae14-685b89c579-htlrh" event={"ID":"c68bc52c-5a5a-49ca-bd1a-4dc908b4aee0","Type":"ContainerStarted","Data":"3c108f2faddce9bfb5e1fa2cc40a9a0c16337408d1cb70173c88eee067757858"} Oct 29 23:17:51.398385 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:17:51.397445 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a9ae14-685b89c579-htlrh" Oct 29 23:17:51.402046 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:17:51.402015 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-a9ae14-75774d5dcb-ddk62" event={"ID":"3ef2b071-92d9-44ae-9068-cf14644d05d5","Type":"ContainerStarted","Data":"29269ad39ce8c4695f85ab24c1d9a7f879d3439f101a7e0e5c5672acae85162d"} Oct 29 23:17:51.402220 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:17:51.402059 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-a9ae14-75774d5dcb-ddk62" event={"ID":"3ef2b071-92d9-44ae-9068-cf14644d05d5","Type":"ContainerStarted","Data":"519dad2a8ebc5a94746918b2e5ab6dee1a22e4f830f9b921af986c1b1f1dcd1d"} Oct 29 23:17:51.402750 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:17:51.402729 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-a9ae14-75774d5dcb-ddk62" Oct 29 23:17:51.439513 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:17:51.439324 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a9ae14-685b89c579-htlrh" podStartSLOduration=2.439309524 podStartE2EDuration="2.439309524s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:17:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:17:51.436589681 +0000 UTC m=+2652.219803106" watchObservedRunningTime="2026-10-29 23:17:51.439309524 +0000 UTC m=+2652.222522949" Oct 29 23:17:51.487674 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:17:51.487607 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-a9ae14-75774d5dcb-ddk62" podStartSLOduration=1.4875900020000001 podStartE2EDuration="1.487590002s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:17:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:17:51.482033334 +0000 UTC m=+2652.265246763" watchObservedRunningTime="2026-10-29 23:17:51.487590002 +0000 UTC m=+2652.270803431" Oct 29 23:17:51.528017 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:17:51.527974 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:17:52.173074 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:17:52.173028 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:17:52.630572 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:17:52.630535 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:17:52.827106 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:17:52.827071 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:17:53.819912 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:17:53.819841 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:17:53.935530 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:17:53.935495 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:17:54.074439 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:17:54.074361 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:17:54.185961 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:17:54.185911 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:17:55.847052 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:17:55.847015 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:17:56.038452 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:17:56.038417 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:17:56.112115 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:17:56.112032 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a37090-84ddb8588c-n4gn5"] Oct 29 23:17:56.119525 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:17:56.119499 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a37090-84ddb8588c-n4gn5" Oct 29 23:17:56.135803 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:17:56.135778 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a37090-84ddb8588c-n4gn5"] Oct 29 23:17:56.139952 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:17:56.139911 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-a37090\"" Oct 29 23:17:56.148031 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:17:56.148007 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-a37090-dockercfg-lg547\"" Oct 29 23:17:56.153951 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:17:56.153930 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a37090-58c6cbb588-lj97b"] Oct 29 23:17:56.159510 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:17:56.159487 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-a37090-58c6cbb588-lj97b" Oct 29 23:17:56.188356 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:17:56.177233 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:17:56.188356 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:17:56.178519 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a37090-58c6cbb588-lj97b"] Oct 29 23:17:56.354818 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:17:56.337573 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7gk5\" (UniqueName: \"kubernetes.io/projected/eb4eaa12-cde8-43d8-9af0-9534a90f9c6b-kube-api-access-r7gk5\") pod \"payload-pre-processing-e2e-inf-a-a37090-84ddb8588c-n4gn5\" (UID: \"eb4eaa12-cde8-43d8-9af0-9534a90f9c6b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a37090-84ddb8588c-n4gn5" Oct 29 23:17:56.390528 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:17:56.367313 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/159d07ab-681d-4431-9d7d-e08bcee96624-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-a37090-58c6cbb588-lj97b\" (UID: \"159d07ab-681d-4431-9d7d-e08bcee96624\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a37090-58c6cbb588-lj97b" Oct 29 23:17:56.397359 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:17:56.390845 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxk8j\" (UniqueName: \"kubernetes.io/projected/159d07ab-681d-4431-9d7d-e08bcee96624-kube-api-access-dxk8j\") pod \"payload-processing-e2e-inf-a-a37090-58c6cbb588-lj97b\" (UID: \"159d07ab-681d-4431-9d7d-e08bcee96624\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a37090-58c6cbb588-lj97b" Oct 29 23:17:56.417354 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:17:56.397618 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb4eaa12-cde8-43d8-9af0-9534a90f9c6b-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-a37090-84ddb8588c-n4gn5\" (UID: \"eb4eaa12-cde8-43d8-9af0-9534a90f9c6b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a37090-84ddb8588c-n4gn5" Oct 29 23:17:56.540358 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:17:56.534826 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxk8j\" (UniqueName: \"kubernetes.io/projected/159d07ab-681d-4431-9d7d-e08bcee96624-kube-api-access-dxk8j\") pod \"payload-processing-e2e-inf-a-a37090-58c6cbb588-lj97b\" (UID: \"159d07ab-681d-4431-9d7d-e08bcee96624\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a37090-58c6cbb588-lj97b" Oct 29 23:17:56.540358 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:17:56.534946 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb4eaa12-cde8-43d8-9af0-9534a90f9c6b-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-a37090-84ddb8588c-n4gn5\" (UID: \"eb4eaa12-cde8-43d8-9af0-9534a90f9c6b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a37090-84ddb8588c-n4gn5" Oct 29 23:17:56.540358 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:17:56.535434 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7gk5\" (UniqueName: \"kubernetes.io/projected/eb4eaa12-cde8-43d8-9af0-9534a90f9c6b-kube-api-access-r7gk5\") pod \"payload-pre-processing-e2e-inf-a-a37090-84ddb8588c-n4gn5\" (UID: \"eb4eaa12-cde8-43d8-9af0-9534a90f9c6b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a37090-84ddb8588c-n4gn5" Oct 29 23:17:56.540358 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:17:56.535959 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/159d07ab-681d-4431-9d7d-e08bcee96624-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-a37090-58c6cbb588-lj97b\" (UID: \"159d07ab-681d-4431-9d7d-e08bcee96624\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a37090-58c6cbb588-lj97b" Oct 29 23:17:56.540358 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:17:56.536587 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb4eaa12-cde8-43d8-9af0-9534a90f9c6b-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-a37090-84ddb8588c-n4gn5\" (UID: \"eb4eaa12-cde8-43d8-9af0-9534a90f9c6b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a37090-84ddb8588c-n4gn5" Oct 29 23:17:56.540358 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:17:56.539315 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/159d07ab-681d-4431-9d7d-e08bcee96624-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-a37090-58c6cbb588-lj97b\" (UID: \"159d07ab-681d-4431-9d7d-e08bcee96624\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a37090-58c6cbb588-lj97b" Oct 29 23:17:56.546350 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:17:56.542764 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-fba83e-openshift-default-7544fcb869-hrggn"] Oct 29 23:17:56.554562 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:17:56.554107 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-fba83e-openshift-default-7544fcb869-hrggn" Oct 29 23:17:56.568435 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:17:56.568370 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxk8j\" (UniqueName: \"kubernetes.io/projected/159d07ab-681d-4431-9d7d-e08bcee96624-kube-api-access-dxk8j\") pod \"payload-processing-e2e-inf-a-a37090-58c6cbb588-lj97b\" (UID: \"159d07ab-681d-4431-9d7d-e08bcee96624\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a37090-58c6cbb588-lj97b" Oct 29 23:17:56.581815 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:17:56.581784 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-fba83e-openshift-default-7544fcb869-hrggn"] Oct 29 23:17:56.585875 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:17:56.585846 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7gk5\" (UniqueName: \"kubernetes.io/projected/eb4eaa12-cde8-43d8-9af0-9534a90f9c6b-kube-api-access-r7gk5\") pod \"payload-pre-processing-e2e-inf-a-a37090-84ddb8588c-n4gn5\" (UID: \"eb4eaa12-cde8-43d8-9af0-9534a90f9c6b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a37090-84ddb8588c-n4gn5" Oct 29 23:17:56.591850 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:17:56.591610 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-a37090-58c6cbb588-lj97b" Oct 29 23:17:56.629991 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:17:56.626792 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-fba83e-openshift-default-dockercfg-s7cj7\"" Oct 29 23:17:56.645716 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:17:56.642895 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d01c437-34ee-4d97-a1cc-83037a6fbdaa-workload-certs\") pod \"e2e-inf-a-fba83e-openshift-default-7544fcb869-hrggn\" (UID: \"5d01c437-34ee-4d97-a1cc-83037a6fbdaa\") " pod="openshift-ingress/e2e-inf-a-fba83e-openshift-default-7544fcb869-hrggn" Oct 29 23:17:56.645716 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:17:56.643001 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d01c437-34ee-4d97-a1cc-83037a6fbdaa-workload-socket\") pod \"e2e-inf-a-fba83e-openshift-default-7544fcb869-hrggn\" (UID: \"5d01c437-34ee-4d97-a1cc-83037a6fbdaa\") " pod="openshift-ingress/e2e-inf-a-fba83e-openshift-default-7544fcb869-hrggn" Oct 29 23:17:56.645716 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:17:56.643114 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d01c437-34ee-4d97-a1cc-83037a6fbdaa-credential-socket\") pod \"e2e-inf-a-fba83e-openshift-default-7544fcb869-hrggn\" (UID: \"5d01c437-34ee-4d97-a1cc-83037a6fbdaa\") " pod="openshift-ingress/e2e-inf-a-fba83e-openshift-default-7544fcb869-hrggn" Oct 29 23:17:56.645716 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:17:56.643178 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d01c437-34ee-4d97-a1cc-83037a6fbdaa-istio-podinfo\") pod \"e2e-inf-a-fba83e-openshift-default-7544fcb869-hrggn\" (UID: \"5d01c437-34ee-4d97-a1cc-83037a6fbdaa\") " pod="openshift-ingress/e2e-inf-a-fba83e-openshift-default-7544fcb869-hrggn" Oct 29 23:17:56.645716 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:17:56.643250 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d01c437-34ee-4d97-a1cc-83037a6fbdaa-istio-token\") pod \"e2e-inf-a-fba83e-openshift-default-7544fcb869-hrggn\" (UID: \"5d01c437-34ee-4d97-a1cc-83037a6fbdaa\") " pod="openshift-ingress/e2e-inf-a-fba83e-openshift-default-7544fcb869-hrggn" Oct 29 23:17:56.645716 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:17:56.643366 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d01c437-34ee-4d97-a1cc-83037a6fbdaa-istio-envoy\") pod \"e2e-inf-a-fba83e-openshift-default-7544fcb869-hrggn\" (UID: \"5d01c437-34ee-4d97-a1cc-83037a6fbdaa\") " pod="openshift-ingress/e2e-inf-a-fba83e-openshift-default-7544fcb869-hrggn" Oct 29 23:17:56.645716 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:17:56.643449 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqxwz\" (UniqueName: \"kubernetes.io/projected/5d01c437-34ee-4d97-a1cc-83037a6fbdaa-kube-api-access-nqxwz\") pod \"e2e-inf-a-fba83e-openshift-default-7544fcb869-hrggn\" (UID: \"5d01c437-34ee-4d97-a1cc-83037a6fbdaa\") " pod="openshift-ingress/e2e-inf-a-fba83e-openshift-default-7544fcb869-hrggn" Oct 29 23:17:56.645716 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:17:56.643486 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d01c437-34ee-4d97-a1cc-83037a6fbdaa-istiod-ca-cert\") pod \"e2e-inf-a-fba83e-openshift-default-7544fcb869-hrggn\" (UID: \"5d01c437-34ee-4d97-a1cc-83037a6fbdaa\") " pod="openshift-ingress/e2e-inf-a-fba83e-openshift-default-7544fcb869-hrggn" Oct 29 23:17:56.645716 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:17:56.643537 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d01c437-34ee-4d97-a1cc-83037a6fbdaa-istio-data\") pod \"e2e-inf-a-fba83e-openshift-default-7544fcb869-hrggn\" (UID: \"5d01c437-34ee-4d97-a1cc-83037a6fbdaa\") " pod="openshift-ingress/e2e-inf-a-fba83e-openshift-default-7544fcb869-hrggn" Oct 29 23:17:56.645716 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:17:56.645140 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:17:56.793079 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:17:56.740444 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a37090-84ddb8588c-n4gn5" Oct 29 23:17:56.793079 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:17:56.746960 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqxwz\" (UniqueName: \"kubernetes.io/projected/5d01c437-34ee-4d97-a1cc-83037a6fbdaa-kube-api-access-nqxwz\") pod \"e2e-inf-a-fba83e-openshift-default-7544fcb869-hrggn\" (UID: \"5d01c437-34ee-4d97-a1cc-83037a6fbdaa\") " pod="openshift-ingress/e2e-inf-a-fba83e-openshift-default-7544fcb869-hrggn" Oct 29 23:17:56.793079 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:17:56.747015 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d01c437-34ee-4d97-a1cc-83037a6fbdaa-istiod-ca-cert\") pod \"e2e-inf-a-fba83e-openshift-default-7544fcb869-hrggn\" (UID: \"5d01c437-34ee-4d97-a1cc-83037a6fbdaa\") " pod="openshift-ingress/e2e-inf-a-fba83e-openshift-default-7544fcb869-hrggn" Oct 29 23:17:56.793079 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:17:56.747069 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d01c437-34ee-4d97-a1cc-83037a6fbdaa-istio-data\") pod \"e2e-inf-a-fba83e-openshift-default-7544fcb869-hrggn\" (UID: \"5d01c437-34ee-4d97-a1cc-83037a6fbdaa\") " pod="openshift-ingress/e2e-inf-a-fba83e-openshift-default-7544fcb869-hrggn" Oct 29 23:17:56.793079 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:17:56.747388 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d01c437-34ee-4d97-a1cc-83037a6fbdaa-workload-certs\") pod \"e2e-inf-a-fba83e-openshift-default-7544fcb869-hrggn\" (UID: \"5d01c437-34ee-4d97-a1cc-83037a6fbdaa\") " pod="openshift-ingress/e2e-inf-a-fba83e-openshift-default-7544fcb869-hrggn" Oct 29 23:17:56.793079 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:17:56.747490 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d01c437-34ee-4d97-a1cc-83037a6fbdaa-workload-socket\") pod \"e2e-inf-a-fba83e-openshift-default-7544fcb869-hrggn\" (UID: \"5d01c437-34ee-4d97-a1cc-83037a6fbdaa\") " pod="openshift-ingress/e2e-inf-a-fba83e-openshift-default-7544fcb869-hrggn" Oct 29 23:17:56.793079 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:17:56.747553 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d01c437-34ee-4d97-a1cc-83037a6fbdaa-credential-socket\") pod \"e2e-inf-a-fba83e-openshift-default-7544fcb869-hrggn\" (UID: \"5d01c437-34ee-4d97-a1cc-83037a6fbdaa\") " pod="openshift-ingress/e2e-inf-a-fba83e-openshift-default-7544fcb869-hrggn" Oct 29 23:17:56.793079 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:17:56.747655 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d01c437-34ee-4d97-a1cc-83037a6fbdaa-istio-podinfo\") pod \"e2e-inf-a-fba83e-openshift-default-7544fcb869-hrggn\" (UID: \"5d01c437-34ee-4d97-a1cc-83037a6fbdaa\") " pod="openshift-ingress/e2e-inf-a-fba83e-openshift-default-7544fcb869-hrggn" Oct 29 23:17:56.793079 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:17:56.747743 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d01c437-34ee-4d97-a1cc-83037a6fbdaa-istio-token\") pod \"e2e-inf-a-fba83e-openshift-default-7544fcb869-hrggn\" (UID: \"5d01c437-34ee-4d97-a1cc-83037a6fbdaa\") " pod="openshift-ingress/e2e-inf-a-fba83e-openshift-default-7544fcb869-hrggn" Oct 29 23:17:56.793079 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:17:56.747848 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d01c437-34ee-4d97-a1cc-83037a6fbdaa-istio-envoy\") pod \"e2e-inf-a-fba83e-openshift-default-7544fcb869-hrggn\" (UID: \"5d01c437-34ee-4d97-a1cc-83037a6fbdaa\") " pod="openshift-ingress/e2e-inf-a-fba83e-openshift-default-7544fcb869-hrggn" Oct 29 23:17:56.793079 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:17:56.748108 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d01c437-34ee-4d97-a1cc-83037a6fbdaa-istiod-ca-cert\") pod \"e2e-inf-a-fba83e-openshift-default-7544fcb869-hrggn\" (UID: \"5d01c437-34ee-4d97-a1cc-83037a6fbdaa\") " pod="openshift-ingress/e2e-inf-a-fba83e-openshift-default-7544fcb869-hrggn" Oct 29 23:17:56.793079 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:17:56.748649 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d01c437-34ee-4d97-a1cc-83037a6fbdaa-workload-socket\") pod \"e2e-inf-a-fba83e-openshift-default-7544fcb869-hrggn\" (UID: \"5d01c437-34ee-4d97-a1cc-83037a6fbdaa\") " pod="openshift-ingress/e2e-inf-a-fba83e-openshift-default-7544fcb869-hrggn" Oct 29 23:17:56.793079 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:17:56.748909 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d01c437-34ee-4d97-a1cc-83037a6fbdaa-istio-data\") pod \"e2e-inf-a-fba83e-openshift-default-7544fcb869-hrggn\" (UID: \"5d01c437-34ee-4d97-a1cc-83037a6fbdaa\") " pod="openshift-ingress/e2e-inf-a-fba83e-openshift-default-7544fcb869-hrggn" Oct 29 23:17:56.793079 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:17:56.749135 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d01c437-34ee-4d97-a1cc-83037a6fbdaa-workload-certs\") pod \"e2e-inf-a-fba83e-openshift-default-7544fcb869-hrggn\" (UID: \"5d01c437-34ee-4d97-a1cc-83037a6fbdaa\") " pod="openshift-ingress/e2e-inf-a-fba83e-openshift-default-7544fcb869-hrggn" Oct 29 23:17:56.793079 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:17:56.750117 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d01c437-34ee-4d97-a1cc-83037a6fbdaa-credential-socket\") pod \"e2e-inf-a-fba83e-openshift-default-7544fcb869-hrggn\" (UID: \"5d01c437-34ee-4d97-a1cc-83037a6fbdaa\") " pod="openshift-ingress/e2e-inf-a-fba83e-openshift-default-7544fcb869-hrggn" Oct 29 23:17:56.793079 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:17:56.763176 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d01c437-34ee-4d97-a1cc-83037a6fbdaa-istio-envoy\") pod \"e2e-inf-a-fba83e-openshift-default-7544fcb869-hrggn\" (UID: \"5d01c437-34ee-4d97-a1cc-83037a6fbdaa\") " pod="openshift-ingress/e2e-inf-a-fba83e-openshift-default-7544fcb869-hrggn" Oct 29 23:17:56.793079 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:17:56.763690 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d01c437-34ee-4d97-a1cc-83037a6fbdaa-istio-podinfo\") pod \"e2e-inf-a-fba83e-openshift-default-7544fcb869-hrggn\" (UID: \"5d01c437-34ee-4d97-a1cc-83037a6fbdaa\") " pod="openshift-ingress/e2e-inf-a-fba83e-openshift-default-7544fcb869-hrggn" Oct 29 23:17:56.793079 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:17:56.787083 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqxwz\" (UniqueName: \"kubernetes.io/projected/5d01c437-34ee-4d97-a1cc-83037a6fbdaa-kube-api-access-nqxwz\") pod \"e2e-inf-a-fba83e-openshift-default-7544fcb869-hrggn\" (UID: \"5d01c437-34ee-4d97-a1cc-83037a6fbdaa\") " pod="openshift-ingress/e2e-inf-a-fba83e-openshift-default-7544fcb869-hrggn" Oct 29 23:17:56.793079 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:17:56.788051 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d01c437-34ee-4d97-a1cc-83037a6fbdaa-istio-token\") pod \"e2e-inf-a-fba83e-openshift-default-7544fcb869-hrggn\" (UID: \"5d01c437-34ee-4d97-a1cc-83037a6fbdaa\") " pod="openshift-ingress/e2e-inf-a-fba83e-openshift-default-7544fcb869-hrggn" Oct 29 23:17:56.981151 ip-10-0-140-74 kubenswrapper[5057]: W1029 23:17:56.980954 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod159d07ab_681d_4431_9d7d_e08bcee96624.slice/crio-f3798d6b125fc942cadd4271397f2469da96052179a385b870d4d43738036e8a WatchSource:0}: Error finding container f3798d6b125fc942cadd4271397f2469da96052179a385b870d4d43738036e8a: Status 404 returned error can't find the container with id f3798d6b125fc942cadd4271397f2469da96052179a385b870d4d43738036e8a Oct 29 23:17:56.984873 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:17:56.984843 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a37090-58c6cbb588-lj97b"] Oct 29 23:17:57.008133 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:17:57.008104 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-fba83e-openshift-default-7544fcb869-hrggn" Oct 29 23:17:57.122231 ip-10-0-140-74 kubenswrapper[5057]: W1029 23:17:57.117893 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb4eaa12_cde8_43d8_9af0_9534a90f9c6b.slice/crio-a39a8bae9b75b04d7e49a24bf51f46637908d417f4e8c8e276e8a8e9f7353f5e WatchSource:0}: Error finding container a39a8bae9b75b04d7e49a24bf51f46637908d417f4e8c8e276e8a8e9f7353f5e: Status 404 returned error can't find the container with id a39a8bae9b75b04d7e49a24bf51f46637908d417f4e8c8e276e8a8e9f7353f5e Oct 29 23:17:57.129028 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:17:57.128878 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a37090-84ddb8588c-n4gn5"] Oct 29 23:17:57.217596 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:17:57.217205 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:17:57.336170 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:17:57.336131 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:17:57.339129 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:17:57.339091 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 23:17:57.339254 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:17:57.339170 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 23:17:57.339254 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:17:57.339213 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 23:17:57.342284 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:17:57.342262 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-fba83e-openshift-default-7544fcb869-hrggn"] Oct 29 23:17:57.489324 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:17:57.489224 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a37090-84ddb8588c-n4gn5" event={"ID":"eb4eaa12-cde8-43d8-9af0-9534a90f9c6b","Type":"ContainerStarted","Data":"20440706175fed5a0dc35c1ba1e6aa0329ba5032d54d7f576193d92285a8d72a"} Oct 29 23:17:57.489324 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:17:57.489359 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a37090-84ddb8588c-n4gn5" event={"ID":"eb4eaa12-cde8-43d8-9af0-9534a90f9c6b","Type":"ContainerStarted","Data":"a39a8bae9b75b04d7e49a24bf51f46637908d417f4e8c8e276e8a8e9f7353f5e"} Oct 29 23:17:57.489626 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:17:57.489604 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a37090-84ddb8588c-n4gn5" Oct 29 23:17:57.492214 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:17:57.492182 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-a37090-58c6cbb588-lj97b" event={"ID":"159d07ab-681d-4431-9d7d-e08bcee96624","Type":"ContainerStarted","Data":"36f85597af859c5e4303f518d0d31d815a44e513f74da021785b2120bba5b1e3"} Oct 29 23:17:57.492431 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:17:57.492246 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-a37090-58c6cbb588-lj97b" event={"ID":"159d07ab-681d-4431-9d7d-e08bcee96624","Type":"ContainerStarted","Data":"f3798d6b125fc942cadd4271397f2469da96052179a385b870d4d43738036e8a"} Oct 29 23:17:57.493386 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:17:57.493192 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-a37090-58c6cbb588-lj97b" Oct 29 23:17:57.495927 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:17:57.495897 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-fba83e-openshift-default-7544fcb869-hrggn" event={"ID":"5d01c437-34ee-4d97-a1cc-83037a6fbdaa","Type":"ContainerStarted","Data":"2797dddcf89763789ef39efda61992dbd84a9d6d5fe7e069d8bc3255c7c317c0"} Oct 29 23:17:57.496029 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:17:57.495960 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-fba83e-openshift-default-7544fcb869-hrggn" event={"ID":"5d01c437-34ee-4d97-a1cc-83037a6fbdaa","Type":"ContainerStarted","Data":"9dd471111f9618d2fa1b6cb867c6d9361647a559da26febf63f5145db6e71290"} Oct 29 23:17:57.536416 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:17:57.536367 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a37090-84ddb8588c-n4gn5" podStartSLOduration=1.536353182 podStartE2EDuration="1.536353182s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:17:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:17:57.532862251 +0000 UTC m=+2658.316075680" watchObservedRunningTime="2026-10-29 23:17:57.536353182 +0000 UTC m=+2658.319566607" Oct 29 23:17:57.572456 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:17:57.572402 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-a37090-58c6cbb588-lj97b" podStartSLOduration=1.572388927 podStartE2EDuration="1.572388927s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:17:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:17:57.572029277 +0000 UTC m=+2658.355242702" watchObservedRunningTime="2026-10-29 23:17:57.572388927 +0000 UTC m=+2658.355602353" Oct 29 23:17:57.622733 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:17:57.622668 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-fba83e-openshift-default-7544fcb869-hrggn" podStartSLOduration=1.622653003 podStartE2EDuration="1.622653003s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:17:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:17:57.617011964 +0000 UTC m=+2658.400225414" watchObservedRunningTime="2026-10-29 23:17:57.622653003 +0000 UTC m=+2658.405866428" Oct 29 23:17:58.008687 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:17:58.008651 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-fba83e-openshift-default-7544fcb869-hrggn" Oct 29 23:17:58.010444 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:17:58.010414 5057 patch_prober.go:29] interesting pod/e2e-inf-a-fba83e-openshift-default-7544fcb869-hrggn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" start-of-body= Oct 29 23:17:58.010583 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:17:58.010466 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-fba83e-openshift-default-7544fcb869-hrggn" podUID="5d01c437-34ee-4d97-a1cc-83037a6fbdaa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" Oct 29 23:17:58.519921 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:17:58.519855 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:17:58.970735 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:17:58.970687 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:17:59.019537 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:17:59.019502 5057 patch_prober.go:29] interesting pod/e2e-inf-a-fba83e-openshift-default-7544fcb869-hrggn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" start-of-body= Oct 29 23:17:59.019999 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:17:59.019737 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-fba83e-openshift-default-7544fcb869-hrggn" podUID="5d01c437-34ee-4d97-a1cc-83037a6fbdaa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" Oct 29 23:17:59.114069 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:17:59.114030 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:17:59.154938 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:17:59.154904 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-dcacb9-86d7d89559-xk4m6"] Oct 29 23:17:59.161614 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:17:59.161591 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-dcacb9-86d7d89559-xk4m6" Oct 29 23:17:59.180559 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:17:59.180528 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-dcacb9-86d7d89559-xk4m6"] Oct 29 23:17:59.230651 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:17:59.230501 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-dcacb9\"" Oct 29 23:17:59.230953 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:17:59.230908 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-dcacb9-dockercfg-p5qlj\"" Oct 29 23:17:59.239666 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:17:59.239633 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-dcacb9-767c887c7b-jrk26"] Oct 29 23:17:59.268474 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:17:59.268446 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-dcacb9-767c887c7b-jrk26" Oct 29 23:17:59.275103 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:17:59.275066 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-dcacb9-767c887c7b-jrk26"] Oct 29 23:17:59.449383 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:17:59.432478 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gsqb2\" (UniqueName: \"kubernetes.io/projected/38161411-6a7a-4a65-aacf-e39baf965f8e-kube-api-access-gsqb2\") pod \"payload-pre-processing-e2e-inf-a-dcacb9-86d7d89559-xk4m6\" (UID: \"38161411-6a7a-4a65-aacf-e39baf965f8e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-dcacb9-86d7d89559-xk4m6" Oct 29 23:17:59.449383 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:17:59.432732 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38161411-6a7a-4a65-aacf-e39baf965f8e-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-dcacb9-86d7d89559-xk4m6\" (UID: \"38161411-6a7a-4a65-aacf-e39baf965f8e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-dcacb9-86d7d89559-xk4m6" Oct 29 23:17:59.541474 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:17:59.538815 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gsqb2\" (UniqueName: \"kubernetes.io/projected/38161411-6a7a-4a65-aacf-e39baf965f8e-kube-api-access-gsqb2\") pod \"payload-pre-processing-e2e-inf-a-dcacb9-86d7d89559-xk4m6\" (UID: \"38161411-6a7a-4a65-aacf-e39baf965f8e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-dcacb9-86d7d89559-xk4m6" Oct 29 23:17:59.541474 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:17:59.538897 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sn2sd\" (UniqueName: \"kubernetes.io/projected/26ab3c63-6e66-4981-b86c-bdf7055db34f-kube-api-access-sn2sd\") pod \"payload-processing-e2e-inf-a-dcacb9-767c887c7b-jrk26\" (UID: \"26ab3c63-6e66-4981-b86c-bdf7055db34f\") " pod="openshift-ingress/payload-processing-e2e-inf-a-dcacb9-767c887c7b-jrk26" Oct 29 23:17:59.541474 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:17:59.539069 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26ab3c63-6e66-4981-b86c-bdf7055db34f-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-dcacb9-767c887c7b-jrk26\" (UID: \"26ab3c63-6e66-4981-b86c-bdf7055db34f\") " pod="openshift-ingress/payload-processing-e2e-inf-a-dcacb9-767c887c7b-jrk26" Oct 29 23:17:59.541474 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:17:59.539148 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38161411-6a7a-4a65-aacf-e39baf965f8e-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-dcacb9-86d7d89559-xk4m6\" (UID: \"38161411-6a7a-4a65-aacf-e39baf965f8e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-dcacb9-86d7d89559-xk4m6" Oct 29 23:17:59.554104 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:17:59.547419 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38161411-6a7a-4a65-aacf-e39baf965f8e-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-dcacb9-86d7d89559-xk4m6\" (UID: \"38161411-6a7a-4a65-aacf-e39baf965f8e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-dcacb9-86d7d89559-xk4m6" Oct 29 23:17:59.665370 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:17:59.662389 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sn2sd\" (UniqueName: \"kubernetes.io/projected/26ab3c63-6e66-4981-b86c-bdf7055db34f-kube-api-access-sn2sd\") pod \"payload-processing-e2e-inf-a-dcacb9-767c887c7b-jrk26\" (UID: \"26ab3c63-6e66-4981-b86c-bdf7055db34f\") " pod="openshift-ingress/payload-processing-e2e-inf-a-dcacb9-767c887c7b-jrk26" Oct 29 23:17:59.665370 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:17:59.662585 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26ab3c63-6e66-4981-b86c-bdf7055db34f-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-dcacb9-767c887c7b-jrk26\" (UID: \"26ab3c63-6e66-4981-b86c-bdf7055db34f\") " pod="openshift-ingress/payload-processing-e2e-inf-a-dcacb9-767c887c7b-jrk26" Oct 29 23:17:59.665370 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:17:59.663846 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26ab3c63-6e66-4981-b86c-bdf7055db34f-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-dcacb9-767c887c7b-jrk26\" (UID: \"26ab3c63-6e66-4981-b86c-bdf7055db34f\") " pod="openshift-ingress/payload-processing-e2e-inf-a-dcacb9-767c887c7b-jrk26" Oct 29 23:17:59.742064 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:17:59.742026 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gsqb2\" (UniqueName: \"kubernetes.io/projected/38161411-6a7a-4a65-aacf-e39baf965f8e-kube-api-access-gsqb2\") pod \"payload-pre-processing-e2e-inf-a-dcacb9-86d7d89559-xk4m6\" (UID: \"38161411-6a7a-4a65-aacf-e39baf965f8e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-dcacb9-86d7d89559-xk4m6" Oct 29 23:17:59.745917 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:17:59.745889 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sn2sd\" (UniqueName: \"kubernetes.io/projected/26ab3c63-6e66-4981-b86c-bdf7055db34f-kube-api-access-sn2sd\") pod \"payload-processing-e2e-inf-a-dcacb9-767c887c7b-jrk26\" (UID: \"26ab3c63-6e66-4981-b86c-bdf7055db34f\") " pod="openshift-ingress/payload-processing-e2e-inf-a-dcacb9-767c887c7b-jrk26" Oct 29 23:17:59.838084 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:17:59.837991 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-dcacb9-86d7d89559-xk4m6" Oct 29 23:17:59.919351 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:17:59.905182 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-dcacb9-767c887c7b-jrk26" Oct 29 23:18:00.028023 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:18:00.025799 5057 patch_prober.go:29] interesting pod/e2e-inf-a-fba83e-openshift-default-7544fcb869-hrggn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" start-of-body= Oct 29 23:18:00.028023 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:18:00.025867 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-fba83e-openshift-default-7544fcb869-hrggn" podUID="5d01c437-34ee-4d97-a1cc-83037a6fbdaa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" Oct 29 23:18:00.222643 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:18:00.222603 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-dcacb9-86d7d89559-xk4m6"] Oct 29 23:18:00.224402 ip-10-0-140-74 kubenswrapper[5057]: W1029 23:18:00.224373 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38161411_6a7a_4a65_aacf_e39baf965f8e.slice/crio-a84e586a6dce7e5e458fe1f7635c81cce8aa562f97e1f1406b2b180fbaa11ff4 WatchSource:0}: Error finding container a84e586a6dce7e5e458fe1f7635c81cce8aa562f97e1f1406b2b180fbaa11ff4: Status 404 returned error can't find the container with id a84e586a6dce7e5e458fe1f7635c81cce8aa562f97e1f1406b2b180fbaa11ff4 Oct 29 23:18:00.296352 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:18:00.290141 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-dcacb9-767c887c7b-jrk26"] Oct 29 23:18:00.314025 ip-10-0-140-74 kubenswrapper[5057]: W1029 23:18:00.313974 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26ab3c63_6e66_4981_b86c_bdf7055db34f.slice/crio-625553809c5ee865c0ceb62978a86c5120bdd73dc2a6747105a414afcc143d5d WatchSource:0}: Error finding container 625553809c5ee865c0ceb62978a86c5120bdd73dc2a6747105a414afcc143d5d: Status 404 returned error can't find the container with id 625553809c5ee865c0ceb62978a86c5120bdd73dc2a6747105a414afcc143d5d Oct 29 23:18:00.861975 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:18:00.859608 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-dcacb9-767c887c7b-jrk26" event={"ID":"26ab3c63-6e66-4981-b86c-bdf7055db34f","Type":"ContainerStarted","Data":"09baca9166765ceb8dd71e72fb60b7cdeabed1278bb49142c70a5587b32ab390"} Oct 29 23:18:00.861975 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:18:00.859681 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-dcacb9-767c887c7b-jrk26" event={"ID":"26ab3c63-6e66-4981-b86c-bdf7055db34f","Type":"ContainerStarted","Data":"625553809c5ee865c0ceb62978a86c5120bdd73dc2a6747105a414afcc143d5d"} Oct 29 23:18:00.861975 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:18:00.860948 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-dcacb9-767c887c7b-jrk26" Oct 29 23:18:00.882502 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:18:00.881442 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-dcacb9-86d7d89559-xk4m6" event={"ID":"38161411-6a7a-4a65-aacf-e39baf965f8e","Type":"ContainerStarted","Data":"7af091fe73f5cc4c7a442becd64d2f40546de358686d9b023fcd45d2cef5ec3a"} Oct 29 23:18:00.882502 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:18:00.881507 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-dcacb9-86d7d89559-xk4m6" event={"ID":"38161411-6a7a-4a65-aacf-e39baf965f8e","Type":"ContainerStarted","Data":"a84e586a6dce7e5e458fe1f7635c81cce8aa562f97e1f1406b2b180fbaa11ff4"} Oct 29 23:18:00.882911 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:18:00.882888 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-dcacb9-86d7d89559-xk4m6" Oct 29 23:18:01.013888 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:18:01.013844 5057 patch_prober.go:29] interesting pod/e2e-inf-a-fba83e-openshift-default-7544fcb869-hrggn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" start-of-body= Oct 29 23:18:01.014095 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:18:01.013909 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-fba83e-openshift-default-7544fcb869-hrggn" podUID="5d01c437-34ee-4d97-a1cc-83037a6fbdaa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" Oct 29 23:18:01.021454 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:18:01.021399 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-dcacb9-86d7d89559-xk4m6" podStartSLOduration=2.021381521 podStartE2EDuration="2.021381521s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:17:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:18:01.004593139 +0000 UTC m=+2661.787806567" watchObservedRunningTime="2026-10-29 23:18:01.021381521 +0000 UTC m=+2661.804594949" Oct 29 23:18:01.021587 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:18:01.021510 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-dcacb9-767c887c7b-jrk26" podStartSLOduration=2.02150467 podStartE2EDuration="2.02150467s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:17:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:18:00.95209646 +0000 UTC m=+2661.735309902" watchObservedRunningTime="2026-10-29 23:18:01.02150467 +0000 UTC m=+2661.804718097" Oct 29 23:18:01.418351 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:18:01.415418 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:18:02.010816 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:18:02.010767 5057 patch_prober.go:29] interesting pod/e2e-inf-a-fba83e-openshift-default-7544fcb869-hrggn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" start-of-body= Oct 29 23:18:02.011031 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:18:02.010843 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-fba83e-openshift-default-7544fcb869-hrggn" podUID="5d01c437-34ee-4d97-a1cc-83037a6fbdaa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" Oct 29 23:18:03.010755 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:18:03.010711 5057 patch_prober.go:29] interesting pod/e2e-inf-a-fba83e-openshift-default-7544fcb869-hrggn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" start-of-body= Oct 29 23:18:03.011263 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:18:03.010776 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-fba83e-openshift-default-7544fcb869-hrggn" podUID="5d01c437-34ee-4d97-a1cc-83037a6fbdaa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" Oct 29 23:18:03.099356 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:18:03.095192 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:18:03.218645 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:18:03.218612 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:18:03.428885 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:18:03.428784 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a9ae14-685b89c579-htlrh" Oct 29 23:18:03.429449 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:18:03.429417 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-a9ae14-75774d5dcb-ddk62" Oct 29 23:18:03.605355 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:18:03.603865 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:18:04.024525 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:18:04.021611 5057 patch_prober.go:29] interesting pod/e2e-inf-a-fba83e-openshift-default-7544fcb869-hrggn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" start-of-body= Oct 29 23:18:04.024525 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:18:04.021711 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-fba83e-openshift-default-7544fcb869-hrggn" podUID="5d01c437-34ee-4d97-a1cc-83037a6fbdaa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" Oct 29 23:18:04.059036 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:18:04.058998 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:18:04.268149 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:18:04.268114 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:18:05.015154 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:18:05.015125 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-fba83e-openshift-default-7544fcb869-hrggn" Oct 29 23:18:05.015725 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:18:05.015704 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-fba83e-openshift-default-7544fcb869-hrggn" Oct 29 23:18:05.016520 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:18:05.016497 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-fba83e-openshift-default-7544fcb869-hrggn" Oct 29 23:18:05.529046 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:18:05.529013 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:18:07.391945 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:18:07.391844 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:18:07.993162 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:18:07.993125 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:18:08.123056 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:18:08.123014 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:18:08.521813 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:18:08.521782 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a37090-84ddb8588c-n4gn5" Oct 29 23:18:08.522223 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:18:08.521867 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-a37090-58c6cbb588-lj97b" Oct 29 23:18:09.225694 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:18:09.225652 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:18:09.731691 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:18:09.731466 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:18:09.841261 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:18:09.841227 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:18:09.895370 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:18:09.895301 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-1a8107-6db566b5cf-6jd5j"] Oct 29 23:18:09.900772 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:18:09.900751 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1a8107-6db566b5cf-6jd5j" Oct 29 23:18:09.922187 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:18:09.922155 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-1a8107-6db566b5cf-6jd5j"] Oct 29 23:18:09.923729 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:18:09.923646 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-1a8107\"" Oct 29 23:18:09.923869 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:18:09.923727 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-1a8107-dockercfg-lhwmf\"" Oct 29 23:18:09.952313 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:18:09.952279 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7q5wb\" (UniqueName: \"kubernetes.io/projected/56e0eac5-6393-4933-a3ed-4e80fd849bf5-kube-api-access-7q5wb\") pod \"payload-pre-processing-e2e-inf-b-1a8107-6db566b5cf-6jd5j\" (UID: \"56e0eac5-6393-4933-a3ed-4e80fd849bf5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1a8107-6db566b5cf-6jd5j" Oct 29 23:18:09.952487 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:18:09.952457 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56e0eac5-6393-4933-a3ed-4e80fd849bf5-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-1a8107-6db566b5cf-6jd5j\" (UID: \"56e0eac5-6393-4933-a3ed-4e80fd849bf5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1a8107-6db566b5cf-6jd5j" Oct 29 23:18:10.063429 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:18:10.063293 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-1a8107-79c55c5798-wmrhd"] Oct 29 23:18:10.094380 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:18:10.068686 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56e0eac5-6393-4933-a3ed-4e80fd849bf5-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-1a8107-6db566b5cf-6jd5j\" (UID: \"56e0eac5-6393-4933-a3ed-4e80fd849bf5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1a8107-6db566b5cf-6jd5j" Oct 29 23:18:10.269360 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:18:10.266782 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7q5wb\" (UniqueName: \"kubernetes.io/projected/56e0eac5-6393-4933-a3ed-4e80fd849bf5-kube-api-access-7q5wb\") pod \"payload-pre-processing-e2e-inf-b-1a8107-6db566b5cf-6jd5j\" (UID: \"56e0eac5-6393-4933-a3ed-4e80fd849bf5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1a8107-6db566b5cf-6jd5j" Oct 29 23:18:10.269360 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:18:10.094297 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56e0eac5-6393-4933-a3ed-4e80fd849bf5-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-1a8107-6db566b5cf-6jd5j\" (UID: \"56e0eac5-6393-4933-a3ed-4e80fd849bf5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1a8107-6db566b5cf-6jd5j" Oct 29 23:18:10.408431 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:18:10.391469 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7q5wb\" (UniqueName: \"kubernetes.io/projected/56e0eac5-6393-4933-a3ed-4e80fd849bf5-kube-api-access-7q5wb\") pod \"payload-pre-processing-e2e-inf-b-1a8107-6db566b5cf-6jd5j\" (UID: \"56e0eac5-6393-4933-a3ed-4e80fd849bf5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1a8107-6db566b5cf-6jd5j" Oct 29 23:18:10.447981 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:18:10.447948 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-1a8107-79c55c5798-wmrhd"] Oct 29 23:18:10.449225 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:18:10.449201 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-1a8107-79c55c5798-wmrhd" Oct 29 23:18:10.524455 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:18:10.524424 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1a8107-6db566b5cf-6jd5j" Oct 29 23:18:10.604989 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:18:10.602311 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a4256ac-b349-4288-a6c9-92b4ed74e3e6-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-1a8107-79c55c5798-wmrhd\" (UID: \"7a4256ac-b349-4288-a6c9-92b4ed74e3e6\") " pod="openshift-ingress/payload-processing-e2e-inf-b-1a8107-79c55c5798-wmrhd" Oct 29 23:18:10.604989 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:18:10.602610 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d58sl\" (UniqueName: \"kubernetes.io/projected/7a4256ac-b349-4288-a6c9-92b4ed74e3e6-kube-api-access-d58sl\") pod \"payload-processing-e2e-inf-b-1a8107-79c55c5798-wmrhd\" (UID: \"7a4256ac-b349-4288-a6c9-92b4ed74e3e6\") " pod="openshift-ingress/payload-processing-e2e-inf-b-1a8107-79c55c5798-wmrhd" Oct 29 23:18:10.704566 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:18:10.704530 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d58sl\" (UniqueName: \"kubernetes.io/projected/7a4256ac-b349-4288-a6c9-92b4ed74e3e6-kube-api-access-d58sl\") pod \"payload-processing-e2e-inf-b-1a8107-79c55c5798-wmrhd\" (UID: \"7a4256ac-b349-4288-a6c9-92b4ed74e3e6\") " pod="openshift-ingress/payload-processing-e2e-inf-b-1a8107-79c55c5798-wmrhd" Oct 29 23:18:10.704794 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:18:10.704772 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a4256ac-b349-4288-a6c9-92b4ed74e3e6-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-1a8107-79c55c5798-wmrhd\" (UID: \"7a4256ac-b349-4288-a6c9-92b4ed74e3e6\") " pod="openshift-ingress/payload-processing-e2e-inf-b-1a8107-79c55c5798-wmrhd" Oct 29 23:18:10.705775 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:18:10.705749 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a4256ac-b349-4288-a6c9-92b4ed74e3e6-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-1a8107-79c55c5798-wmrhd\" (UID: \"7a4256ac-b349-4288-a6c9-92b4ed74e3e6\") " pod="openshift-ingress/payload-processing-e2e-inf-b-1a8107-79c55c5798-wmrhd" Oct 29 23:18:10.727746 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:18:10.727713 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d58sl\" (UniqueName: \"kubernetes.io/projected/7a4256ac-b349-4288-a6c9-92b4ed74e3e6-kube-api-access-d58sl\") pod \"payload-processing-e2e-inf-b-1a8107-79c55c5798-wmrhd\" (UID: \"7a4256ac-b349-4288-a6c9-92b4ed74e3e6\") " pod="openshift-ingress/payload-processing-e2e-inf-b-1a8107-79c55c5798-wmrhd" Oct 29 23:18:10.769963 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:18:10.769933 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-1a8107-79c55c5798-wmrhd" Oct 29 23:18:10.845752 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:18:10.844144 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-1a8107-6db566b5cf-6jd5j"] Oct 29 23:18:10.845752 ip-10-0-140-74 kubenswrapper[5057]: W1029 23:18:10.844269 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56e0eac5_6393_4933_a3ed_4e80fd849bf5.slice/crio-3a32422386f7178883cd2d4741a229b570956e12385ce7aaefb413123ba8cb41 WatchSource:0}: Error finding container 3a32422386f7178883cd2d4741a229b570956e12385ce7aaefb413123ba8cb41: Status 404 returned error can't find the container with id 3a32422386f7178883cd2d4741a229b570956e12385ce7aaefb413123ba8cb41 Oct 29 23:18:11.098664 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:18:11.098632 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-1a8107-79c55c5798-wmrhd"] Oct 29 23:18:11.099756 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:18:11.099728 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:18:11.111106 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:18:11.111081 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-1a8107-79c55c5798-wmrhd" event={"ID":"7a4256ac-b349-4288-a6c9-92b4ed74e3e6","Type":"ContainerStarted","Data":"027af2c9815f3f4181d723844f96b5e94acaea272651de6649c40dfdd58797f2"} Oct 29 23:18:11.114678 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:18:11.114641 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1a8107-6db566b5cf-6jd5j" event={"ID":"56e0eac5-6393-4933-a3ed-4e80fd849bf5","Type":"ContainerStarted","Data":"06c6274d405753bba9393c5739324878e51136a5c2e3db78aa8096cc9f28c564"} Oct 29 23:18:11.114939 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:18:11.114694 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1a8107-6db566b5cf-6jd5j" event={"ID":"56e0eac5-6393-4933-a3ed-4e80fd849bf5","Type":"ContainerStarted","Data":"3a32422386f7178883cd2d4741a229b570956e12385ce7aaefb413123ba8cb41"} Oct 29 23:18:11.114939 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:18:11.114803 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1a8107-6db566b5cf-6jd5j" Oct 29 23:18:11.161358 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:18:11.161302 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1a8107-6db566b5cf-6jd5j" podStartSLOduration=2.161287351 podStartE2EDuration="2.161287351s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:18:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:18:11.154635411 +0000 UTC m=+2671.937848838" watchObservedRunningTime="2026-10-29 23:18:11.161287351 +0000 UTC m=+2671.944500776" Oct 29 23:18:11.209571 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:18:11.209540 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:18:11.343405 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:18:11.338739 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:18:12.125257 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:18:12.125204 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-1a8107-79c55c5798-wmrhd" event={"ID":"7a4256ac-b349-4288-a6c9-92b4ed74e3e6","Type":"ContainerStarted","Data":"2a646f693a7588e1442fe7d7821877f1e595bcad97a1eff4feffbd0266d87083"} Oct 29 23:18:12.125817 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:18:12.125403 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-1a8107-79c55c5798-wmrhd" Oct 29 23:18:12.199352 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:18:12.195073 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-1a8107-79c55c5798-wmrhd" podStartSLOduration=3.195041956 podStartE2EDuration="3.195041956s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:18:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:18:12.194696708 +0000 UTC m=+2672.977910135" watchObservedRunningTime="2026-10-29 23:18:12.195041956 +0000 UTC m=+2672.978255383" Oct 29 23:18:12.969355 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:18:12.968555 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-dcacb9-86d7d89559-xk4m6" Oct 29 23:18:12.989265 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:18:12.979027 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-dcacb9-767c887c7b-jrk26" Oct 29 23:18:13.096354 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:18:13.095700 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:18:15.487433 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:18:15.487397 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:18:17.105826 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:18:17.105782 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:18:18.775597 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:18:18.775564 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:18:19.030649 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:18:19.030561 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d2dnn" containerName="extractor" Oct 29 23:18:20.517011 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:18:20.516966 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:18:22.129840 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:18:22.129812 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1a8107-6db566b5cf-6jd5j" Oct 29 23:18:22.200303 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:18:22.200271 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:18:23.150638 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:18:23.150606 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-1a8107-79c55c5798-wmrhd" Oct 29 23:18:23.718126 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:18:23.718090 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:18:25.429429 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:18:25.429383 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:18:27.044777 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:18:27.044722 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:18:28.761986 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:18:28.761953 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:18:30.464294 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:18:30.464258 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:18:32.129227 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:18:32.129188 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:18:33.470183 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:18:33.470139 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:18:34.208407 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:18:34.208368 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:18:34.680909 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:18:34.680815 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:18:34.826065 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:18:34.826034 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:18:34.962118 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:18:34.962076 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:18:35.917741 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:18:35.917699 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:18:36.478060 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:18:36.478011 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:18:37.610962 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:18:37.610928 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:18:39.318348 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:18:39.318291 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:18:39.864871 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:18:39.864840 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-vj6md_aa6a190e-3283-458a-9dfd-2de7cb193fc5/console-operator/2.log" Oct 29 23:18:39.912522 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:18:39.912495 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-vj6md_aa6a190e-3283-458a-9dfd-2de7cb193fc5/console-operator/2.log" Oct 29 23:18:40.823125 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:18:40.823091 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:18:41.294341 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:18:41.294290 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e17e54f8-rqrcd"] Oct 29 23:18:41.300318 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:18:41.300296 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e17e54f8-rqrcd" Oct 29 23:18:41.324278 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:18:41.324241 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:18:41.325218 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:18:41.325196 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dmx57\"" Oct 29 23:18:41.349955 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:18:41.349920 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e17e54f8-rqrcd"] Oct 29 23:18:41.382809 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:18:41.382766 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40f5dbc7-03cf-4d4f-8257-5dd830f64b96-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e17e54f8-rqrcd\" (UID: \"40f5dbc7-03cf-4d4f-8257-5dd830f64b96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e17e54f8-rqrcd" Oct 29 23:18:41.382984 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:18:41.382917 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40f5dbc7-03cf-4d4f-8257-5dd830f64b96-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e17e54f8-rqrcd\" (UID: \"40f5dbc7-03cf-4d4f-8257-5dd830f64b96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e17e54f8-rqrcd" Oct 29 23:18:41.382984 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:18:41.382969 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2fmf\" (UniqueName: \"kubernetes.io/projected/40f5dbc7-03cf-4d4f-8257-5dd830f64b96-kube-api-access-d2fmf\") pod \"maas-api-revoke-sub-e17e54f8-rqrcd\" (UID: \"40f5dbc7-03cf-4d4f-8257-5dd830f64b96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e17e54f8-rqrcd" Oct 29 23:18:41.485362 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:18:41.485319 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40f5dbc7-03cf-4d4f-8257-5dd830f64b96-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e17e54f8-rqrcd\" (UID: \"40f5dbc7-03cf-4d4f-8257-5dd830f64b96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e17e54f8-rqrcd" Oct 29 23:18:41.485537 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:18:41.485446 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40f5dbc7-03cf-4d4f-8257-5dd830f64b96-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e17e54f8-rqrcd\" (UID: \"40f5dbc7-03cf-4d4f-8257-5dd830f64b96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e17e54f8-rqrcd" Oct 29 23:18:41.485537 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:18:41.485470 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2fmf\" (UniqueName: \"kubernetes.io/projected/40f5dbc7-03cf-4d4f-8257-5dd830f64b96-kube-api-access-d2fmf\") pod \"maas-api-revoke-sub-e17e54f8-rqrcd\" (UID: \"40f5dbc7-03cf-4d4f-8257-5dd830f64b96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e17e54f8-rqrcd" Oct 29 23:18:41.485960 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:18:41.485937 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40f5dbc7-03cf-4d4f-8257-5dd830f64b96-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e17e54f8-rqrcd\" (UID: \"40f5dbc7-03cf-4d4f-8257-5dd830f64b96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e17e54f8-rqrcd" Oct 29 23:18:41.487711 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:18:41.487688 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40f5dbc7-03cf-4d4f-8257-5dd830f64b96-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e17e54f8-rqrcd\" (UID: \"40f5dbc7-03cf-4d4f-8257-5dd830f64b96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e17e54f8-rqrcd" Oct 29 23:18:41.508540 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:18:41.508516 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2fmf\" (UniqueName: \"kubernetes.io/projected/40f5dbc7-03cf-4d4f-8257-5dd830f64b96-kube-api-access-d2fmf\") pod \"maas-api-revoke-sub-e17e54f8-rqrcd\" (UID: \"40f5dbc7-03cf-4d4f-8257-5dd830f64b96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e17e54f8-rqrcd" Oct 29 23:18:41.617461 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:18:41.617372 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e17e54f8-rqrcd" Oct 29 23:18:41.934083 ip-10-0-140-74 kubenswrapper[5057]: W1029 23:18:41.934055 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40f5dbc7_03cf_4d4f_8257_5dd830f64b96.slice/crio-6847aebe84d8ef65f0b882bd59491eb28e3ad43aea24cf216e3a20a66b19fc98 WatchSource:0}: Error finding container 6847aebe84d8ef65f0b882bd59491eb28e3ad43aea24cf216e3a20a66b19fc98: Status 404 returned error can't find the container with id 6847aebe84d8ef65f0b882bd59491eb28e3ad43aea24cf216e3a20a66b19fc98 Oct 29 23:18:41.937516 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:18:41.937476 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e17e54f8-rqrcd"] Oct 29 23:18:42.555735 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:18:42.555696 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e17e54f8-rqrcd" event={"ID":"40f5dbc7-03cf-4d4f-8257-5dd830f64b96","Type":"ContainerStarted","Data":"f5d5cdc1327016cfc7d43a99858ee2558845fc5ab2094d7b066fdb45db2d5836"} Oct 29 23:18:42.555951 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:18:42.555762 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e17e54f8-rqrcd" event={"ID":"40f5dbc7-03cf-4d4f-8257-5dd830f64b96","Type":"ContainerStarted","Data":"6847aebe84d8ef65f0b882bd59491eb28e3ad43aea24cf216e3a20a66b19fc98"} Oct 29 23:18:42.596931 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:18:42.596862 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e17e54f8-rqrcd" podStartSLOduration=1.5968393170000001 podStartE2EDuration="1.596839317s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:18:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:18:42.594552296 +0000 UTC m=+2703.377765723" watchObservedRunningTime="2026-10-29 23:18:42.596839317 +0000 UTC m=+2703.380052747" Oct 29 23:18:42.609473 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:18:42.609438 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:18:43.568202 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:18:43.568160 5057 generic.go:356] "Generic (PLEG): container finished" podID="40f5dbc7-03cf-4d4f-8257-5dd830f64b96" containerID="f5d5cdc1327016cfc7d43a99858ee2558845fc5ab2094d7b066fdb45db2d5836" exitCode=0 Oct 29 23:18:43.568716 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:18:43.568238 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e17e54f8-rqrcd" event={"ID":"40f5dbc7-03cf-4d4f-8257-5dd830f64b96","Type":"ContainerDied","Data":"f5d5cdc1327016cfc7d43a99858ee2558845fc5ab2094d7b066fdb45db2d5836"} Oct 29 23:18:44.251627 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:18:44.251570 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:18:44.383882 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:18:44.380018 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:18:44.775214 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:18:44.775191 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e17e54f8-rqrcd" Oct 29 23:18:44.961484 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:18:44.961447 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/40f5dbc7-03cf-4d4f-8257-5dd830f64b96-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40f5dbc7-03cf-4d4f-8257-5dd830f64b96-maas-api-xks-ca\") pod \"40f5dbc7-03cf-4d4f-8257-5dd830f64b96\" (UID: \"40f5dbc7-03cf-4d4f-8257-5dd830f64b96\") " Oct 29 23:18:44.961709 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:18:44.961692 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40f5dbc7-03cf-4d4f-8257-5dd830f64b96-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40f5dbc7-03cf-4d4f-8257-5dd830f64b96-maas-api-openshift-ca\") pod \"40f5dbc7-03cf-4d4f-8257-5dd830f64b96\" (UID: \"40f5dbc7-03cf-4d4f-8257-5dd830f64b96\") " Oct 29 23:18:44.961792 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:18:44.961758 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40f5dbc7-03cf-4d4f-8257-5dd830f64b96-kube-api-access-d2fmf\" (UniqueName: \"kubernetes.io/projected/40f5dbc7-03cf-4d4f-8257-5dd830f64b96-kube-api-access-d2fmf\") pod \"40f5dbc7-03cf-4d4f-8257-5dd830f64b96\" (UID: \"40f5dbc7-03cf-4d4f-8257-5dd830f64b96\") " Oct 29 23:18:44.962070 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:18:44.962044 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40f5dbc7-03cf-4d4f-8257-5dd830f64b96-maas-api-openshift-ca" pod "40f5dbc7-03cf-4d4f-8257-5dd830f64b96" (UID: "40f5dbc7-03cf-4d4f-8257-5dd830f64b96"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:18:44.962498 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:18:44.962477 5057 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40f5dbc7-03cf-4d4f-8257-5dd830f64b96-maas-api-openshift-ca\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:18:44.963617 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:18:44.963595 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40f5dbc7-03cf-4d4f-8257-5dd830f64b96-maas-api-xks-ca" pod "40f5dbc7-03cf-4d4f-8257-5dd830f64b96" (UID: "40f5dbc7-03cf-4d4f-8257-5dd830f64b96"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:18:44.963856 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:18:44.963831 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40f5dbc7-03cf-4d4f-8257-5dd830f64b96-kube-api-access-d2fmf" pod "40f5dbc7-03cf-4d4f-8257-5dd830f64b96" (UID: "40f5dbc7-03cf-4d4f-8257-5dd830f64b96"). InnerVolumeSpecName "kube-api-access-d2fmf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:18:45.064535 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:18:45.064497 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d2fmf\" (UniqueName: \"kubernetes.io/projected/40f5dbc7-03cf-4d4f-8257-5dd830f64b96-kube-api-access-d2fmf\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:18:45.064535 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:18:45.064534 5057 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40f5dbc7-03cf-4d4f-8257-5dd830f64b96-maas-api-xks-ca\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:18:45.591580 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:18:45.591544 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e17e54f8-rqrcd" Oct 29 23:18:45.591580 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:18:45.591564 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e17e54f8-rqrcd" event={"ID":"40f5dbc7-03cf-4d4f-8257-5dd830f64b96","Type":"ContainerDied","Data":"6847aebe84d8ef65f0b882bd59491eb28e3ad43aea24cf216e3a20a66b19fc98"} Oct 29 23:18:45.591844 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:18:45.591611 5057 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6847aebe84d8ef65f0b882bd59491eb28e3ad43aea24cf216e3a20a66b19fc98" Oct 29 23:18:45.989047 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:18:45.989011 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:18:47.605647 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:18:47.605604 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:18:49.030317 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:18:49.030277 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d2dnn" containerName="extractor" Oct 29 23:18:49.179807 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:18:49.179763 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:18:50.829911 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:18:50.829874 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:18:50.948811 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:18:50.948724 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:18:51.060533 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:18:51.060499 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:18:52.824654 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:18:52.824621 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:18:54.296681 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:18:54.296642 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:18:55.924397 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:18:55.924357 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:18:56.045231 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:18:56.045176 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:18:57.662230 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:18:57.662188 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:18:57.965023 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:18:57.964935 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:18:58.532229 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:18:58.532199 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-845d5270-5vjc9"] Oct 29 23:18:58.533580 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:18:58.533556 5057 cpu_manager.go:414] "containerMap: removing container" podUID="40f5dbc7-03cf-4d4f-8257-5dd830f64b96" containerName="revoke-keys" Oct 29 23:18:58.533580 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:18:58.533578 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40f5dbc7-03cf-4d4f-8257-5dd830f64b96" containerName="revoke-keys" Oct 29 23:18:58.533763 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:18:58.533753 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="40f5dbc7-03cf-4d4f-8257-5dd830f64b96" containerName="revoke-keys" Oct 29 23:18:58.539733 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:18:58.539712 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-845d5270-5vjc9" Oct 29 23:18:58.575128 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:18:58.575084 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-845d5270-5vjc9"] Oct 29 23:18:58.618969 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:18:58.618935 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dmx57\"" Oct 29 23:18:58.619511 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:18:58.619487 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:18:58.648101 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:18:58.648054 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8c139aa-aaf8-4d0f-87b6-17f20c7c258c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-845d5270-5vjc9\" (UID: \"a8c139aa-aaf8-4d0f-87b6-17f20c7c258c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-845d5270-5vjc9" Oct 29 23:18:58.648289 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:18:58.648199 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-blhqb\" (UniqueName: \"kubernetes.io/projected/a8c139aa-aaf8-4d0f-87b6-17f20c7c258c-kube-api-access-blhqb\") pod \"maas-api-revoke-sub-845d5270-5vjc9\" (UID: \"a8c139aa-aaf8-4d0f-87b6-17f20c7c258c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-845d5270-5vjc9" Oct 29 23:18:58.648541 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:18:58.648514 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8c139aa-aaf8-4d0f-87b6-17f20c7c258c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-845d5270-5vjc9\" (UID: \"a8c139aa-aaf8-4d0f-87b6-17f20c7c258c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-845d5270-5vjc9" Oct 29 23:18:58.750547 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:18:58.750506 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8c139aa-aaf8-4d0f-87b6-17f20c7c258c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-845d5270-5vjc9\" (UID: \"a8c139aa-aaf8-4d0f-87b6-17f20c7c258c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-845d5270-5vjc9" Oct 29 23:18:58.750978 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:18:58.750704 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8c139aa-aaf8-4d0f-87b6-17f20c7c258c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-845d5270-5vjc9\" (UID: \"a8c139aa-aaf8-4d0f-87b6-17f20c7c258c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-845d5270-5vjc9" Oct 29 23:18:58.750978 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:18:58.750750 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blhqb\" (UniqueName: \"kubernetes.io/projected/a8c139aa-aaf8-4d0f-87b6-17f20c7c258c-kube-api-access-blhqb\") pod \"maas-api-revoke-sub-845d5270-5vjc9\" (UID: \"a8c139aa-aaf8-4d0f-87b6-17f20c7c258c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-845d5270-5vjc9" Oct 29 23:18:58.751515 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:18:58.751490 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8c139aa-aaf8-4d0f-87b6-17f20c7c258c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-845d5270-5vjc9\" (UID: \"a8c139aa-aaf8-4d0f-87b6-17f20c7c258c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-845d5270-5vjc9" Oct 29 23:18:58.753036 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:18:58.753009 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8c139aa-aaf8-4d0f-87b6-17f20c7c258c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-845d5270-5vjc9\" (UID: \"a8c139aa-aaf8-4d0f-87b6-17f20c7c258c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-845d5270-5vjc9" Oct 29 23:18:58.774486 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:18:58.774459 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-blhqb\" (UniqueName: \"kubernetes.io/projected/a8c139aa-aaf8-4d0f-87b6-17f20c7c258c-kube-api-access-blhqb\") pod \"maas-api-revoke-sub-845d5270-5vjc9\" (UID: \"a8c139aa-aaf8-4d0f-87b6-17f20c7c258c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-845d5270-5vjc9" Oct 29 23:18:58.859757 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:18:58.859682 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-845d5270-5vjc9" Oct 29 23:18:59.177408 ip-10-0-140-74 kubenswrapper[5057]: W1029 23:18:59.177371 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8c139aa_aaf8_4d0f_87b6_17f20c7c258c.slice/crio-f0a70572d6bbbcca58c59d8d85b4acfc8d7051e8353e968b47a774dfa69d71d8 WatchSource:0}: Error finding container f0a70572d6bbbcca58c59d8d85b4acfc8d7051e8353e968b47a774dfa69d71d8: Status 404 returned error can't find the container with id f0a70572d6bbbcca58c59d8d85b4acfc8d7051e8353e968b47a774dfa69d71d8 Oct 29 23:18:59.179956 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:18:59.179932 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-845d5270-5vjc9"] Oct 29 23:18:59.387503 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:18:59.387466 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:18:59.790693 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:18:59.790600 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-845d5270-5vjc9" event={"ID":"a8c139aa-aaf8-4d0f-87b6-17f20c7c258c","Type":"ContainerStarted","Data":"5f7d9ba17db099a54b9f7485a67459c041680ce4f3dd962e9a3386b5b9788f78"} Oct 29 23:18:59.790693 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:18:59.790656 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-845d5270-5vjc9" event={"ID":"a8c139aa-aaf8-4d0f-87b6-17f20c7c258c","Type":"ContainerStarted","Data":"f0a70572d6bbbcca58c59d8d85b4acfc8d7051e8353e968b47a774dfa69d71d8"} Oct 29 23:18:59.832453 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:18:59.832400 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-845d5270-5vjc9" podStartSLOduration=1.832384401 podStartE2EDuration="1.832384401s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:18:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:18:59.829983372 +0000 UTC m=+2720.613196798" watchObservedRunningTime="2026-10-29 23:18:59.832384401 +0000 UTC m=+2720.615597826" Oct 29 23:19:00.803879 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:00.803840 5057 generic.go:356] "Generic (PLEG): container finished" podID="a8c139aa-aaf8-4d0f-87b6-17f20c7c258c" containerID="5f7d9ba17db099a54b9f7485a67459c041680ce4f3dd962e9a3386b5b9788f78" exitCode=0 Oct 29 23:19:00.804431 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:00.803930 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-845d5270-5vjc9" event={"ID":"a8c139aa-aaf8-4d0f-87b6-17f20c7c258c","Type":"ContainerDied","Data":"5f7d9ba17db099a54b9f7485a67459c041680ce4f3dd962e9a3386b5b9788f78"} Oct 29 23:19:01.034503 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:01.034469 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:19:01.202412 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:01.196183 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v2tfm"] Oct 29 23:19:02.015723 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:02.015698 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-845d5270-5vjc9" Oct 29 23:19:02.142458 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:02.142379 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8c139aa-aaf8-4d0f-87b6-17f20c7c258c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8c139aa-aaf8-4d0f-87b6-17f20c7c258c-maas-api-openshift-ca\") pod \"a8c139aa-aaf8-4d0f-87b6-17f20c7c258c\" (UID: \"a8c139aa-aaf8-4d0f-87b6-17f20c7c258c\") " Oct 29 23:19:02.142609 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:02.142511 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a8c139aa-aaf8-4d0f-87b6-17f20c7c258c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8c139aa-aaf8-4d0f-87b6-17f20c7c258c-maas-api-xks-ca\") pod \"a8c139aa-aaf8-4d0f-87b6-17f20c7c258c\" (UID: \"a8c139aa-aaf8-4d0f-87b6-17f20c7c258c\") " Oct 29 23:19:02.142609 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:02.142540 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8c139aa-aaf8-4d0f-87b6-17f20c7c258c-kube-api-access-blhqb\" (UniqueName: \"kubernetes.io/projected/a8c139aa-aaf8-4d0f-87b6-17f20c7c258c-kube-api-access-blhqb\") pod \"a8c139aa-aaf8-4d0f-87b6-17f20c7c258c\" (UID: \"a8c139aa-aaf8-4d0f-87b6-17f20c7c258c\") " Oct 29 23:19:02.142729 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:02.142694 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8c139aa-aaf8-4d0f-87b6-17f20c7c258c-maas-api-openshift-ca" pod "a8c139aa-aaf8-4d0f-87b6-17f20c7c258c" (UID: "a8c139aa-aaf8-4d0f-87b6-17f20c7c258c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:19:02.143305 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:02.143287 5057 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8c139aa-aaf8-4d0f-87b6-17f20c7c258c-maas-api-openshift-ca\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:19:02.144756 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:02.144732 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8c139aa-aaf8-4d0f-87b6-17f20c7c258c-kube-api-access-blhqb" pod "a8c139aa-aaf8-4d0f-87b6-17f20c7c258c" (UID: "a8c139aa-aaf8-4d0f-87b6-17f20c7c258c"). InnerVolumeSpecName "kube-api-access-blhqb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:19:02.144756 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:02.144736 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8c139aa-aaf8-4d0f-87b6-17f20c7c258c-maas-api-xks-ca" pod "a8c139aa-aaf8-4d0f-87b6-17f20c7c258c" (UID: "a8c139aa-aaf8-4d0f-87b6-17f20c7c258c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:19:02.245046 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:02.245009 5057 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8c139aa-aaf8-4d0f-87b6-17f20c7c258c-maas-api-xks-ca\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:19:02.245201 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:02.245050 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-blhqb\" (UniqueName: \"kubernetes.io/projected/a8c139aa-aaf8-4d0f-87b6-17f20c7c258c-kube-api-access-blhqb\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:19:02.826949 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:02.826916 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-845d5270-5vjc9" Oct 29 23:19:02.827157 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:02.826950 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-845d5270-5vjc9" event={"ID":"a8c139aa-aaf8-4d0f-87b6-17f20c7c258c","Type":"ContainerDied","Data":"f0a70572d6bbbcca58c59d8d85b4acfc8d7051e8353e968b47a774dfa69d71d8"} Oct 29 23:19:02.827157 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:02.827008 5057 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f0a70572d6bbbcca58c59d8d85b4acfc8d7051e8353e968b47a774dfa69d71d8" Oct 29 23:19:03.625376 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:03.625289 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:19:05.364491 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:05.364456 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:19:05.483696 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:05.483662 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:19:05.628901 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:05.628809 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:19:07.312236 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:07.312128 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:19:07.711296 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:07.711249 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:19:09.207775 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:09.207725 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:19:10.839761 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:10.839705 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:19:12.867100 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:12.867060 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:19:14.438800 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:14.438756 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:19:16.097135 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:16.097097 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:19:17.616155 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:17.616100 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:19:18.324009 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:18.323949 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:19:19.033287 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:19:19.033248 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d2dnn" containerName="extractor" Oct 29 23:19:19.276268 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:19.276231 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:19:20.947105 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:20.947020 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:19:22.614092 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:22.614050 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:19:23.629927 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:23.629886 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:19:23.950206 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:23.950172 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fba83e-6rkw7"] Oct 29 23:19:23.951402 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:23.951375 5057 cpu_manager.go:414] "containerMap: removing container" podUID="a8c139aa-aaf8-4d0f-87b6-17f20c7c258c" containerName="revoke-keys" Oct 29 23:19:23.951402 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:23.951397 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8c139aa-aaf8-4d0f-87b6-17f20c7c258c" containerName="revoke-keys" Oct 29 23:19:23.951667 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:23.951641 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8c139aa-aaf8-4d0f-87b6-17f20c7c258c" containerName="revoke-keys" Oct 29 23:19:23.956464 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:23.956444 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fba83e-6rkw7" Oct 29 23:19:23.966929 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:23.966901 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e466db5c-d1a0-4b7e-87dc-4a5ffcd23a3b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-fba83e-6rkw7\" (UID: \"e466db5c-d1a0-4b7e-87dc-4a5ffcd23a3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fba83e-6rkw7" Oct 29 23:19:23.967195 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:23.967171 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e466db5c-d1a0-4b7e-87dc-4a5ffcd23a3b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-fba83e-6rkw7\" (UID: \"e466db5c-d1a0-4b7e-87dc-4a5ffcd23a3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fba83e-6rkw7" Oct 29 23:19:23.967363 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:23.967343 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76hkf\" (UniqueName: \"kubernetes.io/projected/e466db5c-d1a0-4b7e-87dc-4a5ffcd23a3b-kube-api-access-76hkf\") pod \"maas-api-revoke-keys-e2e-inf-a-fba83e-6rkw7\" (UID: \"e466db5c-d1a0-4b7e-87dc-4a5ffcd23a3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fba83e-6rkw7" Oct 29 23:19:23.975396 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:23.975366 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fba83e-6rkw7"] Oct 29 23:19:23.978206 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:23.978184 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:19:23.978398 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:23.978285 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dmx57\"" Oct 29 23:19:24.068788 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:24.068744 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76hkf\" (UniqueName: \"kubernetes.io/projected/e466db5c-d1a0-4b7e-87dc-4a5ffcd23a3b-kube-api-access-76hkf\") pod \"maas-api-revoke-keys-e2e-inf-a-fba83e-6rkw7\" (UID: \"e466db5c-d1a0-4b7e-87dc-4a5ffcd23a3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fba83e-6rkw7" Oct 29 23:19:24.068968 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:24.068892 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e466db5c-d1a0-4b7e-87dc-4a5ffcd23a3b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-fba83e-6rkw7\" (UID: \"e466db5c-d1a0-4b7e-87dc-4a5ffcd23a3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fba83e-6rkw7" Oct 29 23:19:24.069169 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:24.069154 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e466db5c-d1a0-4b7e-87dc-4a5ffcd23a3b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-fba83e-6rkw7\" (UID: \"e466db5c-d1a0-4b7e-87dc-4a5ffcd23a3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fba83e-6rkw7" Oct 29 23:19:24.069855 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:24.069825 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e466db5c-d1a0-4b7e-87dc-4a5ffcd23a3b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-fba83e-6rkw7\" (UID: \"e466db5c-d1a0-4b7e-87dc-4a5ffcd23a3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fba83e-6rkw7" Oct 29 23:19:24.071314 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:24.071291 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e466db5c-d1a0-4b7e-87dc-4a5ffcd23a3b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-fba83e-6rkw7\" (UID: \"e466db5c-d1a0-4b7e-87dc-4a5ffcd23a3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fba83e-6rkw7" Oct 29 23:19:24.097617 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:24.097576 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-76hkf\" (UniqueName: \"kubernetes.io/projected/e466db5c-d1a0-4b7e-87dc-4a5ffcd23a3b-kube-api-access-76hkf\") pod \"maas-api-revoke-keys-e2e-inf-a-fba83e-6rkw7\" (UID: \"e466db5c-d1a0-4b7e-87dc-4a5ffcd23a3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fba83e-6rkw7" Oct 29 23:19:24.289847 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:24.289749 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fba83e-6rkw7" Oct 29 23:19:24.606388 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:24.606271 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:19:24.616648 ip-10-0-140-74 kubenswrapper[5057]: W1029 23:19:24.616616 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode466db5c_d1a0_4b7e_87dc_4a5ffcd23a3b.slice/crio-a6f961e7d901a5e7577b4e4ece70f1344c121ccb5edf06dc4f5b8576cba3d9c0 WatchSource:0}: Error finding container a6f961e7d901a5e7577b4e4ece70f1344c121ccb5edf06dc4f5b8576cba3d9c0: Status 404 returned error can't find the container with id a6f961e7d901a5e7577b4e4ece70f1344c121ccb5edf06dc4f5b8576cba3d9c0 Oct 29 23:19:24.617733 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:24.617709 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fba83e-6rkw7"] Oct 29 23:19:25.135399 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:25.135363 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fba83e-6rkw7" event={"ID":"e466db5c-d1a0-4b7e-87dc-4a5ffcd23a3b","Type":"ContainerStarted","Data":"a6f961e7d901a5e7577b4e4ece70f1344c121ccb5edf06dc4f5b8576cba3d9c0"} Oct 29 23:19:26.147786 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:26.147699 5057 generic.go:356] "Generic (PLEG): container finished" podID="e466db5c-d1a0-4b7e-87dc-4a5ffcd23a3b" containerID="481b7d38fc7cfd35eba8cf8f267e267c4656064cf9eda4c3d791042a227b2b9c" exitCode=0 Oct 29 23:19:26.148225 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:26.147781 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fba83e-6rkw7" event={"ID":"e466db5c-d1a0-4b7e-87dc-4a5ffcd23a3b","Type":"ContainerDied","Data":"481b7d38fc7cfd35eba8cf8f267e267c4656064cf9eda4c3d791042a227b2b9c"} Oct 29 23:19:26.408839 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:26.408737 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:19:27.399827 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:27.399795 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fba83e-6rkw7" Oct 29 23:19:27.439381 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:27.439293 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e466db5c-d1a0-4b7e-87dc-4a5ffcd23a3b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e466db5c-d1a0-4b7e-87dc-4a5ffcd23a3b-maas-api-xks-ca\") pod \"e466db5c-d1a0-4b7e-87dc-4a5ffcd23a3b\" (UID: \"e466db5c-d1a0-4b7e-87dc-4a5ffcd23a3b\") " Oct 29 23:19:27.439550 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:27.439410 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e466db5c-d1a0-4b7e-87dc-4a5ffcd23a3b-kube-api-access-76hkf\" (UniqueName: \"kubernetes.io/projected/e466db5c-d1a0-4b7e-87dc-4a5ffcd23a3b-kube-api-access-76hkf\") pod \"e466db5c-d1a0-4b7e-87dc-4a5ffcd23a3b\" (UID: \"e466db5c-d1a0-4b7e-87dc-4a5ffcd23a3b\") " Oct 29 23:19:27.439632 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:27.439597 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e466db5c-d1a0-4b7e-87dc-4a5ffcd23a3b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e466db5c-d1a0-4b7e-87dc-4a5ffcd23a3b-maas-api-openshift-ca\") pod \"e466db5c-d1a0-4b7e-87dc-4a5ffcd23a3b\" (UID: \"e466db5c-d1a0-4b7e-87dc-4a5ffcd23a3b\") " Oct 29 23:19:27.440706 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:27.440564 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e466db5c-d1a0-4b7e-87dc-4a5ffcd23a3b-maas-api-openshift-ca" pod "e466db5c-d1a0-4b7e-87dc-4a5ffcd23a3b" (UID: "e466db5c-d1a0-4b7e-87dc-4a5ffcd23a3b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:19:27.443945 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:27.443143 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e466db5c-d1a0-4b7e-87dc-4a5ffcd23a3b-maas-api-xks-ca" pod "e466db5c-d1a0-4b7e-87dc-4a5ffcd23a3b" (UID: "e466db5c-d1a0-4b7e-87dc-4a5ffcd23a3b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:19:27.444580 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:27.443988 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e466db5c-d1a0-4b7e-87dc-4a5ffcd23a3b-kube-api-access-76hkf" pod "e466db5c-d1a0-4b7e-87dc-4a5ffcd23a3b" (UID: "e466db5c-d1a0-4b7e-87dc-4a5ffcd23a3b"). InnerVolumeSpecName "kube-api-access-76hkf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:19:27.547079 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:27.547043 5057 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e466db5c-d1a0-4b7e-87dc-4a5ffcd23a3b-maas-api-xks-ca\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:19:27.547079 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:27.547075 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-76hkf\" (UniqueName: \"kubernetes.io/projected/e466db5c-d1a0-4b7e-87dc-4a5ffcd23a3b-kube-api-access-76hkf\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:19:27.547380 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:27.547092 5057 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e466db5c-d1a0-4b7e-87dc-4a5ffcd23a3b-maas-api-openshift-ca\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:19:28.007087 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:28.007043 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:19:28.186991 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:28.186960 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fba83e-6rkw7" Oct 29 23:19:28.187259 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:28.186998 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fba83e-6rkw7" event={"ID":"e466db5c-d1a0-4b7e-87dc-4a5ffcd23a3b","Type":"ContainerDied","Data":"a6f961e7d901a5e7577b4e4ece70f1344c121ccb5edf06dc4f5b8576cba3d9c0"} Oct 29 23:19:28.187259 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:28.187058 5057 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a6f961e7d901a5e7577b4e4ece70f1344c121ccb5edf06dc4f5b8576cba3d9c0" Oct 29 23:19:29.654083 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:29.654046 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:19:31.342297 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:31.342265 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:19:31.588898 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:31.588686 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:19:33.126189 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:33.126153 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:19:34.373856 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:34.373818 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-fba83e-6f5cbb76f8-gs9rk"] Oct 29 23:19:34.406802 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:34.406764 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-fba83e-76fd9c4d99-56dzs"] Oct 29 23:19:34.522156 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:34.522120 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:19:34.677411 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:34.670720 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:19:34.714487 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:34.714417 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-4djh7"] Oct 29 23:19:34.717546 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:34.717498 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-4djh7" podUID="3637f82c-7b89-4a17-aa32-5a99d1647969" containerName="istio-proxy" containerID="cri-o://69a57ffaa5960e83b11d95ab053b106f4a165c5cfeab2b1a38f0812f04a96877" gracePeriod=30 Oct 29 23:19:34.736668 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:34.736628 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-4hl56"] Oct 29 23:19:34.736922 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:34.736851 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-4hl56" podUID="d6a73f9e-f3a4-4ed8-ac2e-ff657d266514" containerName="istio-proxy" containerID="cri-o://94a64970649f984d97dda6e5464a9ac044586483af1dcaf6d5805691d5f33105" gracePeriod=30 Oct 29 23:19:34.744700 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:34.744674 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-d6kwq"] Oct 29 23:19:34.744870 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:34.744849 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-d6kwq" podUID="73a11a50-d1bd-4eb5-8685-13e724462b24" containerName="istio-proxy" containerID="cri-o://a811b8b4a0b8f411df2de3f13a231f5e31f54595ff9fd7f756502cb58143437b" gracePeriod=30 Oct 29 23:19:34.885083 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:34.882292 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:19:35.031634 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:35.031588 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:19:36.470492 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:36.470452 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:19:37.341383 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:37.341315 5057 patch_prober.go:29] interesting pod/e2e-worker-w1-e131af-openshift-default-c476c699c-4hl56 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 23:19:37.341571 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:37.341399 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-4hl56" podUID="d6a73f9e-f3a4-4ed8-ac2e-ff657d266514" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 23:19:37.596275 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:37.596188 5057 patch_prober.go:29] interesting pod/e2e-worker-w1-e131af-openshift-default-c476c699c-4djh7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 23:19:37.596868 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:37.596831 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-4djh7" podUID="3637f82c-7b89-4a17-aa32-5a99d1647969" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 23:19:38.085984 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:38.085945 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-g26dp"] Oct 29 23:19:38.086202 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:38.086179 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-g26dp" podUID="c5a6ed02-7acb-46de-9107-81a50942fec6" containerName="istio-proxy" containerID="cri-o://a49f83c0b74f10a2028641bd8083619178b8b43f6810524e831c6bfc19036507" gracePeriod=30 Oct 29 23:19:38.096360 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:38.096315 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-ksjql"] Oct 29 23:19:38.096574 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:38.096547 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-ksjql" podUID="88b81626-da3c-4b1d-b9d8-3dbece193e80" containerName="istio-proxy" containerID="cri-o://7ca2261d53f6a83819eec2b97eca9011fbbe73946a9253be4bb99c6c1f5652a1" gracePeriod=30 Oct 29 23:19:38.156912 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:38.156867 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:19:38.367469 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:38.367381 5057 patch_prober.go:29] interesting pod/e2e-worker-w1-e131af-openshift-default-c476c699c-d6kwq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 23:19:38.367614 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:38.367463 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-d6kwq" podUID="73a11a50-d1bd-4eb5-8685-13e724462b24" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 23:19:39.374936 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:39.374896 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-fba83e-6f5cbb76f8-gs9rk" podUID="56a3f0c8-6d9f-428a-9ba5-f22d397c56ce" containerName="payload-processing" containerID="cri-o://053db1251fba784cbc50d1d01ae7472dbc35a76f1d809de9a78d77e21aa6bfb5" gracePeriod=25 Oct 29 23:19:39.407114 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:39.407073 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fba83e-76fd9c4d99-56dzs" podUID="0d2f8a82-947a-43b7-a4fd-a68dded7c872" containerName="payload-pre-processing" containerID="cri-o://3065b8e517bb24b274141413adcdfde01326c3c2cc9d3e774e27c31873ac50f8" gracePeriod=25 Oct 29 23:19:39.759910 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:39.759872 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:19:40.132603 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:40.129752 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fba83e-76fd9c4d99-56dzs" podUID="0d2f8a82-947a-43b7-a4fd-a68dded7c872" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:19:40.150800 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:40.150768 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-4djh7" Oct 29 23:19:40.227042 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:40.227008 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-d6kwq" Oct 29 23:19:40.234010 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:40.233985 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-4hl56" Oct 29 23:19:40.269867 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:40.269833 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3637f82c-7b89-4a17-aa32-5a99d1647969-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3637f82c-7b89-4a17-aa32-5a99d1647969-workload-socket\") pod \"3637f82c-7b89-4a17-aa32-5a99d1647969\" (UID: \"3637f82c-7b89-4a17-aa32-5a99d1647969\") " Oct 29 23:19:40.270056 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:40.269907 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3637f82c-7b89-4a17-aa32-5a99d1647969-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3637f82c-7b89-4a17-aa32-5a99d1647969-istio-data\") pod \"3637f82c-7b89-4a17-aa32-5a99d1647969\" (UID: \"3637f82c-7b89-4a17-aa32-5a99d1647969\") " Oct 29 23:19:40.270056 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:40.269957 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3637f82c-7b89-4a17-aa32-5a99d1647969-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3637f82c-7b89-4a17-aa32-5a99d1647969-istiod-ca-cert\") pod \"3637f82c-7b89-4a17-aa32-5a99d1647969\" (UID: \"3637f82c-7b89-4a17-aa32-5a99d1647969\") " Oct 29 23:19:40.270189 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:40.270136 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3637f82c-7b89-4a17-aa32-5a99d1647969-workload-socket" pod "3637f82c-7b89-4a17-aa32-5a99d1647969" (UID: "3637f82c-7b89-4a17-aa32-5a99d1647969"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:19:40.270189 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:40.270175 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3637f82c-7b89-4a17-aa32-5a99d1647969-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3637f82c-7b89-4a17-aa32-5a99d1647969-istio-podinfo\") pod \"3637f82c-7b89-4a17-aa32-5a99d1647969\" (UID: \"3637f82c-7b89-4a17-aa32-5a99d1647969\") " Oct 29 23:19:40.270299 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:40.270211 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3637f82c-7b89-4a17-aa32-5a99d1647969-istio-data" pod "3637f82c-7b89-4a17-aa32-5a99d1647969" (UID: "3637f82c-7b89-4a17-aa32-5a99d1647969"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:19:40.270366 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:40.270302 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3637f82c-7b89-4a17-aa32-5a99d1647969-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3637f82c-7b89-4a17-aa32-5a99d1647969-workload-certs\") pod \"3637f82c-7b89-4a17-aa32-5a99d1647969\" (UID: \"3637f82c-7b89-4a17-aa32-5a99d1647969\") " Oct 29 23:19:40.270448 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:40.270426 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3637f82c-7b89-4a17-aa32-5a99d1647969-istiod-ca-cert" pod "3637f82c-7b89-4a17-aa32-5a99d1647969" (UID: "3637f82c-7b89-4a17-aa32-5a99d1647969"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:19:40.270490 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:40.270477 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3637f82c-7b89-4a17-aa32-5a99d1647969-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3637f82c-7b89-4a17-aa32-5a99d1647969-credential-socket\") pod \"3637f82c-7b89-4a17-aa32-5a99d1647969\" (UID: \"3637f82c-7b89-4a17-aa32-5a99d1647969\") " Oct 29 23:19:40.270542 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:40.270527 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3637f82c-7b89-4a17-aa32-5a99d1647969-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3637f82c-7b89-4a17-aa32-5a99d1647969-istio-envoy\") pod \"3637f82c-7b89-4a17-aa32-5a99d1647969\" (UID: \"3637f82c-7b89-4a17-aa32-5a99d1647969\") " Oct 29 23:19:40.270699 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:40.270594 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3637f82c-7b89-4a17-aa32-5a99d1647969-kube-api-access-4mjdk\" (UniqueName: \"kubernetes.io/projected/3637f82c-7b89-4a17-aa32-5a99d1647969-kube-api-access-4mjdk\") pod \"3637f82c-7b89-4a17-aa32-5a99d1647969\" (UID: \"3637f82c-7b89-4a17-aa32-5a99d1647969\") " Oct 29 23:19:40.270699 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:40.270598 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3637f82c-7b89-4a17-aa32-5a99d1647969-workload-certs" pod "3637f82c-7b89-4a17-aa32-5a99d1647969" (UID: "3637f82c-7b89-4a17-aa32-5a99d1647969"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:19:40.270699 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:40.270639 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3637f82c-7b89-4a17-aa32-5a99d1647969-istio-token\" (UniqueName: \"kubernetes.io/projected/3637f82c-7b89-4a17-aa32-5a99d1647969-istio-token\") pod \"3637f82c-7b89-4a17-aa32-5a99d1647969\" (UID: \"3637f82c-7b89-4a17-aa32-5a99d1647969\") " Oct 29 23:19:40.271003 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:40.270769 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3637f82c-7b89-4a17-aa32-5a99d1647969-credential-socket" pod "3637f82c-7b89-4a17-aa32-5a99d1647969" (UID: "3637f82c-7b89-4a17-aa32-5a99d1647969"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:19:40.271588 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:40.271563 5057 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3637f82c-7b89-4a17-aa32-5a99d1647969-workload-certs\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:19:40.271752 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:40.271593 5057 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3637f82c-7b89-4a17-aa32-5a99d1647969-credential-socket\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:19:40.271752 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:40.271606 5057 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3637f82c-7b89-4a17-aa32-5a99d1647969-workload-socket\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:19:40.271752 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:40.271620 5057 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3637f82c-7b89-4a17-aa32-5a99d1647969-istio-data\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:19:40.271752 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:40.271634 5057 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3637f82c-7b89-4a17-aa32-5a99d1647969-istiod-ca-cert\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:19:40.275227 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:40.275202 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3637f82c-7b89-4a17-aa32-5a99d1647969-istio-token" pod "3637f82c-7b89-4a17-aa32-5a99d1647969" (UID: "3637f82c-7b89-4a17-aa32-5a99d1647969"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:19:40.277571 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:40.276429 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3637f82c-7b89-4a17-aa32-5a99d1647969-kube-api-access-4mjdk" pod "3637f82c-7b89-4a17-aa32-5a99d1647969" (UID: "3637f82c-7b89-4a17-aa32-5a99d1647969"). InnerVolumeSpecName "kube-api-access-4mjdk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:19:40.277571 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:40.277481 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3637f82c-7b89-4a17-aa32-5a99d1647969-istio-podinfo" pod "3637f82c-7b89-4a17-aa32-5a99d1647969" (UID: "3637f82c-7b89-4a17-aa32-5a99d1647969"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:19:40.278041 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:40.278015 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3637f82c-7b89-4a17-aa32-5a99d1647969-istio-envoy" pod "3637f82c-7b89-4a17-aa32-5a99d1647969" (UID: "3637f82c-7b89-4a17-aa32-5a99d1647969"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:19:40.373226 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:40.373139 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73a11a50-d1bd-4eb5-8685-13e724462b24-istio-token\" (UniqueName: \"kubernetes.io/projected/73a11a50-d1bd-4eb5-8685-13e724462b24-istio-token\") pod \"73a11a50-d1bd-4eb5-8685-13e724462b24\" (UID: \"73a11a50-d1bd-4eb5-8685-13e724462b24\") " Oct 29 23:19:40.373226 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:40.373183 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73a11a50-d1bd-4eb5-8685-13e724462b24-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73a11a50-d1bd-4eb5-8685-13e724462b24-istio-data\") pod \"73a11a50-d1bd-4eb5-8685-13e724462b24\" (UID: \"73a11a50-d1bd-4eb5-8685-13e724462b24\") " Oct 29 23:19:40.373541 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:40.373227 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6a73f9e-f3a4-4ed8-ac2e-ff657d266514-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6a73f9e-f3a4-4ed8-ac2e-ff657d266514-istio-data\") pod \"d6a73f9e-f3a4-4ed8-ac2e-ff657d266514\" (UID: \"d6a73f9e-f3a4-4ed8-ac2e-ff657d266514\") " Oct 29 23:19:40.373541 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:40.373341 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/73a11a50-d1bd-4eb5-8685-13e724462b24-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73a11a50-d1bd-4eb5-8685-13e724462b24-istio-podinfo\") pod \"73a11a50-d1bd-4eb5-8685-13e724462b24\" (UID: \"73a11a50-d1bd-4eb5-8685-13e724462b24\") " Oct 29 23:19:40.373541 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:40.373383 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73a11a50-d1bd-4eb5-8685-13e724462b24-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73a11a50-d1bd-4eb5-8685-13e724462b24-istio-envoy\") pod \"73a11a50-d1bd-4eb5-8685-13e724462b24\" (UID: \"73a11a50-d1bd-4eb5-8685-13e724462b24\") " Oct 29 23:19:40.373541 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:40.373432 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73a11a50-d1bd-4eb5-8685-13e724462b24-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73a11a50-d1bd-4eb5-8685-13e724462b24-istiod-ca-cert\") pod \"73a11a50-d1bd-4eb5-8685-13e724462b24\" (UID: \"73a11a50-d1bd-4eb5-8685-13e724462b24\") " Oct 29 23:19:40.373541 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:40.373537 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73a11a50-d1bd-4eb5-8685-13e724462b24-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73a11a50-d1bd-4eb5-8685-13e724462b24-credential-socket\") pod \"73a11a50-d1bd-4eb5-8685-13e724462b24\" (UID: \"73a11a50-d1bd-4eb5-8685-13e724462b24\") " Oct 29 23:19:40.373827 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:40.373616 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6a73f9e-f3a4-4ed8-ac2e-ff657d266514-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6a73f9e-f3a4-4ed8-ac2e-ff657d266514-credential-socket\") pod \"d6a73f9e-f3a4-4ed8-ac2e-ff657d266514\" (UID: \"d6a73f9e-f3a4-4ed8-ac2e-ff657d266514\") " Oct 29 23:19:40.373827 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:40.373668 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73a11a50-d1bd-4eb5-8685-13e724462b24-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73a11a50-d1bd-4eb5-8685-13e724462b24-workload-socket\") pod \"73a11a50-d1bd-4eb5-8685-13e724462b24\" (UID: \"73a11a50-d1bd-4eb5-8685-13e724462b24\") " Oct 29 23:19:40.373827 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:40.373761 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6a73f9e-f3a4-4ed8-ac2e-ff657d266514-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6a73f9e-f3a4-4ed8-ac2e-ff657d266514-workload-socket\") pod \"d6a73f9e-f3a4-4ed8-ac2e-ff657d266514\" (UID: \"d6a73f9e-f3a4-4ed8-ac2e-ff657d266514\") " Oct 29 23:19:40.373827 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:40.373800 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73a11a50-d1bd-4eb5-8685-13e724462b24-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73a11a50-d1bd-4eb5-8685-13e724462b24-workload-certs\") pod \"73a11a50-d1bd-4eb5-8685-13e724462b24\" (UID: \"73a11a50-d1bd-4eb5-8685-13e724462b24\") " Oct 29 23:19:40.374029 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:40.373842 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6a73f9e-f3a4-4ed8-ac2e-ff657d266514-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6a73f9e-f3a4-4ed8-ac2e-ff657d266514-workload-certs\") pod \"d6a73f9e-f3a4-4ed8-ac2e-ff657d266514\" (UID: \"d6a73f9e-f3a4-4ed8-ac2e-ff657d266514\") " Oct 29 23:19:40.374029 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:40.373841 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73a11a50-d1bd-4eb5-8685-13e724462b24-istio-data" pod "73a11a50-d1bd-4eb5-8685-13e724462b24" (UID: "73a11a50-d1bd-4eb5-8685-13e724462b24"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:19:40.374029 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:40.373937 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d6a73f9e-f3a4-4ed8-ac2e-ff657d266514-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6a73f9e-f3a4-4ed8-ac2e-ff657d266514-istio-podinfo\") pod \"d6a73f9e-f3a4-4ed8-ac2e-ff657d266514\" (UID: \"d6a73f9e-f3a4-4ed8-ac2e-ff657d266514\") " Oct 29 23:19:40.374210 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:40.374041 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6a73f9e-f3a4-4ed8-ac2e-ff657d266514-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6a73f9e-f3a4-4ed8-ac2e-ff657d266514-istio-envoy\") pod \"d6a73f9e-f3a4-4ed8-ac2e-ff657d266514\" (UID: \"d6a73f9e-f3a4-4ed8-ac2e-ff657d266514\") " Oct 29 23:19:40.374210 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:40.374087 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d6a73f9e-f3a4-4ed8-ac2e-ff657d266514-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6a73f9e-f3a4-4ed8-ac2e-ff657d266514-istiod-ca-cert\") pod \"d6a73f9e-f3a4-4ed8-ac2e-ff657d266514\" (UID: \"d6a73f9e-f3a4-4ed8-ac2e-ff657d266514\") " Oct 29 23:19:40.374210 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:40.374097 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6a73f9e-f3a4-4ed8-ac2e-ff657d266514-credential-socket" pod "d6a73f9e-f3a4-4ed8-ac2e-ff657d266514" (UID: "d6a73f9e-f3a4-4ed8-ac2e-ff657d266514"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:19:40.374210 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:40.374151 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6a73f9e-f3a4-4ed8-ac2e-ff657d266514-kube-api-access-f9qpj\" (UniqueName: \"kubernetes.io/projected/d6a73f9e-f3a4-4ed8-ac2e-ff657d266514-kube-api-access-f9qpj\") pod \"d6a73f9e-f3a4-4ed8-ac2e-ff657d266514\" (UID: \"d6a73f9e-f3a4-4ed8-ac2e-ff657d266514\") " Oct 29 23:19:40.374210 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:40.374178 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6a73f9e-f3a4-4ed8-ac2e-ff657d266514-istio-token\" (UniqueName: \"kubernetes.io/projected/d6a73f9e-f3a4-4ed8-ac2e-ff657d266514-istio-token\") pod \"d6a73f9e-f3a4-4ed8-ac2e-ff657d266514\" (UID: \"d6a73f9e-f3a4-4ed8-ac2e-ff657d266514\") " Oct 29 23:19:40.374504 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:40.374259 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73a11a50-d1bd-4eb5-8685-13e724462b24-kube-api-access-g4rll\" (UniqueName: \"kubernetes.io/projected/73a11a50-d1bd-4eb5-8685-13e724462b24-kube-api-access-g4rll\") pod \"73a11a50-d1bd-4eb5-8685-13e724462b24\" (UID: \"73a11a50-d1bd-4eb5-8685-13e724462b24\") " Oct 29 23:19:40.374504 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:40.374292 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73a11a50-d1bd-4eb5-8685-13e724462b24-credential-socket" pod "73a11a50-d1bd-4eb5-8685-13e724462b24" (UID: "73a11a50-d1bd-4eb5-8685-13e724462b24"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:19:40.374504 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:40.374317 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73a11a50-d1bd-4eb5-8685-13e724462b24-istiod-ca-cert" pod "73a11a50-d1bd-4eb5-8685-13e724462b24" (UID: "73a11a50-d1bd-4eb5-8685-13e724462b24"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:19:40.374871 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:40.374845 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6a73f9e-f3a4-4ed8-ac2e-ff657d266514-workload-socket" pod "d6a73f9e-f3a4-4ed8-ac2e-ff657d266514" (UID: "d6a73f9e-f3a4-4ed8-ac2e-ff657d266514"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:19:40.375293 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:40.375272 5057 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3637f82c-7b89-4a17-aa32-5a99d1647969-istio-envoy\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:19:40.375754 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:40.375301 5057 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73a11a50-d1bd-4eb5-8685-13e724462b24-credential-socket\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:19:40.375754 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:40.375318 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4mjdk\" (UniqueName: \"kubernetes.io/projected/3637f82c-7b89-4a17-aa32-5a99d1647969-kube-api-access-4mjdk\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:19:40.375754 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:40.375373 5057 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3637f82c-7b89-4a17-aa32-5a99d1647969-istio-token\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:19:40.375754 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:40.375388 5057 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6a73f9e-f3a4-4ed8-ac2e-ff657d266514-credential-socket\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:19:40.375754 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:40.375401 5057 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6a73f9e-f3a4-4ed8-ac2e-ff657d266514-workload-socket\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:19:40.375754 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:40.375414 5057 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3637f82c-7b89-4a17-aa32-5a99d1647969-istio-podinfo\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:19:40.375754 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:40.375427 5057 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73a11a50-d1bd-4eb5-8685-13e724462b24-istio-data\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:19:40.375754 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:40.375440 5057 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73a11a50-d1bd-4eb5-8685-13e724462b24-istiod-ca-cert\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:19:40.375754 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:40.375674 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73a11a50-d1bd-4eb5-8685-13e724462b24-workload-socket" pod "73a11a50-d1bd-4eb5-8685-13e724462b24" (UID: "73a11a50-d1bd-4eb5-8685-13e724462b24"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:19:40.376210 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:40.375994 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6a73f9e-f3a4-4ed8-ac2e-ff657d266514-istiod-ca-cert" pod "d6a73f9e-f3a4-4ed8-ac2e-ff657d266514" (UID: "d6a73f9e-f3a4-4ed8-ac2e-ff657d266514"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:19:40.376694 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:40.376356 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6a73f9e-f3a4-4ed8-ac2e-ff657d266514-istio-data" pod "d6a73f9e-f3a4-4ed8-ac2e-ff657d266514" (UID: "d6a73f9e-f3a4-4ed8-ac2e-ff657d266514"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:19:40.377080 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:40.376883 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73a11a50-d1bd-4eb5-8685-13e724462b24-workload-certs" pod "73a11a50-d1bd-4eb5-8685-13e724462b24" (UID: "73a11a50-d1bd-4eb5-8685-13e724462b24"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:19:40.377080 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:40.376935 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d6a73f9e-f3a4-4ed8-ac2e-ff657d266514-istio-podinfo" pod "d6a73f9e-f3a4-4ed8-ac2e-ff657d266514" (UID: "d6a73f9e-f3a4-4ed8-ac2e-ff657d266514"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:19:40.377398 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:40.377352 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6a73f9e-f3a4-4ed8-ac2e-ff657d266514-workload-certs" pod "d6a73f9e-f3a4-4ed8-ac2e-ff657d266514" (UID: "d6a73f9e-f3a4-4ed8-ac2e-ff657d266514"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:19:40.377398 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:40.377322 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73a11a50-d1bd-4eb5-8685-13e724462b24-kube-api-access-g4rll" pod "73a11a50-d1bd-4eb5-8685-13e724462b24" (UID: "73a11a50-d1bd-4eb5-8685-13e724462b24"). InnerVolumeSpecName "kube-api-access-g4rll". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:19:40.377811 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:40.377775 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73a11a50-d1bd-4eb5-8685-13e724462b24-istio-token" pod "73a11a50-d1bd-4eb5-8685-13e724462b24" (UID: "73a11a50-d1bd-4eb5-8685-13e724462b24"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:19:40.378408 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:40.378384 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6a73f9e-f3a4-4ed8-ac2e-ff657d266514-istio-envoy" pod "d6a73f9e-f3a4-4ed8-ac2e-ff657d266514" (UID: "d6a73f9e-f3a4-4ed8-ac2e-ff657d266514"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:19:40.379453 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:40.379411 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6a73f9e-f3a4-4ed8-ac2e-ff657d266514-istio-token" pod "d6a73f9e-f3a4-4ed8-ac2e-ff657d266514" (UID: "d6a73f9e-f3a4-4ed8-ac2e-ff657d266514"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:19:40.379621 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:40.379592 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/73a11a50-d1bd-4eb5-8685-13e724462b24-istio-podinfo" pod "73a11a50-d1bd-4eb5-8685-13e724462b24" (UID: "73a11a50-d1bd-4eb5-8685-13e724462b24"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:19:40.379723 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:40.379617 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73a11a50-d1bd-4eb5-8685-13e724462b24-istio-envoy" pod "73a11a50-d1bd-4eb5-8685-13e724462b24" (UID: "73a11a50-d1bd-4eb5-8685-13e724462b24"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:19:40.380048 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:40.380019 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6a73f9e-f3a4-4ed8-ac2e-ff657d266514-kube-api-access-f9qpj" pod "d6a73f9e-f3a4-4ed8-ac2e-ff657d266514" (UID: "d6a73f9e-f3a4-4ed8-ac2e-ff657d266514"). InnerVolumeSpecName "kube-api-access-f9qpj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:19:40.477674 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:40.477643 5057 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73a11a50-d1bd-4eb5-8685-13e724462b24-istio-podinfo\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:19:40.477674 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:40.477670 5057 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73a11a50-d1bd-4eb5-8685-13e724462b24-istio-envoy\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:19:40.477674 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:40.477678 5057 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73a11a50-d1bd-4eb5-8685-13e724462b24-workload-socket\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:19:40.478010 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:40.477686 5057 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73a11a50-d1bd-4eb5-8685-13e724462b24-workload-certs\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:19:40.478010 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:40.477695 5057 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6a73f9e-f3a4-4ed8-ac2e-ff657d266514-workload-certs\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:19:40.478010 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:40.477702 5057 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6a73f9e-f3a4-4ed8-ac2e-ff657d266514-istio-podinfo\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:19:40.478010 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:40.477710 5057 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6a73f9e-f3a4-4ed8-ac2e-ff657d266514-istio-envoy\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:19:40.478010 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:40.477717 5057 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6a73f9e-f3a4-4ed8-ac2e-ff657d266514-istiod-ca-cert\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:19:40.478010 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:40.477728 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f9qpj\" (UniqueName: \"kubernetes.io/projected/d6a73f9e-f3a4-4ed8-ac2e-ff657d266514-kube-api-access-f9qpj\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:19:40.478010 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:40.477739 5057 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6a73f9e-f3a4-4ed8-ac2e-ff657d266514-istio-token\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:19:40.478010 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:40.477748 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g4rll\" (UniqueName: \"kubernetes.io/projected/73a11a50-d1bd-4eb5-8685-13e724462b24-kube-api-access-g4rll\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:19:40.478010 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:40.477757 5057 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73a11a50-d1bd-4eb5-8685-13e724462b24-istio-token\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:19:40.478010 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:40.477766 5057 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6a73f9e-f3a4-4ed8-ac2e-ff657d266514-istio-data\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:19:40.484891 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:40.484856 5057 generic.go:356] "Generic (PLEG): container finished" podID="d6a73f9e-f3a4-4ed8-ac2e-ff657d266514" containerID="94a64970649f984d97dda6e5464a9ac044586483af1dcaf6d5805691d5f33105" exitCode=0 Oct 29 23:19:40.485050 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:40.484909 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-4hl56" Oct 29 23:19:40.485050 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:40.484941 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-4hl56" event={"ID":"d6a73f9e-f3a4-4ed8-ac2e-ff657d266514","Type":"ContainerDied","Data":"94a64970649f984d97dda6e5464a9ac044586483af1dcaf6d5805691d5f33105"} Oct 29 23:19:40.485050 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:40.485010 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-4hl56" event={"ID":"d6a73f9e-f3a4-4ed8-ac2e-ff657d266514","Type":"ContainerDied","Data":"e8a14577541f109aca0405ac3de2c523839a00b4921b8bfd61c9793cd1299048"} Oct 29 23:19:40.485050 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:40.485039 5057 scope.go:122] "RemoveContainer" containerID="94a64970649f984d97dda6e5464a9ac044586483af1dcaf6d5805691d5f33105" Oct 29 23:19:40.487126 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:40.487100 5057 generic.go:356] "Generic (PLEG): container finished" podID="73a11a50-d1bd-4eb5-8685-13e724462b24" containerID="a811b8b4a0b8f411df2de3f13a231f5e31f54595ff9fd7f756502cb58143437b" exitCode=0 Oct 29 23:19:40.487268 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:40.487170 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-d6kwq" Oct 29 23:19:40.487268 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:40.487187 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-d6kwq" event={"ID":"73a11a50-d1bd-4eb5-8685-13e724462b24","Type":"ContainerDied","Data":"a811b8b4a0b8f411df2de3f13a231f5e31f54595ff9fd7f756502cb58143437b"} Oct 29 23:19:40.487268 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:40.487246 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-d6kwq" event={"ID":"73a11a50-d1bd-4eb5-8685-13e724462b24","Type":"ContainerDied","Data":"3b155c9810fd51aa18d796a6172489415fd9795d42875892b87865685aa277d1"} Oct 29 23:19:40.493479 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:40.493454 5057 generic.go:356] "Generic (PLEG): container finished" podID="3637f82c-7b89-4a17-aa32-5a99d1647969" containerID="69a57ffaa5960e83b11d95ab053b106f4a165c5cfeab2b1a38f0812f04a96877" exitCode=0 Oct 29 23:19:40.493618 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:40.493498 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-4djh7" Oct 29 23:19:40.493722 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:40.493693 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-4djh7" event={"ID":"3637f82c-7b89-4a17-aa32-5a99d1647969","Type":"ContainerDied","Data":"69a57ffaa5960e83b11d95ab053b106f4a165c5cfeab2b1a38f0812f04a96877"} Oct 29 23:19:40.493781 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:40.493737 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-4djh7" event={"ID":"3637f82c-7b89-4a17-aa32-5a99d1647969","Type":"ContainerDied","Data":"11902b12f09c433ec14e784ae489a99cfaa1bd479bc5cb51144749a97d05a502"} Oct 29 23:19:40.502782 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:40.502766 5057 scope.go:122] "RemoveContainer" containerID="94a64970649f984d97dda6e5464a9ac044586483af1dcaf6d5805691d5f33105" Oct 29 23:19:40.503071 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:19:40.503049 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94a64970649f984d97dda6e5464a9ac044586483af1dcaf6d5805691d5f33105\": container with ID starting with 94a64970649f984d97dda6e5464a9ac044586483af1dcaf6d5805691d5f33105 not found: ID does not exist" containerID="94a64970649f984d97dda6e5464a9ac044586483af1dcaf6d5805691d5f33105" Oct 29 23:19:40.503154 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:40.503078 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94a64970649f984d97dda6e5464a9ac044586483af1dcaf6d5805691d5f33105"} err="failed to get container status \"94a64970649f984d97dda6e5464a9ac044586483af1dcaf6d5805691d5f33105\": rpc error: code = NotFound desc = could not find container \"94a64970649f984d97dda6e5464a9ac044586483af1dcaf6d5805691d5f33105\": container with ID starting with 94a64970649f984d97dda6e5464a9ac044586483af1dcaf6d5805691d5f33105 not found: ID does not exist" Oct 29 23:19:40.503154 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:40.503095 5057 scope.go:122] "RemoveContainer" containerID="a811b8b4a0b8f411df2de3f13a231f5e31f54595ff9fd7f756502cb58143437b" Oct 29 23:19:40.514634 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:40.514607 5057 scope.go:122] "RemoveContainer" containerID="a811b8b4a0b8f411df2de3f13a231f5e31f54595ff9fd7f756502cb58143437b" Oct 29 23:19:40.514919 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:19:40.514894 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a811b8b4a0b8f411df2de3f13a231f5e31f54595ff9fd7f756502cb58143437b\": container with ID starting with a811b8b4a0b8f411df2de3f13a231f5e31f54595ff9fd7f756502cb58143437b not found: ID does not exist" containerID="a811b8b4a0b8f411df2de3f13a231f5e31f54595ff9fd7f756502cb58143437b" Oct 29 23:19:40.514983 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:40.514929 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a811b8b4a0b8f411df2de3f13a231f5e31f54595ff9fd7f756502cb58143437b"} err="failed to get container status \"a811b8b4a0b8f411df2de3f13a231f5e31f54595ff9fd7f756502cb58143437b\": rpc error: code = NotFound desc = could not find container \"a811b8b4a0b8f411df2de3f13a231f5e31f54595ff9fd7f756502cb58143437b\": container with ID starting with a811b8b4a0b8f411df2de3f13a231f5e31f54595ff9fd7f756502cb58143437b not found: ID does not exist" Oct 29 23:19:40.514983 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:40.514950 5057 scope.go:122] "RemoveContainer" containerID="69a57ffaa5960e83b11d95ab053b106f4a165c5cfeab2b1a38f0812f04a96877" Oct 29 23:19:40.526521 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:40.526493 5057 scope.go:122] "RemoveContainer" containerID="69a57ffaa5960e83b11d95ab053b106f4a165c5cfeab2b1a38f0812f04a96877" Oct 29 23:19:40.526814 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:19:40.526793 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69a57ffaa5960e83b11d95ab053b106f4a165c5cfeab2b1a38f0812f04a96877\": container with ID starting with 69a57ffaa5960e83b11d95ab053b106f4a165c5cfeab2b1a38f0812f04a96877 not found: ID does not exist" containerID="69a57ffaa5960e83b11d95ab053b106f4a165c5cfeab2b1a38f0812f04a96877" Oct 29 23:19:40.526869 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:40.526821 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69a57ffaa5960e83b11d95ab053b106f4a165c5cfeab2b1a38f0812f04a96877"} err="failed to get container status \"69a57ffaa5960e83b11d95ab053b106f4a165c5cfeab2b1a38f0812f04a96877\": rpc error: code = NotFound desc = could not find container \"69a57ffaa5960e83b11d95ab053b106f4a165c5cfeab2b1a38f0812f04a96877\": container with ID starting with 69a57ffaa5960e83b11d95ab053b106f4a165c5cfeab2b1a38f0812f04a96877 not found: ID does not exist" Oct 29 23:19:40.548504 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:40.548465 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-4hl56"] Oct 29 23:19:40.554451 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:40.554426 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-4hl56"] Oct 29 23:19:40.567455 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:40.567433 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-lfnnh"] Oct 29 23:19:40.567677 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:40.567647 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-lfnnh" podUID="e344401d-57e1-47dc-bf88-dfee4e4b0914" containerName="istio-proxy" containerID="cri-o://0286fb961c72f6b96a5a3fa53564bef1a75256f6dd1408cda4e5d78f2d7d92af" gracePeriod=30 Oct 29 23:19:40.572891 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:40.572870 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-kk86b"] Oct 29 23:19:40.573320 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:40.573129 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-kk86b" podUID="79d262bd-3d59-44ec-b375-28e6c7a57c1c" containerName="istio-proxy" containerID="cri-o://48a12c1f45b59989923a819fd04006a6c7dec61a36e3b7f7579aeffe08f209a0" gracePeriod=30 Oct 29 23:19:40.744224 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:40.744191 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-4djh7"] Oct 29 23:19:40.752559 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:40.752525 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-4djh7"] Oct 29 23:19:40.773715 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:40.773688 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-d6kwq"] Oct 29 23:19:40.782068 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:40.782037 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-d6kwq"] Oct 29 23:19:41.138059 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:41.137959 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-fba83e-6f5cbb76f8-gs9rk" podUID="56a3f0c8-6d9f-428a-9ba5-f22d397c56ce" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:19:41.336065 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:41.336004 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3637f82c-7b89-4a17-aa32-5a99d1647969" path="/var/lib/kubelet/pods/3637f82c-7b89-4a17-aa32-5a99d1647969/volumes" Oct 29 23:19:41.336668 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:41.336651 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73a11a50-d1bd-4eb5-8685-13e724462b24" path="/var/lib/kubelet/pods/73a11a50-d1bd-4eb5-8685-13e724462b24/volumes" Oct 29 23:19:41.337188 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:41.337173 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6a73f9e-f3a4-4ed8-ac2e-ff657d266514" path="/var/lib/kubelet/pods/d6a73f9e-f3a4-4ed8-ac2e-ff657d266514/volumes" Oct 29 23:19:41.435222 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:41.435144 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:19:43.110992 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:43.110959 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:19:43.401240 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:43.401164 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:19:43.505926 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:43.505894 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-g26dp" Oct 29 23:19:43.511235 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:43.511205 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-ksjql" Oct 29 23:19:43.514028 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:43.514000 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:19:43.592499 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:43.592469 5057 generic.go:356] "Generic (PLEG): container finished" podID="c5a6ed02-7acb-46de-9107-81a50942fec6" containerID="a49f83c0b74f10a2028641bd8083619178b8b43f6810524e831c6bfc19036507" exitCode=0 Oct 29 23:19:43.593148 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:43.592602 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-g26dp" event={"ID":"c5a6ed02-7acb-46de-9107-81a50942fec6","Type":"ContainerDied","Data":"a49f83c0b74f10a2028641bd8083619178b8b43f6810524e831c6bfc19036507"} Oct 29 23:19:43.593259 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:43.593226 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-g26dp" event={"ID":"c5a6ed02-7acb-46de-9107-81a50942fec6","Type":"ContainerDied","Data":"862ba5faa0a2630a3d653ca0286736e7fa8c9fb0eb4ca445fffdc47ee218f259"} Oct 29 23:19:43.593324 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:43.593279 5057 scope.go:122] "RemoveContainer" containerID="a49f83c0b74f10a2028641bd8083619178b8b43f6810524e831c6bfc19036507" Oct 29 23:19:43.593429 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:43.592697 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-g26dp" Oct 29 23:19:43.595415 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:43.595392 5057 generic.go:356] "Generic (PLEG): container finished" podID="88b81626-da3c-4b1d-b9d8-3dbece193e80" containerID="7ca2261d53f6a83819eec2b97eca9011fbbe73946a9253be4bb99c6c1f5652a1" exitCode=0 Oct 29 23:19:43.595612 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:43.595449 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-ksjql" Oct 29 23:19:43.595701 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:43.595469 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-ksjql" event={"ID":"88b81626-da3c-4b1d-b9d8-3dbece193e80","Type":"ContainerDied","Data":"7ca2261d53f6a83819eec2b97eca9011fbbe73946a9253be4bb99c6c1f5652a1"} Oct 29 23:19:43.595761 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:43.595723 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-ksjql" event={"ID":"88b81626-da3c-4b1d-b9d8-3dbece193e80","Type":"ContainerDied","Data":"43721d5d5af6fe8fdaa916dd51d681c7dd42aaaf405bd62c9f95f8bf4be90e05"} Oct 29 23:19:43.646770 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:43.646736 5057 scope.go:122] "RemoveContainer" containerID="a49f83c0b74f10a2028641bd8083619178b8b43f6810524e831c6bfc19036507" Oct 29 23:19:43.647095 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:19:43.647063 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a49f83c0b74f10a2028641bd8083619178b8b43f6810524e831c6bfc19036507\": container with ID starting with a49f83c0b74f10a2028641bd8083619178b8b43f6810524e831c6bfc19036507 not found: ID does not exist" containerID="a49f83c0b74f10a2028641bd8083619178b8b43f6810524e831c6bfc19036507" Oct 29 23:19:43.647176 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:43.647106 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a49f83c0b74f10a2028641bd8083619178b8b43f6810524e831c6bfc19036507"} err="failed to get container status \"a49f83c0b74f10a2028641bd8083619178b8b43f6810524e831c6bfc19036507\": rpc error: code = NotFound desc = could not find container \"a49f83c0b74f10a2028641bd8083619178b8b43f6810524e831c6bfc19036507\": container with ID starting with a49f83c0b74f10a2028641bd8083619178b8b43f6810524e831c6bfc19036507 not found: ID does not exist" Oct 29 23:19:43.647176 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:43.647129 5057 scope.go:122] "RemoveContainer" containerID="7ca2261d53f6a83819eec2b97eca9011fbbe73946a9253be4bb99c6c1f5652a1" Oct 29 23:19:43.663650 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:43.663619 5057 scope.go:122] "RemoveContainer" containerID="7ca2261d53f6a83819eec2b97eca9011fbbe73946a9253be4bb99c6c1f5652a1" Oct 29 23:19:43.663887 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:19:43.663861 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ca2261d53f6a83819eec2b97eca9011fbbe73946a9253be4bb99c6c1f5652a1\": container with ID starting with 7ca2261d53f6a83819eec2b97eca9011fbbe73946a9253be4bb99c6c1f5652a1 not found: ID does not exist" containerID="7ca2261d53f6a83819eec2b97eca9011fbbe73946a9253be4bb99c6c1f5652a1" Oct 29 23:19:43.663976 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:43.663896 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ca2261d53f6a83819eec2b97eca9011fbbe73946a9253be4bb99c6c1f5652a1"} err="failed to get container status \"7ca2261d53f6a83819eec2b97eca9011fbbe73946a9253be4bb99c6c1f5652a1\": rpc error: code = NotFound desc = could not find container \"7ca2261d53f6a83819eec2b97eca9011fbbe73946a9253be4bb99c6c1f5652a1\": container with ID starting with 7ca2261d53f6a83819eec2b97eca9011fbbe73946a9253be4bb99c6c1f5652a1 not found: ID does not exist" Oct 29 23:19:43.681840 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:43.681810 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88b81626-da3c-4b1d-b9d8-3dbece193e80-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88b81626-da3c-4b1d-b9d8-3dbece193e80-credential-socket\") pod \"88b81626-da3c-4b1d-b9d8-3dbece193e80\" (UID: \"88b81626-da3c-4b1d-b9d8-3dbece193e80\") " Oct 29 23:19:43.681968 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:43.681928 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c5a6ed02-7acb-46de-9107-81a50942fec6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5a6ed02-7acb-46de-9107-81a50942fec6-istio-podinfo\") pod \"c5a6ed02-7acb-46de-9107-81a50942fec6\" (UID: \"c5a6ed02-7acb-46de-9107-81a50942fec6\") " Oct 29 23:19:43.681968 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:43.681964 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/88b81626-da3c-4b1d-b9d8-3dbece193e80-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88b81626-da3c-4b1d-b9d8-3dbece193e80-istio-podinfo\") pod \"88b81626-da3c-4b1d-b9d8-3dbece193e80\" (UID: \"88b81626-da3c-4b1d-b9d8-3dbece193e80\") " Oct 29 23:19:43.682101 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:43.682017 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88b81626-da3c-4b1d-b9d8-3dbece193e80-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88b81626-da3c-4b1d-b9d8-3dbece193e80-workload-socket\") pod \"88b81626-da3c-4b1d-b9d8-3dbece193e80\" (UID: \"88b81626-da3c-4b1d-b9d8-3dbece193e80\") " Oct 29 23:19:43.682101 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:43.682062 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88b81626-da3c-4b1d-b9d8-3dbece193e80-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88b81626-da3c-4b1d-b9d8-3dbece193e80-istio-data\") pod \"88b81626-da3c-4b1d-b9d8-3dbece193e80\" (UID: \"88b81626-da3c-4b1d-b9d8-3dbece193e80\") " Oct 29 23:19:43.682222 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:43.682089 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88b81626-da3c-4b1d-b9d8-3dbece193e80-credential-socket" pod "88b81626-da3c-4b1d-b9d8-3dbece193e80" (UID: "88b81626-da3c-4b1d-b9d8-3dbece193e80"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:19:43.682222 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:43.682124 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5a6ed02-7acb-46de-9107-81a50942fec6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5a6ed02-7acb-46de-9107-81a50942fec6-istio-envoy\") pod \"c5a6ed02-7acb-46de-9107-81a50942fec6\" (UID: \"c5a6ed02-7acb-46de-9107-81a50942fec6\") " Oct 29 23:19:43.682222 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:43.682162 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5a6ed02-7acb-46de-9107-81a50942fec6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5a6ed02-7acb-46de-9107-81a50942fec6-workload-socket\") pod \"c5a6ed02-7acb-46de-9107-81a50942fec6\" (UID: \"c5a6ed02-7acb-46de-9107-81a50942fec6\") " Oct 29 23:19:43.682443 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:43.682246 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5a6ed02-7acb-46de-9107-81a50942fec6-kube-api-access-6gvr6\" (UniqueName: \"kubernetes.io/projected/c5a6ed02-7acb-46de-9107-81a50942fec6-kube-api-access-6gvr6\") pod \"c5a6ed02-7acb-46de-9107-81a50942fec6\" (UID: \"c5a6ed02-7acb-46de-9107-81a50942fec6\") " Oct 29 23:19:43.682443 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:43.682306 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88b81626-da3c-4b1d-b9d8-3dbece193e80-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88b81626-da3c-4b1d-b9d8-3dbece193e80-workload-certs\") pod \"88b81626-da3c-4b1d-b9d8-3dbece193e80\" (UID: \"88b81626-da3c-4b1d-b9d8-3dbece193e80\") " Oct 29 23:19:43.682443 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:43.682388 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5a6ed02-7acb-46de-9107-81a50942fec6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5a6ed02-7acb-46de-9107-81a50942fec6-workload-certs\") pod \"c5a6ed02-7acb-46de-9107-81a50942fec6\" (UID: \"c5a6ed02-7acb-46de-9107-81a50942fec6\") " Oct 29 23:19:43.682443 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:43.682420 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88b81626-da3c-4b1d-b9d8-3dbece193e80-istio-token\" (UniqueName: \"kubernetes.io/projected/88b81626-da3c-4b1d-b9d8-3dbece193e80-istio-token\") pod \"88b81626-da3c-4b1d-b9d8-3dbece193e80\" (UID: \"88b81626-da3c-4b1d-b9d8-3dbece193e80\") " Oct 29 23:19:43.682665 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:43.682512 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/88b81626-da3c-4b1d-b9d8-3dbece193e80-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88b81626-da3c-4b1d-b9d8-3dbece193e80-istiod-ca-cert\") pod \"88b81626-da3c-4b1d-b9d8-3dbece193e80\" (UID: \"88b81626-da3c-4b1d-b9d8-3dbece193e80\") " Oct 29 23:19:43.682665 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:43.682551 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88b81626-da3c-4b1d-b9d8-3dbece193e80-kube-api-access-qcmjw\" (UniqueName: \"kubernetes.io/projected/88b81626-da3c-4b1d-b9d8-3dbece193e80-kube-api-access-qcmjw\") pod \"88b81626-da3c-4b1d-b9d8-3dbece193e80\" (UID: \"88b81626-da3c-4b1d-b9d8-3dbece193e80\") " Oct 29 23:19:43.682665 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:43.682635 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5a6ed02-7acb-46de-9107-81a50942fec6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5a6ed02-7acb-46de-9107-81a50942fec6-credential-socket\") pod \"c5a6ed02-7acb-46de-9107-81a50942fec6\" (UID: \"c5a6ed02-7acb-46de-9107-81a50942fec6\") " Oct 29 23:19:43.682779 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:43.682716 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5a6ed02-7acb-46de-9107-81a50942fec6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5a6ed02-7acb-46de-9107-81a50942fec6-istio-data\") pod \"c5a6ed02-7acb-46de-9107-81a50942fec6\" (UID: \"c5a6ed02-7acb-46de-9107-81a50942fec6\") " Oct 29 23:19:43.682779 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:43.682741 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5a6ed02-7acb-46de-9107-81a50942fec6-istio-token\" (UniqueName: \"kubernetes.io/projected/c5a6ed02-7acb-46de-9107-81a50942fec6-istio-token\") pod \"c5a6ed02-7acb-46de-9107-81a50942fec6\" (UID: \"c5a6ed02-7acb-46de-9107-81a50942fec6\") " Oct 29 23:19:43.682860 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:43.682793 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88b81626-da3c-4b1d-b9d8-3dbece193e80-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88b81626-da3c-4b1d-b9d8-3dbece193e80-istio-envoy\") pod \"88b81626-da3c-4b1d-b9d8-3dbece193e80\" (UID: \"88b81626-da3c-4b1d-b9d8-3dbece193e80\") " Oct 29 23:19:43.682896 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:43.682863 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5a6ed02-7acb-46de-9107-81a50942fec6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5a6ed02-7acb-46de-9107-81a50942fec6-istiod-ca-cert\") pod \"c5a6ed02-7acb-46de-9107-81a50942fec6\" (UID: \"c5a6ed02-7acb-46de-9107-81a50942fec6\") " Oct 29 23:19:43.684473 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:43.684447 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88b81626-da3c-4b1d-b9d8-3dbece193e80-workload-socket" pod "88b81626-da3c-4b1d-b9d8-3dbece193e80" (UID: "88b81626-da3c-4b1d-b9d8-3dbece193e80"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:19:43.684647 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:43.684626 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88b81626-da3c-4b1d-b9d8-3dbece193e80-istio-data" pod "88b81626-da3c-4b1d-b9d8-3dbece193e80" (UID: "88b81626-da3c-4b1d-b9d8-3dbece193e80"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:19:43.684940 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:43.684919 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5a6ed02-7acb-46de-9107-81a50942fec6-workload-certs" pod "c5a6ed02-7acb-46de-9107-81a50942fec6" (UID: "c5a6ed02-7acb-46de-9107-81a50942fec6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:19:43.685124 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:43.685102 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5a6ed02-7acb-46de-9107-81a50942fec6-workload-socket" pod "c5a6ed02-7acb-46de-9107-81a50942fec6" (UID: "c5a6ed02-7acb-46de-9107-81a50942fec6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:19:43.685890 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:43.685195 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/88b81626-da3c-4b1d-b9d8-3dbece193e80-istio-podinfo" pod "88b81626-da3c-4b1d-b9d8-3dbece193e80" (UID: "88b81626-da3c-4b1d-b9d8-3dbece193e80"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:19:43.685890 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:43.685453 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5a6ed02-7acb-46de-9107-81a50942fec6-credential-socket" pod "c5a6ed02-7acb-46de-9107-81a50942fec6" (UID: "c5a6ed02-7acb-46de-9107-81a50942fec6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:19:43.685890 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:43.685647 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88b81626-da3c-4b1d-b9d8-3dbece193e80-istiod-ca-cert" pod "88b81626-da3c-4b1d-b9d8-3dbece193e80" (UID: "88b81626-da3c-4b1d-b9d8-3dbece193e80"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:19:43.685890 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:43.685739 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c5a6ed02-7acb-46de-9107-81a50942fec6-istio-podinfo" pod "c5a6ed02-7acb-46de-9107-81a50942fec6" (UID: "c5a6ed02-7acb-46de-9107-81a50942fec6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:19:43.686455 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:43.686364 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5a6ed02-7acb-46de-9107-81a50942fec6-istio-data" pod "c5a6ed02-7acb-46de-9107-81a50942fec6" (UID: "c5a6ed02-7acb-46de-9107-81a50942fec6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:19:43.686589 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:43.686566 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88b81626-da3c-4b1d-b9d8-3dbece193e80-workload-certs" pod "88b81626-da3c-4b1d-b9d8-3dbece193e80" (UID: "88b81626-da3c-4b1d-b9d8-3dbece193e80"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:19:43.687508 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:43.687472 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5a6ed02-7acb-46de-9107-81a50942fec6-istiod-ca-cert" pod "c5a6ed02-7acb-46de-9107-81a50942fec6" (UID: "c5a6ed02-7acb-46de-9107-81a50942fec6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:19:43.688259 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:43.688231 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88b81626-da3c-4b1d-b9d8-3dbece193e80-kube-api-access-qcmjw" pod "88b81626-da3c-4b1d-b9d8-3dbece193e80" (UID: "88b81626-da3c-4b1d-b9d8-3dbece193e80"). InnerVolumeSpecName "kube-api-access-qcmjw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:19:43.689043 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:43.689005 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5a6ed02-7acb-46de-9107-81a50942fec6-istio-envoy" pod "c5a6ed02-7acb-46de-9107-81a50942fec6" (UID: "c5a6ed02-7acb-46de-9107-81a50942fec6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:19:43.689043 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:43.689030 5057 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5a6ed02-7acb-46de-9107-81a50942fec6-credential-socket\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:19:43.689212 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:43.689055 5057 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5a6ed02-7acb-46de-9107-81a50942fec6-istio-data\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:19:43.689212 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:43.689070 5057 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5a6ed02-7acb-46de-9107-81a50942fec6-istiod-ca-cert\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:19:43.689212 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:43.689085 5057 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88b81626-da3c-4b1d-b9d8-3dbece193e80-credential-socket\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:19:43.689212 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:43.689079 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5a6ed02-7acb-46de-9107-81a50942fec6-istio-token" pod "c5a6ed02-7acb-46de-9107-81a50942fec6" (UID: "c5a6ed02-7acb-46de-9107-81a50942fec6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:19:43.689212 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:43.689098 5057 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5a6ed02-7acb-46de-9107-81a50942fec6-istio-podinfo\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:19:43.689212 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:43.689113 5057 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88b81626-da3c-4b1d-b9d8-3dbece193e80-istio-podinfo\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:19:43.689212 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:43.689127 5057 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88b81626-da3c-4b1d-b9d8-3dbece193e80-workload-socket\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:19:43.689212 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:43.689140 5057 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88b81626-da3c-4b1d-b9d8-3dbece193e80-istio-data\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:19:43.689212 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:43.689154 5057 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5a6ed02-7acb-46de-9107-81a50942fec6-workload-socket\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:19:43.689212 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:43.689162 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88b81626-da3c-4b1d-b9d8-3dbece193e80-istio-token" pod "88b81626-da3c-4b1d-b9d8-3dbece193e80" (UID: "88b81626-da3c-4b1d-b9d8-3dbece193e80"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:19:43.689212 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:43.689167 5057 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88b81626-da3c-4b1d-b9d8-3dbece193e80-workload-certs\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:19:43.689212 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:43.689197 5057 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5a6ed02-7acb-46de-9107-81a50942fec6-workload-certs\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:19:43.689212 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:43.689211 5057 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88b81626-da3c-4b1d-b9d8-3dbece193e80-istiod-ca-cert\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:19:43.689843 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:43.689226 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qcmjw\" (UniqueName: \"kubernetes.io/projected/88b81626-da3c-4b1d-b9d8-3dbece193e80-kube-api-access-qcmjw\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:19:43.689843 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:43.689469 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5a6ed02-7acb-46de-9107-81a50942fec6-kube-api-access-6gvr6" pod "c5a6ed02-7acb-46de-9107-81a50942fec6" (UID: "c5a6ed02-7acb-46de-9107-81a50942fec6"). InnerVolumeSpecName "kube-api-access-6gvr6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:19:43.690538 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:43.690513 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88b81626-da3c-4b1d-b9d8-3dbece193e80-istio-envoy" pod "88b81626-da3c-4b1d-b9d8-3dbece193e80" (UID: "88b81626-da3c-4b1d-b9d8-3dbece193e80"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:19:43.791854 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:43.791815 5057 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5a6ed02-7acb-46de-9107-81a50942fec6-istio-envoy\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:19:43.791854 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:43.791849 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6gvr6\" (UniqueName: \"kubernetes.io/projected/c5a6ed02-7acb-46de-9107-81a50942fec6-kube-api-access-6gvr6\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:19:43.792112 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:43.791864 5057 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88b81626-da3c-4b1d-b9d8-3dbece193e80-istio-token\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:19:43.792112 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:43.791877 5057 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5a6ed02-7acb-46de-9107-81a50942fec6-istio-token\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:19:43.792112 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:43.791890 5057 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88b81626-da3c-4b1d-b9d8-3dbece193e80-istio-envoy\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:19:43.949732 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:43.949701 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-g26dp"] Oct 29 23:19:43.960753 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:43.960727 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-g26dp"] Oct 29 23:19:43.988415 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:43.988380 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-ksjql"] Oct 29 23:19:43.992144 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:43.992122 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-ksjql"] Oct 29 23:19:44.221851 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:44.221773 5057 patch_prober.go:29] interesting pod/e2e-worker-w4-063b46-openshift-default-799d8d64cf-lfnnh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 23:19:44.221851 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:44.221833 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-lfnnh" podUID="e344401d-57e1-47dc-bf88-dfee4e4b0914" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 23:19:44.798546 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:44.797213 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:19:45.334941 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:45.334900 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="88b81626-da3c-4b1d-b9d8-3dbece193e80" path="/var/lib/kubelet/pods/88b81626-da3c-4b1d-b9d8-3dbece193e80/volumes" Oct 29 23:19:45.336052 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:45.335699 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5a6ed02-7acb-46de-9107-81a50942fec6" path="/var/lib/kubelet/pods/c5a6ed02-7acb-46de-9107-81a50942fec6/volumes" Oct 29 23:19:45.816351 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:45.816124 5057 generic.go:356] "Generic (PLEG): container finished" podID="79d262bd-3d59-44ec-b375-28e6c7a57c1c" containerID="48a12c1f45b59989923a819fd04006a6c7dec61a36e3b7f7579aeffe08f209a0" exitCode=0 Oct 29 23:19:45.819903 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:45.816544 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-kk86b" event={"ID":"79d262bd-3d59-44ec-b375-28e6c7a57c1c","Type":"ContainerDied","Data":"48a12c1f45b59989923a819fd04006a6c7dec61a36e3b7f7579aeffe08f209a0"} Oct 29 23:19:45.819903 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:45.818625 5057 generic.go:356] "Generic (PLEG): container finished" podID="e344401d-57e1-47dc-bf88-dfee4e4b0914" containerID="0286fb961c72f6b96a5a3fa53564bef1a75256f6dd1408cda4e5d78f2d7d92af" exitCode=0 Oct 29 23:19:45.819903 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:45.818656 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-lfnnh" event={"ID":"e344401d-57e1-47dc-bf88-dfee4e4b0914","Type":"ContainerDied","Data":"0286fb961c72f6b96a5a3fa53564bef1a75256f6dd1408cda4e5d78f2d7d92af"} Oct 29 23:19:45.854287 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:45.854027 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:19:45.921047 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:45.909825 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-lfnnh" Oct 29 23:19:45.975721 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:45.975680 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e344401d-57e1-47dc-bf88-dfee4e4b0914-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e344401d-57e1-47dc-bf88-dfee4e4b0914-istiod-ca-cert\") pod \"e344401d-57e1-47dc-bf88-dfee4e4b0914\" (UID: \"e344401d-57e1-47dc-bf88-dfee4e4b0914\") " Oct 29 23:19:45.975923 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:45.975778 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e344401d-57e1-47dc-bf88-dfee4e4b0914-kube-api-access-j8htl\" (UniqueName: \"kubernetes.io/projected/e344401d-57e1-47dc-bf88-dfee4e4b0914-kube-api-access-j8htl\") pod \"e344401d-57e1-47dc-bf88-dfee4e4b0914\" (UID: \"e344401d-57e1-47dc-bf88-dfee4e4b0914\") " Oct 29 23:19:45.975923 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:45.975855 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e344401d-57e1-47dc-bf88-dfee4e4b0914-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e344401d-57e1-47dc-bf88-dfee4e4b0914-credential-socket\") pod \"e344401d-57e1-47dc-bf88-dfee4e4b0914\" (UID: \"e344401d-57e1-47dc-bf88-dfee4e4b0914\") " Oct 29 23:19:45.975923 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:45.975890 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e344401d-57e1-47dc-bf88-dfee4e4b0914-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e344401d-57e1-47dc-bf88-dfee4e4b0914-workload-socket\") pod \"e344401d-57e1-47dc-bf88-dfee4e4b0914\" (UID: \"e344401d-57e1-47dc-bf88-dfee4e4b0914\") " Oct 29 23:19:45.976112 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:45.975988 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e344401d-57e1-47dc-bf88-dfee4e4b0914-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e344401d-57e1-47dc-bf88-dfee4e4b0914-istio-data\") pod \"e344401d-57e1-47dc-bf88-dfee4e4b0914\" (UID: \"e344401d-57e1-47dc-bf88-dfee4e4b0914\") " Oct 29 23:19:45.976173 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:45.976144 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e344401d-57e1-47dc-bf88-dfee4e4b0914-istio-token\" (UniqueName: \"kubernetes.io/projected/e344401d-57e1-47dc-bf88-dfee4e4b0914-istio-token\") pod \"e344401d-57e1-47dc-bf88-dfee4e4b0914\" (UID: \"e344401d-57e1-47dc-bf88-dfee4e4b0914\") " Oct 29 23:19:45.980872 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:45.980839 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e344401d-57e1-47dc-bf88-dfee4e4b0914-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e344401d-57e1-47dc-bf88-dfee4e4b0914-istio-podinfo\") pod \"e344401d-57e1-47dc-bf88-dfee4e4b0914\" (UID: \"e344401d-57e1-47dc-bf88-dfee4e4b0914\") " Oct 29 23:19:45.981014 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:45.980925 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e344401d-57e1-47dc-bf88-dfee4e4b0914-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e344401d-57e1-47dc-bf88-dfee4e4b0914-istio-envoy\") pod \"e344401d-57e1-47dc-bf88-dfee4e4b0914\" (UID: \"e344401d-57e1-47dc-bf88-dfee4e4b0914\") " Oct 29 23:19:45.981014 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:45.981008 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e344401d-57e1-47dc-bf88-dfee4e4b0914-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e344401d-57e1-47dc-bf88-dfee4e4b0914-workload-certs\") pod \"e344401d-57e1-47dc-bf88-dfee4e4b0914\" (UID: \"e344401d-57e1-47dc-bf88-dfee4e4b0914\") " Oct 29 23:19:45.982201 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:45.982170 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e344401d-57e1-47dc-bf88-dfee4e4b0914-workload-certs" pod "e344401d-57e1-47dc-bf88-dfee4e4b0914" (UID: "e344401d-57e1-47dc-bf88-dfee4e4b0914"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:19:45.982583 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:45.982558 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e344401d-57e1-47dc-bf88-dfee4e4b0914-istiod-ca-cert" pod "e344401d-57e1-47dc-bf88-dfee4e4b0914" (UID: "e344401d-57e1-47dc-bf88-dfee4e4b0914"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:19:45.984178 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:45.984149 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e344401d-57e1-47dc-bf88-dfee4e4b0914-credential-socket" pod "e344401d-57e1-47dc-bf88-dfee4e4b0914" (UID: "e344401d-57e1-47dc-bf88-dfee4e4b0914"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:19:45.984285 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:45.984225 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e344401d-57e1-47dc-bf88-dfee4e4b0914-istio-data" pod "e344401d-57e1-47dc-bf88-dfee4e4b0914" (UID: "e344401d-57e1-47dc-bf88-dfee4e4b0914"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:19:45.984616 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:45.984579 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e344401d-57e1-47dc-bf88-dfee4e4b0914-workload-socket" pod "e344401d-57e1-47dc-bf88-dfee4e4b0914" (UID: "e344401d-57e1-47dc-bf88-dfee4e4b0914"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:19:45.985919 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:45.985892 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e344401d-57e1-47dc-bf88-dfee4e4b0914-kube-api-access-j8htl" pod "e344401d-57e1-47dc-bf88-dfee4e4b0914" (UID: "e344401d-57e1-47dc-bf88-dfee4e4b0914"). InnerVolumeSpecName "kube-api-access-j8htl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:19:46.008041 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:46.006025 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e344401d-57e1-47dc-bf88-dfee4e4b0914-istio-podinfo" pod "e344401d-57e1-47dc-bf88-dfee4e4b0914" (UID: "e344401d-57e1-47dc-bf88-dfee4e4b0914"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:19:46.008041 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:46.006243 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e344401d-57e1-47dc-bf88-dfee4e4b0914-istio-token" pod "e344401d-57e1-47dc-bf88-dfee4e4b0914" (UID: "e344401d-57e1-47dc-bf88-dfee4e4b0914"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:19:46.009469 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:46.009432 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e344401d-57e1-47dc-bf88-dfee4e4b0914-istio-envoy" pod "e344401d-57e1-47dc-bf88-dfee4e4b0914" (UID: "e344401d-57e1-47dc-bf88-dfee4e4b0914"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:19:46.020109 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:46.019810 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-kk86b" Oct 29 23:19:46.084379 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:46.083138 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79d262bd-3d59-44ec-b375-28e6c7a57c1c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79d262bd-3d59-44ec-b375-28e6c7a57c1c-credential-socket\") pod \"79d262bd-3d59-44ec-b375-28e6c7a57c1c\" (UID: \"79d262bd-3d59-44ec-b375-28e6c7a57c1c\") " Oct 29 23:19:46.084379 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:46.083240 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79d262bd-3d59-44ec-b375-28e6c7a57c1c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79d262bd-3d59-44ec-b375-28e6c7a57c1c-workload-socket\") pod \"79d262bd-3d59-44ec-b375-28e6c7a57c1c\" (UID: \"79d262bd-3d59-44ec-b375-28e6c7a57c1c\") " Oct 29 23:19:46.084379 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:46.083348 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79d262bd-3d59-44ec-b375-28e6c7a57c1c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79d262bd-3d59-44ec-b375-28e6c7a57c1c-workload-certs\") pod \"79d262bd-3d59-44ec-b375-28e6c7a57c1c\" (UID: \"79d262bd-3d59-44ec-b375-28e6c7a57c1c\") " Oct 29 23:19:46.084379 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:46.083419 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79d262bd-3d59-44ec-b375-28e6c7a57c1c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79d262bd-3d59-44ec-b375-28e6c7a57c1c-istio-envoy\") pod \"79d262bd-3d59-44ec-b375-28e6c7a57c1c\" (UID: \"79d262bd-3d59-44ec-b375-28e6c7a57c1c\") " Oct 29 23:19:46.084379 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:46.083520 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79d262bd-3d59-44ec-b375-28e6c7a57c1c-istio-token\" (UniqueName: \"kubernetes.io/projected/79d262bd-3d59-44ec-b375-28e6c7a57c1c-istio-token\") pod \"79d262bd-3d59-44ec-b375-28e6c7a57c1c\" (UID: \"79d262bd-3d59-44ec-b375-28e6c7a57c1c\") " Oct 29 23:19:46.084379 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:46.083669 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/79d262bd-3d59-44ec-b375-28e6c7a57c1c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79d262bd-3d59-44ec-b375-28e6c7a57c1c-istio-podinfo\") pod \"79d262bd-3d59-44ec-b375-28e6c7a57c1c\" (UID: \"79d262bd-3d59-44ec-b375-28e6c7a57c1c\") " Oct 29 23:19:46.084379 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:46.083788 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79d262bd-3d59-44ec-b375-28e6c7a57c1c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79d262bd-3d59-44ec-b375-28e6c7a57c1c-istio-data\") pod \"79d262bd-3d59-44ec-b375-28e6c7a57c1c\" (UID: \"79d262bd-3d59-44ec-b375-28e6c7a57c1c\") " Oct 29 23:19:46.084379 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:46.083839 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79d262bd-3d59-44ec-b375-28e6c7a57c1c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79d262bd-3d59-44ec-b375-28e6c7a57c1c-istiod-ca-cert\") pod \"79d262bd-3d59-44ec-b375-28e6c7a57c1c\" (UID: \"79d262bd-3d59-44ec-b375-28e6c7a57c1c\") " Oct 29 23:19:46.084379 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:46.083866 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79d262bd-3d59-44ec-b375-28e6c7a57c1c-kube-api-access-wnnr9\" (UniqueName: \"kubernetes.io/projected/79d262bd-3d59-44ec-b375-28e6c7a57c1c-kube-api-access-wnnr9\") pod \"79d262bd-3d59-44ec-b375-28e6c7a57c1c\" (UID: \"79d262bd-3d59-44ec-b375-28e6c7a57c1c\") " Oct 29 23:19:46.085449 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:46.085419 5057 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e344401d-57e1-47dc-bf88-dfee4e4b0914-credential-socket\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:19:46.085449 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:46.085452 5057 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e344401d-57e1-47dc-bf88-dfee4e4b0914-workload-socket\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:19:46.085643 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:46.085467 5057 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e344401d-57e1-47dc-bf88-dfee4e4b0914-istio-data\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:19:46.085643 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:46.085480 5057 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e344401d-57e1-47dc-bf88-dfee4e4b0914-istio-token\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:19:46.085643 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:46.085492 5057 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e344401d-57e1-47dc-bf88-dfee4e4b0914-istio-podinfo\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:19:46.085643 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:46.085504 5057 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e344401d-57e1-47dc-bf88-dfee4e4b0914-istio-envoy\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:19:46.085643 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:46.085515 5057 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e344401d-57e1-47dc-bf88-dfee4e4b0914-workload-certs\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:19:46.085643 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:46.085526 5057 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e344401d-57e1-47dc-bf88-dfee4e4b0914-istiod-ca-cert\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:19:46.085643 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:46.085539 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j8htl\" (UniqueName: \"kubernetes.io/projected/e344401d-57e1-47dc-bf88-dfee4e4b0914-kube-api-access-j8htl\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:19:46.085996 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:46.085930 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79d262bd-3d59-44ec-b375-28e6c7a57c1c-credential-socket" pod "79d262bd-3d59-44ec-b375-28e6c7a57c1c" (UID: "79d262bd-3d59-44ec-b375-28e6c7a57c1c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:19:46.086970 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:46.086103 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79d262bd-3d59-44ec-b375-28e6c7a57c1c-workload-certs" pod "79d262bd-3d59-44ec-b375-28e6c7a57c1c" (UID: "79d262bd-3d59-44ec-b375-28e6c7a57c1c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:19:46.086970 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:46.086583 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79d262bd-3d59-44ec-b375-28e6c7a57c1c-istio-data" pod "79d262bd-3d59-44ec-b375-28e6c7a57c1c" (UID: "79d262bd-3d59-44ec-b375-28e6c7a57c1c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:19:46.086970 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:46.086925 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79d262bd-3d59-44ec-b375-28e6c7a57c1c-istiod-ca-cert" pod "79d262bd-3d59-44ec-b375-28e6c7a57c1c" (UID: "79d262bd-3d59-44ec-b375-28e6c7a57c1c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:19:46.088172 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:46.088142 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79d262bd-3d59-44ec-b375-28e6c7a57c1c-kube-api-access-wnnr9" pod "79d262bd-3d59-44ec-b375-28e6c7a57c1c" (UID: "79d262bd-3d59-44ec-b375-28e6c7a57c1c"). InnerVolumeSpecName "kube-api-access-wnnr9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:19:46.089283 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:46.089228 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79d262bd-3d59-44ec-b375-28e6c7a57c1c-istio-envoy" pod "79d262bd-3d59-44ec-b375-28e6c7a57c1c" (UID: "79d262bd-3d59-44ec-b375-28e6c7a57c1c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:19:46.089444 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:46.089354 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79d262bd-3d59-44ec-b375-28e6c7a57c1c-workload-socket" pod "79d262bd-3d59-44ec-b375-28e6c7a57c1c" (UID: "79d262bd-3d59-44ec-b375-28e6c7a57c1c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:19:46.097136 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:46.097095 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/79d262bd-3d59-44ec-b375-28e6c7a57c1c-istio-podinfo" pod "79d262bd-3d59-44ec-b375-28e6c7a57c1c" (UID: "79d262bd-3d59-44ec-b375-28e6c7a57c1c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:19:46.099535 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:46.099497 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79d262bd-3d59-44ec-b375-28e6c7a57c1c-istio-token" pod "79d262bd-3d59-44ec-b375-28e6c7a57c1c" (UID: "79d262bd-3d59-44ec-b375-28e6c7a57c1c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:19:46.187295 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:46.187199 5057 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79d262bd-3d59-44ec-b375-28e6c7a57c1c-istiod-ca-cert\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:19:46.187295 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:46.187236 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wnnr9\" (UniqueName: \"kubernetes.io/projected/79d262bd-3d59-44ec-b375-28e6c7a57c1c-kube-api-access-wnnr9\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:19:46.187295 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:46.187247 5057 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79d262bd-3d59-44ec-b375-28e6c7a57c1c-credential-socket\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:19:46.187295 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:46.187258 5057 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79d262bd-3d59-44ec-b375-28e6c7a57c1c-workload-socket\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:19:46.187295 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:46.187272 5057 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79d262bd-3d59-44ec-b375-28e6c7a57c1c-workload-certs\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:19:46.187295 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:46.187283 5057 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79d262bd-3d59-44ec-b375-28e6c7a57c1c-istio-envoy\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:19:46.187295 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:46.187290 5057 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79d262bd-3d59-44ec-b375-28e6c7a57c1c-istio-token\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:19:46.187295 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:46.187298 5057 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79d262bd-3d59-44ec-b375-28e6c7a57c1c-istio-podinfo\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:19:46.187903 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:46.187307 5057 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79d262bd-3d59-44ec-b375-28e6c7a57c1c-istio-data\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:19:46.833680 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:46.833643 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-lfnnh" event={"ID":"e344401d-57e1-47dc-bf88-dfee4e4b0914","Type":"ContainerDied","Data":"f907355d4d947f52e9652e1db188db95407dddb50d94046a27b4950b47ef1b18"} Oct 29 23:19:46.834142 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:46.833651 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-lfnnh" Oct 29 23:19:46.834142 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:46.833699 5057 scope.go:122] "RemoveContainer" containerID="0286fb961c72f6b96a5a3fa53564bef1a75256f6dd1408cda4e5d78f2d7d92af" Oct 29 23:19:46.835297 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:46.835279 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-kk86b" Oct 29 23:19:46.835397 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:46.835370 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-kk86b" event={"ID":"79d262bd-3d59-44ec-b375-28e6c7a57c1c","Type":"ContainerDied","Data":"27307c9c034ca20eab5c21fbbeae0d9c574c12944ade8299055e499741abbcc5"} Oct 29 23:19:46.847081 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:46.847057 5057 scope.go:122] "RemoveContainer" containerID="48a12c1f45b59989923a819fd04006a6c7dec61a36e3b7f7579aeffe08f209a0" Oct 29 23:19:46.903733 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:46.903691 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:19:46.904314 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:46.904282 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-lfnnh"] Oct 29 23:19:46.958680 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:46.958450 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-lfnnh"] Oct 29 23:19:46.965889 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:46.965858 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-kk86b"] Oct 29 23:19:46.971103 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:46.971077 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-kk86b"] Oct 29 23:19:47.333447 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:47.333418 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79d262bd-3d59-44ec-b375-28e6c7a57c1c" path="/var/lib/kubelet/pods/79d262bd-3d59-44ec-b375-28e6c7a57c1c/volumes" Oct 29 23:19:47.333967 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:47.333951 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e344401d-57e1-47dc-bf88-dfee4e4b0914" path="/var/lib/kubelet/pods/e344401d-57e1-47dc-bf88-dfee4e4b0914/volumes" Oct 29 23:19:48.493612 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:48.493573 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:19:49.030048 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:19:49.030013 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d2dnn" containerName="extractor" Oct 29 23:19:50.123784 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:50.123740 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:19:50.127969 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:50.127934 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fba83e-76fd9c4d99-56dzs" podUID="0d2f8a82-947a-43b7-a4fd-a68dded7c872" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:19:50.154590 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:50.154557 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-b2b71e/test-model-b2b71e-kserve-6cf7d6cbd5-cmw67"] Oct 29 23:19:50.155669 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:50.155643 5057 cpu_manager.go:414] "containerMap: removing container" podUID="79d262bd-3d59-44ec-b375-28e6c7a57c1c" containerName="istio-proxy" Oct 29 23:19:50.155669 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:50.155668 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79d262bd-3d59-44ec-b375-28e6c7a57c1c" containerName="istio-proxy" Oct 29 23:19:50.155842 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:50.155685 5057 cpu_manager.go:414] "containerMap: removing container" podUID="d6a73f9e-f3a4-4ed8-ac2e-ff657d266514" containerName="istio-proxy" Oct 29 23:19:50.155842 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:50.155690 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6a73f9e-f3a4-4ed8-ac2e-ff657d266514" containerName="istio-proxy" Oct 29 23:19:50.155842 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:50.155727 5057 cpu_manager.go:414] "containerMap: removing container" podUID="e344401d-57e1-47dc-bf88-dfee4e4b0914" containerName="istio-proxy" Oct 29 23:19:50.155842 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:50.155733 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e344401d-57e1-47dc-bf88-dfee4e4b0914" containerName="istio-proxy" Oct 29 23:19:50.155842 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:50.155743 5057 cpu_manager.go:414] "containerMap: removing container" podUID="c5a6ed02-7acb-46de-9107-81a50942fec6" containerName="istio-proxy" Oct 29 23:19:50.155842 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:50.155784 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5a6ed02-7acb-46de-9107-81a50942fec6" containerName="istio-proxy" Oct 29 23:19:50.155842 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:50.155803 5057 cpu_manager.go:414] "containerMap: removing container" podUID="e466db5c-d1a0-4b7e-87dc-4a5ffcd23a3b" containerName="revoke-keys" Oct 29 23:19:50.155842 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:50.155808 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e466db5c-d1a0-4b7e-87dc-4a5ffcd23a3b" containerName="revoke-keys" Oct 29 23:19:50.155842 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:50.155817 5057 cpu_manager.go:414] "containerMap: removing container" podUID="73a11a50-d1bd-4eb5-8685-13e724462b24" containerName="istio-proxy" Oct 29 23:19:50.155842 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:50.155822 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73a11a50-d1bd-4eb5-8685-13e724462b24" containerName="istio-proxy" Oct 29 23:19:50.155842 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:50.155828 5057 cpu_manager.go:414] "containerMap: removing container" podUID="88b81626-da3c-4b1d-b9d8-3dbece193e80" containerName="istio-proxy" Oct 29 23:19:50.155842 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:50.155832 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88b81626-da3c-4b1d-b9d8-3dbece193e80" containerName="istio-proxy" Oct 29 23:19:50.155842 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:50.155841 5057 cpu_manager.go:414] "containerMap: removing container" podUID="3637f82c-7b89-4a17-aa32-5a99d1647969" containerName="istio-proxy" Oct 29 23:19:50.155842 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:50.155845 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3637f82c-7b89-4a17-aa32-5a99d1647969" containerName="istio-proxy" Oct 29 23:19:50.156841 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:50.156072 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="79d262bd-3d59-44ec-b375-28e6c7a57c1c" containerName="istio-proxy" Oct 29 23:19:50.156841 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:50.156083 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6a73f9e-f3a4-4ed8-ac2e-ff657d266514" containerName="istio-proxy" Oct 29 23:19:50.156841 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:50.156098 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="e344401d-57e1-47dc-bf88-dfee4e4b0914" containerName="istio-proxy" Oct 29 23:19:50.156841 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:50.156105 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5a6ed02-7acb-46de-9107-81a50942fec6" containerName="istio-proxy" Oct 29 23:19:50.156841 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:50.156113 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="e466db5c-d1a0-4b7e-87dc-4a5ffcd23a3b" containerName="revoke-keys" Oct 29 23:19:50.156841 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:50.156119 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="73a11a50-d1bd-4eb5-8685-13e724462b24" containerName="istio-proxy" Oct 29 23:19:50.156841 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:50.156126 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="88b81626-da3c-4b1d-b9d8-3dbece193e80" containerName="istio-proxy" Oct 29 23:19:50.156841 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:50.156133 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="3637f82c-7b89-4a17-aa32-5a99d1647969" containerName="istio-proxy" Oct 29 23:19:50.162465 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:50.162445 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-b2b71e/test-model-b2b71e-kserve-6cf7d6cbd5-cmw67" Oct 29 23:19:50.180406 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:50.179867 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-b2b71e/test-model-b2b71e-kserve-6cf7d6cbd5-cmw67"] Oct 29 23:19:50.183324 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:50.183300 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-b2b71e\"/\"kube-root-ca.crt\"" Oct 29 23:19:50.183985 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:50.183965 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-b2b71e\"/\"default-dockercfg-gvv2d\"" Oct 29 23:19:50.184050 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:50.184025 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-b2b71e\"/\"openshift-service-ca.crt\"" Oct 29 23:19:50.188572 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:50.188548 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-b2b71e\"/\"test-model-b2b71e-kserve-self-signed-certs\"" Oct 29 23:19:50.239514 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:50.239483 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:19:50.279112 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:50.279081 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c711b857-12cd-4bad-bec7-29e25e5e810d-tmp-dir\") pod \"test-model-b2b71e-kserve-6cf7d6cbd5-cmw67\" (UID: \"c711b857-12cd-4bad-bec7-29e25e5e810d\") " pod="ai-tenant-e2e-inf-b-b2b71e/test-model-b2b71e-kserve-6cf7d6cbd5-cmw67" Oct 29 23:19:50.279112 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:50.279115 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rc8gw\" (UniqueName: \"kubernetes.io/projected/c711b857-12cd-4bad-bec7-29e25e5e810d-kube-api-access-rc8gw\") pod \"test-model-b2b71e-kserve-6cf7d6cbd5-cmw67\" (UID: \"c711b857-12cd-4bad-bec7-29e25e5e810d\") " pod="ai-tenant-e2e-inf-b-b2b71e/test-model-b2b71e-kserve-6cf7d6cbd5-cmw67" Oct 29 23:19:50.279323 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:50.279309 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c711b857-12cd-4bad-bec7-29e25e5e810d-tls-certs\") pod \"test-model-b2b71e-kserve-6cf7d6cbd5-cmw67\" (UID: \"c711b857-12cd-4bad-bec7-29e25e5e810d\") " pod="ai-tenant-e2e-inf-b-b2b71e/test-model-b2b71e-kserve-6cf7d6cbd5-cmw67" Oct 29 23:19:50.279400 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:50.279384 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c711b857-12cd-4bad-bec7-29e25e5e810d-home\") pod \"test-model-b2b71e-kserve-6cf7d6cbd5-cmw67\" (UID: \"c711b857-12cd-4bad-bec7-29e25e5e810d\") " pod="ai-tenant-e2e-inf-b-b2b71e/test-model-b2b71e-kserve-6cf7d6cbd5-cmw67" Oct 29 23:19:50.279508 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:50.279487 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c711b857-12cd-4bad-bec7-29e25e5e810d-dshm\") pod \"test-model-b2b71e-kserve-6cf7d6cbd5-cmw67\" (UID: \"c711b857-12cd-4bad-bec7-29e25e5e810d\") " pod="ai-tenant-e2e-inf-b-b2b71e/test-model-b2b71e-kserve-6cf7d6cbd5-cmw67" Oct 29 23:19:50.279644 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:50.279627 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c711b857-12cd-4bad-bec7-29e25e5e810d-model-cache\") pod \"test-model-b2b71e-kserve-6cf7d6cbd5-cmw67\" (UID: \"c711b857-12cd-4bad-bec7-29e25e5e810d\") " pod="ai-tenant-e2e-inf-b-b2b71e/test-model-b2b71e-kserve-6cf7d6cbd5-cmw67" Oct 29 23:19:50.381151 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:50.381058 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c711b857-12cd-4bad-bec7-29e25e5e810d-model-cache\") pod \"test-model-b2b71e-kserve-6cf7d6cbd5-cmw67\" (UID: \"c711b857-12cd-4bad-bec7-29e25e5e810d\") " pod="ai-tenant-e2e-inf-b-b2b71e/test-model-b2b71e-kserve-6cf7d6cbd5-cmw67" Oct 29 23:19:50.381151 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:50.381140 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c711b857-12cd-4bad-bec7-29e25e5e810d-tmp-dir\") pod \"test-model-b2b71e-kserve-6cf7d6cbd5-cmw67\" (UID: \"c711b857-12cd-4bad-bec7-29e25e5e810d\") " pod="ai-tenant-e2e-inf-b-b2b71e/test-model-b2b71e-kserve-6cf7d6cbd5-cmw67" Oct 29 23:19:50.381461 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:50.381169 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rc8gw\" (UniqueName: \"kubernetes.io/projected/c711b857-12cd-4bad-bec7-29e25e5e810d-kube-api-access-rc8gw\") pod \"test-model-b2b71e-kserve-6cf7d6cbd5-cmw67\" (UID: \"c711b857-12cd-4bad-bec7-29e25e5e810d\") " pod="ai-tenant-e2e-inf-b-b2b71e/test-model-b2b71e-kserve-6cf7d6cbd5-cmw67" Oct 29 23:19:50.381461 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:50.381399 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c711b857-12cd-4bad-bec7-29e25e5e810d-tls-certs\") pod \"test-model-b2b71e-kserve-6cf7d6cbd5-cmw67\" (UID: \"c711b857-12cd-4bad-bec7-29e25e5e810d\") " pod="ai-tenant-e2e-inf-b-b2b71e/test-model-b2b71e-kserve-6cf7d6cbd5-cmw67" Oct 29 23:19:50.381584 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:50.381473 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c711b857-12cd-4bad-bec7-29e25e5e810d-home\") pod \"test-model-b2b71e-kserve-6cf7d6cbd5-cmw67\" (UID: \"c711b857-12cd-4bad-bec7-29e25e5e810d\") " pod="ai-tenant-e2e-inf-b-b2b71e/test-model-b2b71e-kserve-6cf7d6cbd5-cmw67" Oct 29 23:19:50.381584 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:50.381502 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c711b857-12cd-4bad-bec7-29e25e5e810d-dshm\") pod \"test-model-b2b71e-kserve-6cf7d6cbd5-cmw67\" (UID: \"c711b857-12cd-4bad-bec7-29e25e5e810d\") " pod="ai-tenant-e2e-inf-b-b2b71e/test-model-b2b71e-kserve-6cf7d6cbd5-cmw67" Oct 29 23:19:50.381584 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:50.381562 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c711b857-12cd-4bad-bec7-29e25e5e810d-model-cache\") pod \"test-model-b2b71e-kserve-6cf7d6cbd5-cmw67\" (UID: \"c711b857-12cd-4bad-bec7-29e25e5e810d\") " pod="ai-tenant-e2e-inf-b-b2b71e/test-model-b2b71e-kserve-6cf7d6cbd5-cmw67" Oct 29 23:19:50.381784 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:50.381762 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c711b857-12cd-4bad-bec7-29e25e5e810d-home\") pod \"test-model-b2b71e-kserve-6cf7d6cbd5-cmw67\" (UID: \"c711b857-12cd-4bad-bec7-29e25e5e810d\") " pod="ai-tenant-e2e-inf-b-b2b71e/test-model-b2b71e-kserve-6cf7d6cbd5-cmw67" Oct 29 23:19:50.381936 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:50.381870 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c711b857-12cd-4bad-bec7-29e25e5e810d-tmp-dir\") pod \"test-model-b2b71e-kserve-6cf7d6cbd5-cmw67\" (UID: \"c711b857-12cd-4bad-bec7-29e25e5e810d\") " pod="ai-tenant-e2e-inf-b-b2b71e/test-model-b2b71e-kserve-6cf7d6cbd5-cmw67" Oct 29 23:19:50.384032 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:50.384007 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c711b857-12cd-4bad-bec7-29e25e5e810d-tls-certs\") pod \"test-model-b2b71e-kserve-6cf7d6cbd5-cmw67\" (UID: \"c711b857-12cd-4bad-bec7-29e25e5e810d\") " pod="ai-tenant-e2e-inf-b-b2b71e/test-model-b2b71e-kserve-6cf7d6cbd5-cmw67" Oct 29 23:19:50.384176 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:50.384066 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c711b857-12cd-4bad-bec7-29e25e5e810d-dshm\") pod \"test-model-b2b71e-kserve-6cf7d6cbd5-cmw67\" (UID: \"c711b857-12cd-4bad-bec7-29e25e5e810d\") " pod="ai-tenant-e2e-inf-b-b2b71e/test-model-b2b71e-kserve-6cf7d6cbd5-cmw67" Oct 29 23:19:50.434368 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:50.422784 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rc8gw\" (UniqueName: \"kubernetes.io/projected/c711b857-12cd-4bad-bec7-29e25e5e810d-kube-api-access-rc8gw\") pod \"test-model-b2b71e-kserve-6cf7d6cbd5-cmw67\" (UID: \"c711b857-12cd-4bad-bec7-29e25e5e810d\") " pod="ai-tenant-e2e-inf-b-b2b71e/test-model-b2b71e-kserve-6cf7d6cbd5-cmw67" Oct 29 23:19:50.487679 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:50.487635 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-b2b71e/test-model-b2b71e-kserve-6cf7d6cbd5-cmw67" Oct 29 23:19:50.901450 ip-10-0-140-74 kubenswrapper[5057]: W1029 23:19:50.901371 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc711b857_12cd_4bad_bec7_29e25e5e810d.slice/crio-7a90ef8b77a3db367339d5e5215783cf7a14cfcd5e1b3b7a028b6f83722eac4d WatchSource:0}: Error finding container 7a90ef8b77a3db367339d5e5215783cf7a14cfcd5e1b3b7a028b6f83722eac4d: Status 404 returned error can't find the container with id 7a90ef8b77a3db367339d5e5215783cf7a14cfcd5e1b3b7a028b6f83722eac4d Oct 29 23:19:50.902490 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:50.902463 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-b2b71e/test-model-b2b71e-kserve-6cf7d6cbd5-cmw67"] Oct 29 23:19:51.137927 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:51.137891 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-fba83e-6f5cbb76f8-gs9rk" podUID="56a3f0c8-6d9f-428a-9ba5-f22d397c56ce" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:19:51.891286 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:51.891247 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-b2b71e/test-model-b2b71e-kserve-6cf7d6cbd5-cmw67" event={"ID":"c711b857-12cd-4bad-bec7-29e25e5e810d","Type":"ContainerStarted","Data":"c33c4413cf6ca3c33363fd6250b4d33b856a8cb109b23f5d76b10e32fe997722"} Oct 29 23:19:51.891605 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:51.891574 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-b2b71e/test-model-b2b71e-kserve-6cf7d6cbd5-cmw67" event={"ID":"c711b857-12cd-4bad-bec7-29e25e5e810d","Type":"ContainerStarted","Data":"7a90ef8b77a3db367339d5e5215783cf7a14cfcd5e1b3b7a028b6f83722eac4d"} Oct 29 23:19:51.935575 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:51.935523 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-b2b71e/test-model-b2b71e-kserve-6cf7d6cbd5-cmw67" podStartSLOduration=1.9355068659999999 podStartE2EDuration="1.935506866s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:19:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:19:51.931453635 +0000 UTC m=+2772.714667061" watchObservedRunningTime="2026-10-29 23:19:51.935506866 +0000 UTC m=+2772.718720333" Oct 29 23:19:52.051775 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:52.051737 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:19:52.491206 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:52.491171 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-b2b71e/test-model-b2b71e-kserve-6cf7d6cbd5-cmw67" Oct 29 23:19:53.886903 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:53.886862 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:19:55.613750 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:55.613711 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:19:57.052306 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:57.052264 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:19:57.531565 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:57.531527 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:19:59.316825 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:59.316778 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:19:59.520373 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:59.520295 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:19:59.646431 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:59.645967 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:19:59.759414 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:59.759386 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-fba83e-6f5cbb76f8-gs9rk" Oct 29 23:19:59.819484 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:59.819458 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fba83e-76fd9c4d99-56dzs" Oct 29 23:19:59.825539 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:59.825508 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56a3f0c8-6d9f-428a-9ba5-f22d397c56ce-kube-api-access-m2vq9\" (UniqueName: \"kubernetes.io/projected/56a3f0c8-6d9f-428a-9ba5-f22d397c56ce-kube-api-access-m2vq9\") pod \"56a3f0c8-6d9f-428a-9ba5-f22d397c56ce\" (UID: \"56a3f0c8-6d9f-428a-9ba5-f22d397c56ce\") " Oct 29 23:19:59.825689 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:59.825580 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56a3f0c8-6d9f-428a-9ba5-f22d397c56ce-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56a3f0c8-6d9f-428a-9ba5-f22d397c56ce-plugins-config-volume\") pod \"56a3f0c8-6d9f-428a-9ba5-f22d397c56ce\" (UID: \"56a3f0c8-6d9f-428a-9ba5-f22d397c56ce\") " Oct 29 23:19:59.826274 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:59.826241 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56a3f0c8-6d9f-428a-9ba5-f22d397c56ce-plugins-config-volume" pod "56a3f0c8-6d9f-428a-9ba5-f22d397c56ce" (UID: "56a3f0c8-6d9f-428a-9ba5-f22d397c56ce"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:19:59.826444 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:59.826421 5057 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56a3f0c8-6d9f-428a-9ba5-f22d397c56ce-plugins-config-volume\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:19:59.829167 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:59.829133 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56a3f0c8-6d9f-428a-9ba5-f22d397c56ce-kube-api-access-m2vq9" pod "56a3f0c8-6d9f-428a-9ba5-f22d397c56ce" (UID: "56a3f0c8-6d9f-428a-9ba5-f22d397c56ce"). InnerVolumeSpecName "kube-api-access-m2vq9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:19:59.927358 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:59.927272 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d2f8a82-947a-43b7-a4fd-a68dded7c872-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d2f8a82-947a-43b7-a4fd-a68dded7c872-plugins-config-volume\") pod \"0d2f8a82-947a-43b7-a4fd-a68dded7c872\" (UID: \"0d2f8a82-947a-43b7-a4fd-a68dded7c872\") " Oct 29 23:19:59.927532 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:59.927502 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d2f8a82-947a-43b7-a4fd-a68dded7c872-kube-api-access-q6slx\" (UniqueName: \"kubernetes.io/projected/0d2f8a82-947a-43b7-a4fd-a68dded7c872-kube-api-access-q6slx\") pod \"0d2f8a82-947a-43b7-a4fd-a68dded7c872\" (UID: \"0d2f8a82-947a-43b7-a4fd-a68dded7c872\") " Oct 29 23:19:59.927730 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:59.927703 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d2f8a82-947a-43b7-a4fd-a68dded7c872-plugins-config-volume" pod "0d2f8a82-947a-43b7-a4fd-a68dded7c872" (UID: "0d2f8a82-947a-43b7-a4fd-a68dded7c872"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:19:59.928125 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:59.928105 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m2vq9\" (UniqueName: \"kubernetes.io/projected/56a3f0c8-6d9f-428a-9ba5-f22d397c56ce-kube-api-access-m2vq9\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:19:59.928172 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:59.928128 5057 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d2f8a82-947a-43b7-a4fd-a68dded7c872-plugins-config-volume\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:19:59.929735 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:19:59.929697 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d2f8a82-947a-43b7-a4fd-a68dded7c872-kube-api-access-q6slx" pod "0d2f8a82-947a-43b7-a4fd-a68dded7c872" (UID: "0d2f8a82-947a-43b7-a4fd-a68dded7c872"). InnerVolumeSpecName "kube-api-access-q6slx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:20:00.029341 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:00.029303 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q6slx\" (UniqueName: \"kubernetes.io/projected/0d2f8a82-947a-43b7-a4fd-a68dded7c872-kube-api-access-q6slx\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:20:00.035515 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:00.035480 5057 generic.go:356] "Generic (PLEG): container finished" podID="0d2f8a82-947a-43b7-a4fd-a68dded7c872" containerID="3065b8e517bb24b274141413adcdfde01326c3c2cc9d3e774e27c31873ac50f8" exitCode=0 Oct 29 23:20:00.035720 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:00.035544 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fba83e-76fd9c4d99-56dzs" Oct 29 23:20:00.035720 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:00.035569 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fba83e-76fd9c4d99-56dzs" event={"ID":"0d2f8a82-947a-43b7-a4fd-a68dded7c872","Type":"ContainerDied","Data":"3065b8e517bb24b274141413adcdfde01326c3c2cc9d3e774e27c31873ac50f8"} Oct 29 23:20:00.035720 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:00.035632 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fba83e-76fd9c4d99-56dzs" event={"ID":"0d2f8a82-947a-43b7-a4fd-a68dded7c872","Type":"ContainerDied","Data":"b18e231e5b49be8f14127222fee0beab7c46bbb63e4ea5fb0d2d950614740f81"} Oct 29 23:20:00.035720 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:00.035671 5057 scope.go:122] "RemoveContainer" containerID="3065b8e517bb24b274141413adcdfde01326c3c2cc9d3e774e27c31873ac50f8" Oct 29 23:20:00.039215 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:00.039191 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e764937-4x99x"] Oct 29 23:20:00.041599 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:00.041575 5057 generic.go:356] "Generic (PLEG): container finished" podID="56a3f0c8-6d9f-428a-9ba5-f22d397c56ce" containerID="053db1251fba784cbc50d1d01ae7472dbc35a76f1d809de9a78d77e21aa6bfb5" exitCode=0 Oct 29 23:20:00.041890 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:00.041607 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-fba83e-6f5cbb76f8-gs9rk" event={"ID":"56a3f0c8-6d9f-428a-9ba5-f22d397c56ce","Type":"ContainerDied","Data":"053db1251fba784cbc50d1d01ae7472dbc35a76f1d809de9a78d77e21aa6bfb5"} Oct 29 23:20:00.041890 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:00.041854 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-fba83e-6f5cbb76f8-gs9rk" event={"ID":"56a3f0c8-6d9f-428a-9ba5-f22d397c56ce","Type":"ContainerDied","Data":"2029a32e8ff8dd6bdf400d5c776199eb105b7ea2bc9e13cf81755133a12e3236"} Oct 29 23:20:00.041890 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:00.041637 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-fba83e-6f5cbb76f8-gs9rk" Oct 29 23:20:00.047242 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:00.047221 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e764937-4x99x"] Oct 29 23:20:00.051042 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:00.050930 5057 scope.go:122] "RemoveContainer" containerID="3065b8e517bb24b274141413adcdfde01326c3c2cc9d3e774e27c31873ac50f8" Oct 29 23:20:00.051364 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:20:00.051302 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3065b8e517bb24b274141413adcdfde01326c3c2cc9d3e774e27c31873ac50f8\": container with ID starting with 3065b8e517bb24b274141413adcdfde01326c3c2cc9d3e774e27c31873ac50f8 not found: ID does not exist" containerID="3065b8e517bb24b274141413adcdfde01326c3c2cc9d3e774e27c31873ac50f8" Oct 29 23:20:00.051364 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:00.051350 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3065b8e517bb24b274141413adcdfde01326c3c2cc9d3e774e27c31873ac50f8"} err="failed to get container status \"3065b8e517bb24b274141413adcdfde01326c3c2cc9d3e774e27c31873ac50f8\": rpc error: code = NotFound desc = could not find container \"3065b8e517bb24b274141413adcdfde01326c3c2cc9d3e774e27c31873ac50f8\": container with ID starting with 3065b8e517bb24b274141413adcdfde01326c3c2cc9d3e774e27c31873ac50f8 not found: ID does not exist" Oct 29 23:20:00.051609 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:00.051374 5057 scope.go:122] "RemoveContainer" containerID="053db1251fba784cbc50d1d01ae7472dbc35a76f1d809de9a78d77e21aa6bfb5" Oct 29 23:20:00.064461 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:00.064440 5057 scope.go:122] "RemoveContainer" containerID="053db1251fba784cbc50d1d01ae7472dbc35a76f1d809de9a78d77e21aa6bfb5" Oct 29 23:20:00.064753 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:20:00.064732 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"053db1251fba784cbc50d1d01ae7472dbc35a76f1d809de9a78d77e21aa6bfb5\": container with ID starting with 053db1251fba784cbc50d1d01ae7472dbc35a76f1d809de9a78d77e21aa6bfb5 not found: ID does not exist" containerID="053db1251fba784cbc50d1d01ae7472dbc35a76f1d809de9a78d77e21aa6bfb5" Oct 29 23:20:00.064819 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:00.064764 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"053db1251fba784cbc50d1d01ae7472dbc35a76f1d809de9a78d77e21aa6bfb5"} err="failed to get container status \"053db1251fba784cbc50d1d01ae7472dbc35a76f1d809de9a78d77e21aa6bfb5\": rpc error: code = NotFound desc = could not find container \"053db1251fba784cbc50d1d01ae7472dbc35a76f1d809de9a78d77e21aa6bfb5\": container with ID starting with 053db1251fba784cbc50d1d01ae7472dbc35a76f1d809de9a78d77e21aa6bfb5 not found: ID does not exist" Oct 29 23:20:00.104863 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:00.104815 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-fba83e-76fd9c4d99-56dzs"] Oct 29 23:20:00.111963 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:00.111935 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-fba83e-76fd9c4d99-56dzs"] Oct 29 23:20:00.144679 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:00.144648 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-fba83e-6f5cbb76f8-gs9rk"] Oct 29 23:20:00.149767 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:00.149735 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-fba83e-6f5cbb76f8-gs9rk"] Oct 29 23:20:00.182905 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:00.182804 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:20:00.490115 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:00.490081 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-b2b71e/test-model-b2b71e-kserve-6cf7d6cbd5-cmw67" Oct 29 23:20:00.503669 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:00.503640 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-b2b71e/test-model-b2b71e-kserve-6cf7d6cbd5-cmw67" Oct 29 23:20:00.678230 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:00.678194 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:20:01.065617 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:01.065589 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-b2b71e/test-model-b2b71e-kserve-6cf7d6cbd5-cmw67" Oct 29 23:20:01.089378 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:01.085868 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-cc6328-openshift-default-84c4dfccc9-8k9hr"] Oct 29 23:20:01.089378 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:01.087521 5057 cpu_manager.go:414] "containerMap: removing container" podUID="56a3f0c8-6d9f-428a-9ba5-f22d397c56ce" containerName="payload-processing" Oct 29 23:20:01.089378 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:01.087544 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56a3f0c8-6d9f-428a-9ba5-f22d397c56ce" containerName="payload-processing" Oct 29 23:20:01.089378 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:01.087658 5057 cpu_manager.go:414] "containerMap: removing container" podUID="0d2f8a82-947a-43b7-a4fd-a68dded7c872" containerName="payload-pre-processing" Oct 29 23:20:01.089378 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:01.087667 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d2f8a82-947a-43b7-a4fd-a68dded7c872" containerName="payload-pre-processing" Oct 29 23:20:01.089378 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:01.087919 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d2f8a82-947a-43b7-a4fd-a68dded7c872" containerName="payload-pre-processing" Oct 29 23:20:01.089378 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:01.087944 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="56a3f0c8-6d9f-428a-9ba5-f22d397c56ce" containerName="payload-processing" Oct 29 23:20:01.094696 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:01.094667 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-cc6328-openshift-default-84c4dfccc9-8k9hr" Oct 29 23:20:01.143304 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:01.143275 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-cc6328-openshift-default-dockercfg-2g58g\"" Oct 29 23:20:01.147189 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:01.147162 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-cc6328-openshift-default-84c4dfccc9-8k9hr"] Oct 29 23:20:01.150114 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:01.150086 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/268ae24d-3a42-472a-97d7-aeacd9989632-istio-podinfo\") pod \"e2e-inf-b-cc6328-openshift-default-84c4dfccc9-8k9hr\" (UID: \"268ae24d-3a42-472a-97d7-aeacd9989632\") " pod="openshift-ingress/e2e-inf-b-cc6328-openshift-default-84c4dfccc9-8k9hr" Oct 29 23:20:01.150292 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:01.150275 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/268ae24d-3a42-472a-97d7-aeacd9989632-istio-envoy\") pod \"e2e-inf-b-cc6328-openshift-default-84c4dfccc9-8k9hr\" (UID: \"268ae24d-3a42-472a-97d7-aeacd9989632\") " pod="openshift-ingress/e2e-inf-b-cc6328-openshift-default-84c4dfccc9-8k9hr" Oct 29 23:20:01.150528 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:01.150502 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/268ae24d-3a42-472a-97d7-aeacd9989632-workload-certs\") pod \"e2e-inf-b-cc6328-openshift-default-84c4dfccc9-8k9hr\" (UID: \"268ae24d-3a42-472a-97d7-aeacd9989632\") " pod="openshift-ingress/e2e-inf-b-cc6328-openshift-default-84c4dfccc9-8k9hr" Oct 29 23:20:01.150644 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:01.150625 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/268ae24d-3a42-472a-97d7-aeacd9989632-istio-token\") pod \"e2e-inf-b-cc6328-openshift-default-84c4dfccc9-8k9hr\" (UID: \"268ae24d-3a42-472a-97d7-aeacd9989632\") " pod="openshift-ingress/e2e-inf-b-cc6328-openshift-default-84c4dfccc9-8k9hr" Oct 29 23:20:01.150706 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:01.150691 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9l5l4\" (UniqueName: \"kubernetes.io/projected/268ae24d-3a42-472a-97d7-aeacd9989632-kube-api-access-9l5l4\") pod \"e2e-inf-b-cc6328-openshift-default-84c4dfccc9-8k9hr\" (UID: \"268ae24d-3a42-472a-97d7-aeacd9989632\") " pod="openshift-ingress/e2e-inf-b-cc6328-openshift-default-84c4dfccc9-8k9hr" Oct 29 23:20:01.150832 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:01.150816 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/268ae24d-3a42-472a-97d7-aeacd9989632-istiod-ca-cert\") pod \"e2e-inf-b-cc6328-openshift-default-84c4dfccc9-8k9hr\" (UID: \"268ae24d-3a42-472a-97d7-aeacd9989632\") " pod="openshift-ingress/e2e-inf-b-cc6328-openshift-default-84c4dfccc9-8k9hr" Oct 29 23:20:01.150893 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:01.150855 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/268ae24d-3a42-472a-97d7-aeacd9989632-istio-data\") pod \"e2e-inf-b-cc6328-openshift-default-84c4dfccc9-8k9hr\" (UID: \"268ae24d-3a42-472a-97d7-aeacd9989632\") " pod="openshift-ingress/e2e-inf-b-cc6328-openshift-default-84c4dfccc9-8k9hr" Oct 29 23:20:01.150893 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:01.150881 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/268ae24d-3a42-472a-97d7-aeacd9989632-workload-socket\") pod \"e2e-inf-b-cc6328-openshift-default-84c4dfccc9-8k9hr\" (UID: \"268ae24d-3a42-472a-97d7-aeacd9989632\") " pod="openshift-ingress/e2e-inf-b-cc6328-openshift-default-84c4dfccc9-8k9hr" Oct 29 23:20:01.151007 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:01.150926 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/268ae24d-3a42-472a-97d7-aeacd9989632-credential-socket\") pod \"e2e-inf-b-cc6328-openshift-default-84c4dfccc9-8k9hr\" (UID: \"268ae24d-3a42-472a-97d7-aeacd9989632\") " pod="openshift-ingress/e2e-inf-b-cc6328-openshift-default-84c4dfccc9-8k9hr" Oct 29 23:20:01.168034 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:01.167955 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:20:01.252540 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:01.252503 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/268ae24d-3a42-472a-97d7-aeacd9989632-istiod-ca-cert\") pod \"e2e-inf-b-cc6328-openshift-default-84c4dfccc9-8k9hr\" (UID: \"268ae24d-3a42-472a-97d7-aeacd9989632\") " pod="openshift-ingress/e2e-inf-b-cc6328-openshift-default-84c4dfccc9-8k9hr" Oct 29 23:20:01.252540 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:01.252551 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/268ae24d-3a42-472a-97d7-aeacd9989632-istio-data\") pod \"e2e-inf-b-cc6328-openshift-default-84c4dfccc9-8k9hr\" (UID: \"268ae24d-3a42-472a-97d7-aeacd9989632\") " pod="openshift-ingress/e2e-inf-b-cc6328-openshift-default-84c4dfccc9-8k9hr" Oct 29 23:20:01.252764 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:01.252573 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/268ae24d-3a42-472a-97d7-aeacd9989632-workload-socket\") pod \"e2e-inf-b-cc6328-openshift-default-84c4dfccc9-8k9hr\" (UID: \"268ae24d-3a42-472a-97d7-aeacd9989632\") " pod="openshift-ingress/e2e-inf-b-cc6328-openshift-default-84c4dfccc9-8k9hr" Oct 29 23:20:01.252764 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:01.252607 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/268ae24d-3a42-472a-97d7-aeacd9989632-credential-socket\") pod \"e2e-inf-b-cc6328-openshift-default-84c4dfccc9-8k9hr\" (UID: \"268ae24d-3a42-472a-97d7-aeacd9989632\") " pod="openshift-ingress/e2e-inf-b-cc6328-openshift-default-84c4dfccc9-8k9hr" Oct 29 23:20:01.252764 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:01.252733 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/268ae24d-3a42-472a-97d7-aeacd9989632-istio-podinfo\") pod \"e2e-inf-b-cc6328-openshift-default-84c4dfccc9-8k9hr\" (UID: \"268ae24d-3a42-472a-97d7-aeacd9989632\") " pod="openshift-ingress/e2e-inf-b-cc6328-openshift-default-84c4dfccc9-8k9hr" Oct 29 23:20:01.252764 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:01.252755 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/268ae24d-3a42-472a-97d7-aeacd9989632-istio-envoy\") pod \"e2e-inf-b-cc6328-openshift-default-84c4dfccc9-8k9hr\" (UID: \"268ae24d-3a42-472a-97d7-aeacd9989632\") " pod="openshift-ingress/e2e-inf-b-cc6328-openshift-default-84c4dfccc9-8k9hr" Oct 29 23:20:01.252895 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:01.252790 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/268ae24d-3a42-472a-97d7-aeacd9989632-workload-certs\") pod \"e2e-inf-b-cc6328-openshift-default-84c4dfccc9-8k9hr\" (UID: \"268ae24d-3a42-472a-97d7-aeacd9989632\") " pod="openshift-ingress/e2e-inf-b-cc6328-openshift-default-84c4dfccc9-8k9hr" Oct 29 23:20:01.252895 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:01.252856 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/268ae24d-3a42-472a-97d7-aeacd9989632-istio-token\") pod \"e2e-inf-b-cc6328-openshift-default-84c4dfccc9-8k9hr\" (UID: \"268ae24d-3a42-472a-97d7-aeacd9989632\") " pod="openshift-ingress/e2e-inf-b-cc6328-openshift-default-84c4dfccc9-8k9hr" Oct 29 23:20:01.252964 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:01.252896 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9l5l4\" (UniqueName: \"kubernetes.io/projected/268ae24d-3a42-472a-97d7-aeacd9989632-kube-api-access-9l5l4\") pod \"e2e-inf-b-cc6328-openshift-default-84c4dfccc9-8k9hr\" (UID: \"268ae24d-3a42-472a-97d7-aeacd9989632\") " pod="openshift-ingress/e2e-inf-b-cc6328-openshift-default-84c4dfccc9-8k9hr" Oct 29 23:20:01.253097 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:01.253065 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/268ae24d-3a42-472a-97d7-aeacd9989632-istio-data\") pod \"e2e-inf-b-cc6328-openshift-default-84c4dfccc9-8k9hr\" (UID: \"268ae24d-3a42-472a-97d7-aeacd9989632\") " pod="openshift-ingress/e2e-inf-b-cc6328-openshift-default-84c4dfccc9-8k9hr" Oct 29 23:20:01.253297 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:01.253277 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/268ae24d-3a42-472a-97d7-aeacd9989632-istiod-ca-cert\") pod \"e2e-inf-b-cc6328-openshift-default-84c4dfccc9-8k9hr\" (UID: \"268ae24d-3a42-472a-97d7-aeacd9989632\") " pod="openshift-ingress/e2e-inf-b-cc6328-openshift-default-84c4dfccc9-8k9hr" Oct 29 23:20:01.253561 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:01.253540 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/268ae24d-3a42-472a-97d7-aeacd9989632-workload-socket\") pod \"e2e-inf-b-cc6328-openshift-default-84c4dfccc9-8k9hr\" (UID: \"268ae24d-3a42-472a-97d7-aeacd9989632\") " pod="openshift-ingress/e2e-inf-b-cc6328-openshift-default-84c4dfccc9-8k9hr" Oct 29 23:20:01.253702 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:01.253675 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/268ae24d-3a42-472a-97d7-aeacd9989632-credential-socket\") pod \"e2e-inf-b-cc6328-openshift-default-84c4dfccc9-8k9hr\" (UID: \"268ae24d-3a42-472a-97d7-aeacd9989632\") " pod="openshift-ingress/e2e-inf-b-cc6328-openshift-default-84c4dfccc9-8k9hr" Oct 29 23:20:01.253904 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:01.253886 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/268ae24d-3a42-472a-97d7-aeacd9989632-workload-certs\") pod \"e2e-inf-b-cc6328-openshift-default-84c4dfccc9-8k9hr\" (UID: \"268ae24d-3a42-472a-97d7-aeacd9989632\") " pod="openshift-ingress/e2e-inf-b-cc6328-openshift-default-84c4dfccc9-8k9hr" Oct 29 23:20:01.255583 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:01.255562 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/268ae24d-3a42-472a-97d7-aeacd9989632-istio-envoy\") pod \"e2e-inf-b-cc6328-openshift-default-84c4dfccc9-8k9hr\" (UID: \"268ae24d-3a42-472a-97d7-aeacd9989632\") " pod="openshift-ingress/e2e-inf-b-cc6328-openshift-default-84c4dfccc9-8k9hr" Oct 29 23:20:01.255760 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:01.255738 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/268ae24d-3a42-472a-97d7-aeacd9989632-istio-podinfo\") pod \"e2e-inf-b-cc6328-openshift-default-84c4dfccc9-8k9hr\" (UID: \"268ae24d-3a42-472a-97d7-aeacd9989632\") " pod="openshift-ingress/e2e-inf-b-cc6328-openshift-default-84c4dfccc9-8k9hr" Oct 29 23:20:01.275925 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:01.275901 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9l5l4\" (UniqueName: \"kubernetes.io/projected/268ae24d-3a42-472a-97d7-aeacd9989632-kube-api-access-9l5l4\") pod \"e2e-inf-b-cc6328-openshift-default-84c4dfccc9-8k9hr\" (UID: \"268ae24d-3a42-472a-97d7-aeacd9989632\") " pod="openshift-ingress/e2e-inf-b-cc6328-openshift-default-84c4dfccc9-8k9hr" Oct 29 23:20:01.276016 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:01.276000 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/268ae24d-3a42-472a-97d7-aeacd9989632-istio-token\") pod \"e2e-inf-b-cc6328-openshift-default-84c4dfccc9-8k9hr\" (UID: \"268ae24d-3a42-472a-97d7-aeacd9989632\") " pod="openshift-ingress/e2e-inf-b-cc6328-openshift-default-84c4dfccc9-8k9hr" Oct 29 23:20:01.280019 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:01.279920 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:20:01.334769 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:01.334696 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d2f8a82-947a-43b7-a4fd-a68dded7c872" path="/var/lib/kubelet/pods/0d2f8a82-947a-43b7-a4fd-a68dded7c872/volumes" Oct 29 23:20:01.335303 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:01.335288 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="37fea902-3b62-4168-8cee-e4b40c29f140" path="/var/lib/kubelet/pods/37fea902-3b62-4168-8cee-e4b40c29f140/volumes" Oct 29 23:20:01.335818 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:01.335802 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56a3f0c8-6d9f-428a-9ba5-f22d397c56ce" path="/var/lib/kubelet/pods/56a3f0c8-6d9f-428a-9ba5-f22d397c56ce/volumes" Oct 29 23:20:01.401201 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:01.401166 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:20:01.444674 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:01.444636 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-cc6328-openshift-default-84c4dfccc9-8k9hr" Oct 29 23:20:01.892592 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:01.891878 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:20:01.896063 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:01.896013 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-cc6328-openshift-default-84c4dfccc9-8k9hr"] Oct 29 23:20:01.901426 ip-10-0-140-74 kubenswrapper[5057]: W1029 23:20:01.896907 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod268ae24d_3a42_472a_97d7_aeacd9989632.slice/crio-1a713655ceb50ffbcdb5ef6492d3c0a9719422a3f8340447e84982c26bb40184 WatchSource:0}: Error finding container 1a713655ceb50ffbcdb5ef6492d3c0a9719422a3f8340447e84982c26bb40184: Status 404 returned error can't find the container with id 1a713655ceb50ffbcdb5ef6492d3c0a9719422a3f8340447e84982c26bb40184 Oct 29 23:20:01.902898 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:01.902855 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 23:20:01.902978 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:01.902953 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 23:20:01.903023 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:01.903002 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 23:20:02.067298 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:02.067264 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-cc6328-openshift-default-84c4dfccc9-8k9hr" event={"ID":"268ae24d-3a42-472a-97d7-aeacd9989632","Type":"ContainerStarted","Data":"060923ec45c1ee57ef9922829f4a5166a251b06042203ba607f48932a190b092"} Oct 29 23:20:02.067530 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:02.067326 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-cc6328-openshift-default-84c4dfccc9-8k9hr" event={"ID":"268ae24d-3a42-472a-97d7-aeacd9989632","Type":"ContainerStarted","Data":"1a713655ceb50ffbcdb5ef6492d3c0a9719422a3f8340447e84982c26bb40184"} Oct 29 23:20:02.122273 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:02.122216 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-cc6328-openshift-default-84c4dfccc9-8k9hr" podStartSLOduration=1.122196336 podStartE2EDuration="1.122196336s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:20:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:20:02.110431593 +0000 UTC m=+2782.893645028" watchObservedRunningTime="2026-10-29 23:20:02.122196336 +0000 UTC m=+2782.905409764" Oct 29 23:20:02.445570 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:02.445536 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-cc6328-openshift-default-84c4dfccc9-8k9hr" Oct 29 23:20:02.447250 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:02.447218 5057 patch_prober.go:29] interesting pod/e2e-inf-b-cc6328-openshift-default-84c4dfccc9-8k9hr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" start-of-body= Oct 29 23:20:02.447411 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:02.447277 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-cc6328-openshift-default-84c4dfccc9-8k9hr" podUID="268ae24d-3a42-472a-97d7-aeacd9989632" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" Oct 29 23:20:03.148205 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:03.148167 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:20:03.282772 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:03.282738 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:20:03.415195 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:03.415111 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:20:03.452504 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:03.452473 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-cc6328-openshift-default-84c4dfccc9-8k9hr" Oct 29 23:20:04.097698 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:04.097665 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-cc6328-openshift-default-84c4dfccc9-8k9hr" Oct 29 23:20:04.098938 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:04.098909 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-cc6328-openshift-default-84c4dfccc9-8k9hr" Oct 29 23:20:04.554244 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:04.554205 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:20:04.817164 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:04.817069 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fba83e-6rkw7"] Oct 29 23:20:04.823006 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:04.822979 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fba83e-6rkw7"] Oct 29 23:20:05.333247 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:05.333208 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e466db5c-d1a0-4b7e-87dc-4a5ffcd23a3b" path="/var/lib/kubelet/pods/e466db5c-d1a0-4b7e-87dc-4a5ffcd23a3b/volumes" Oct 29 23:20:05.648146 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:05.648071 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-024aa6-5599897986-xltss"] Oct 29 23:20:05.655208 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:05.655186 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-024aa6-5599897986-xltss" Oct 29 23:20:05.671353 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:05.671315 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-024aa6-5599897986-xltss"] Oct 29 23:20:05.680270 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:05.679102 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-024aa6\"" Oct 29 23:20:05.714314 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:05.714281 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-024aa6-b99d57cd7-pslx4"] Oct 29 23:20:05.721175 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:05.721151 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-024aa6-b99d57cd7-pslx4" Oct 29 23:20:05.742854 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:05.742823 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-024aa6-b99d57cd7-pslx4"] Oct 29 23:20:05.813462 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:05.768538 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-024aa6-dockercfg-lq4jh\"" Oct 29 23:20:05.885354 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:05.876228 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03c6c1ca-e5e4-41c5-961c-50880f649b13-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-024aa6-5599897986-xltss\" (UID: \"03c6c1ca-e5e4-41c5-961c-50880f649b13\") " pod="openshift-ingress/payload-processing-e2e-inf-b-024aa6-5599897986-xltss" Oct 29 23:20:05.885354 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:05.876279 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-479nm\" (UniqueName: \"kubernetes.io/projected/03c6c1ca-e5e4-41c5-961c-50880f649b13-kube-api-access-479nm\") pod \"payload-processing-e2e-inf-b-024aa6-5599897986-xltss\" (UID: \"03c6c1ca-e5e4-41c5-961c-50880f649b13\") " pod="openshift-ingress/payload-processing-e2e-inf-b-024aa6-5599897986-xltss" Oct 29 23:20:05.885354 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:05.876370 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a5e765c-b65b-4411-92a5-fad9521c9291-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-024aa6-b99d57cd7-pslx4\" (UID: \"7a5e765c-b65b-4411-92a5-fad9521c9291\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-024aa6-b99d57cd7-pslx4" Oct 29 23:20:05.885354 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:05.876463 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hpct\" (UniqueName: \"kubernetes.io/projected/7a5e765c-b65b-4411-92a5-fad9521c9291-kube-api-access-8hpct\") pod \"payload-pre-processing-e2e-inf-b-024aa6-b99d57cd7-pslx4\" (UID: \"7a5e765c-b65b-4411-92a5-fad9521c9291\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-024aa6-b99d57cd7-pslx4" Oct 29 23:20:05.983465 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:05.978368 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03c6c1ca-e5e4-41c5-961c-50880f649b13-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-024aa6-5599897986-xltss\" (UID: \"03c6c1ca-e5e4-41c5-961c-50880f649b13\") " pod="openshift-ingress/payload-processing-e2e-inf-b-024aa6-5599897986-xltss" Oct 29 23:20:05.983465 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:05.978429 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-479nm\" (UniqueName: \"kubernetes.io/projected/03c6c1ca-e5e4-41c5-961c-50880f649b13-kube-api-access-479nm\") pod \"payload-processing-e2e-inf-b-024aa6-5599897986-xltss\" (UID: \"03c6c1ca-e5e4-41c5-961c-50880f649b13\") " pod="openshift-ingress/payload-processing-e2e-inf-b-024aa6-5599897986-xltss" Oct 29 23:20:05.983465 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:05.978499 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a5e765c-b65b-4411-92a5-fad9521c9291-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-024aa6-b99d57cd7-pslx4\" (UID: \"7a5e765c-b65b-4411-92a5-fad9521c9291\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-024aa6-b99d57cd7-pslx4" Oct 29 23:20:05.983465 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:05.978582 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hpct\" (UniqueName: \"kubernetes.io/projected/7a5e765c-b65b-4411-92a5-fad9521c9291-kube-api-access-8hpct\") pod \"payload-pre-processing-e2e-inf-b-024aa6-b99d57cd7-pslx4\" (UID: \"7a5e765c-b65b-4411-92a5-fad9521c9291\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-024aa6-b99d57cd7-pslx4" Oct 29 23:20:05.989863 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:05.987202 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a5e765c-b65b-4411-92a5-fad9521c9291-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-024aa6-b99d57cd7-pslx4\" (UID: \"7a5e765c-b65b-4411-92a5-fad9521c9291\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-024aa6-b99d57cd7-pslx4" Oct 29 23:20:06.002350 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:05.999087 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03c6c1ca-e5e4-41c5-961c-50880f649b13-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-024aa6-5599897986-xltss\" (UID: \"03c6c1ca-e5e4-41c5-961c-50880f649b13\") " pod="openshift-ingress/payload-processing-e2e-inf-b-024aa6-5599897986-xltss" Oct 29 23:20:06.025397 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:06.021876 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hpct\" (UniqueName: \"kubernetes.io/projected/7a5e765c-b65b-4411-92a5-fad9521c9291-kube-api-access-8hpct\") pod \"payload-pre-processing-e2e-inf-b-024aa6-b99d57cd7-pslx4\" (UID: \"7a5e765c-b65b-4411-92a5-fad9521c9291\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-024aa6-b99d57cd7-pslx4" Oct 29 23:20:06.025397 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:06.024998 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-479nm\" (UniqueName: \"kubernetes.io/projected/03c6c1ca-e5e4-41c5-961c-50880f649b13-kube-api-access-479nm\") pod \"payload-processing-e2e-inf-b-024aa6-5599897986-xltss\" (UID: \"03c6c1ca-e5e4-41c5-961c-50880f649b13\") " pod="openshift-ingress/payload-processing-e2e-inf-b-024aa6-5599897986-xltss" Oct 29 23:20:06.045517 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:06.045161 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-024aa6-b99d57cd7-pslx4" Oct 29 23:20:06.120656 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:06.120620 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:20:06.272979 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:06.272893 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-024aa6-5599897986-xltss" Oct 29 23:20:06.411197 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:06.411157 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-024aa6-b99d57cd7-pslx4"] Oct 29 23:20:06.646632 ip-10-0-140-74 kubenswrapper[5057]: W1029 23:20:06.646594 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03c6c1ca_e5e4_41c5_961c_50880f649b13.slice/crio-af1c093b2d109d7383d29e55808cedf1464b9b6fb34855b8c026562809db99bd WatchSource:0}: Error finding container af1c093b2d109d7383d29e55808cedf1464b9b6fb34855b8c026562809db99bd: Status 404 returned error can't find the container with id af1c093b2d109d7383d29e55808cedf1464b9b6fb34855b8c026562809db99bd Oct 29 23:20:06.647646 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:06.647616 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-024aa6-5599897986-xltss"] Oct 29 23:20:06.858652 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:06.858567 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:20:07.134567 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:07.134485 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-024aa6-b99d57cd7-pslx4" event={"ID":"7a5e765c-b65b-4411-92a5-fad9521c9291","Type":"ContainerStarted","Data":"a40acdcd1dd204a8e3273392b221ff5ebe518c72f4fb3d8b53d68302c894c7b2"} Oct 29 23:20:07.134567 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:07.134536 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-024aa6-b99d57cd7-pslx4" event={"ID":"7a5e765c-b65b-4411-92a5-fad9521c9291","Type":"ContainerStarted","Data":"57bbe331c3f257180b6981b4778d0bbc0279b4531420da65936dfcb2c18e766a"} Oct 29 23:20:07.134567 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:07.134563 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-024aa6-b99d57cd7-pslx4" Oct 29 23:20:07.136408 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:07.136381 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-024aa6-5599897986-xltss" event={"ID":"03c6c1ca-e5e4-41c5-961c-50880f649b13","Type":"ContainerStarted","Data":"52484e13b9750f79456b156ed836d5869497879cee846f908fc2a8e542def12e"} Oct 29 23:20:07.136528 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:07.136427 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-024aa6-5599897986-xltss" event={"ID":"03c6c1ca-e5e4-41c5-961c-50880f649b13","Type":"ContainerStarted","Data":"af1c093b2d109d7383d29e55808cedf1464b9b6fb34855b8c026562809db99bd"} Oct 29 23:20:07.136528 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:07.136472 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-024aa6-5599897986-xltss" Oct 29 23:20:07.177320 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:07.177265 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-024aa6-b99d57cd7-pslx4" podStartSLOduration=2.177248711 podStartE2EDuration="2.177248711s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:20:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:20:07.173105587 +0000 UTC m=+2787.956319013" watchObservedRunningTime="2026-10-29 23:20:07.177248711 +0000 UTC m=+2787.960462141" Oct 29 23:20:07.217286 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:07.217219 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-024aa6-5599897986-xltss" podStartSLOduration=2.217201552 podStartE2EDuration="2.217201552s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:20:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:20:07.212101518 +0000 UTC m=+2787.995314944" watchObservedRunningTime="2026-10-29 23:20:07.217201552 +0000 UTC m=+2788.000414982" Oct 29 23:20:07.473847 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:07.473804 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:20:07.593926 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:07.593885 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-cc6328-6fd658c9b8-vf7tb"] Oct 29 23:20:07.599718 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:07.599689 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-cc6328-6fd658c9b8-vf7tb" Oct 29 23:20:07.604363 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:07.604296 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:20:07.614444 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:07.614411 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-cc6328\"" Oct 29 23:20:07.624266 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:07.624242 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-cc6328-6fd658c9b8-vf7tb"] Oct 29 23:20:07.647242 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:07.647207 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb9f22eb-eef1-4b1d-8fec-3fa6a2555222-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-cc6328-6fd658c9b8-vf7tb\" (UID: \"fb9f22eb-eef1-4b1d-8fec-3fa6a2555222\") " pod="openshift-ingress/payload-processing-e2e-inf-b-cc6328-6fd658c9b8-vf7tb" Oct 29 23:20:07.647455 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:07.647420 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45shh\" (UniqueName: \"kubernetes.io/projected/fb9f22eb-eef1-4b1d-8fec-3fa6a2555222-kube-api-access-45shh\") pod \"payload-processing-e2e-inf-b-cc6328-6fd658c9b8-vf7tb\" (UID: \"fb9f22eb-eef1-4b1d-8fec-3fa6a2555222\") " pod="openshift-ingress/payload-processing-e2e-inf-b-cc6328-6fd658c9b8-vf7tb" Oct 29 23:20:07.663548 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:07.663515 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-cc6328-64879d8c8c-62gfk"] Oct 29 23:20:07.696351 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:07.693818 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cc6328-64879d8c8c-62gfk" Oct 29 23:20:07.720394 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:07.704992 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-cc6328-64879d8c8c-62gfk"] Oct 29 23:20:07.735547 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:07.723710 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-cc6328-dockercfg-q7c9j\"" Oct 29 23:20:07.806357 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:07.802657 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45shh\" (UniqueName: \"kubernetes.io/projected/fb9f22eb-eef1-4b1d-8fec-3fa6a2555222-kube-api-access-45shh\") pod \"payload-processing-e2e-inf-b-cc6328-6fd658c9b8-vf7tb\" (UID: \"fb9f22eb-eef1-4b1d-8fec-3fa6a2555222\") " pod="openshift-ingress/payload-processing-e2e-inf-b-cc6328-6fd658c9b8-vf7tb" Oct 29 23:20:07.806357 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:07.803020 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3eb0219c-cf7b-4f66-b484-a78a2a5514a1-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-cc6328-64879d8c8c-62gfk\" (UID: \"3eb0219c-cf7b-4f66-b484-a78a2a5514a1\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cc6328-64879d8c8c-62gfk" Oct 29 23:20:07.806357 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:07.803437 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb9f22eb-eef1-4b1d-8fec-3fa6a2555222-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-cc6328-6fd658c9b8-vf7tb\" (UID: \"fb9f22eb-eef1-4b1d-8fec-3fa6a2555222\") " pod="openshift-ingress/payload-processing-e2e-inf-b-cc6328-6fd658c9b8-vf7tb" Oct 29 23:20:07.806357 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:07.803486 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6lp6g\" (UniqueName: \"kubernetes.io/projected/3eb0219c-cf7b-4f66-b484-a78a2a5514a1-kube-api-access-6lp6g\") pod \"payload-pre-processing-e2e-inf-b-cc6328-64879d8c8c-62gfk\" (UID: \"3eb0219c-cf7b-4f66-b484-a78a2a5514a1\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cc6328-64879d8c8c-62gfk" Oct 29 23:20:07.970117 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:07.970086 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb9f22eb-eef1-4b1d-8fec-3fa6a2555222-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-cc6328-6fd658c9b8-vf7tb\" (UID: \"fb9f22eb-eef1-4b1d-8fec-3fa6a2555222\") " pod="openshift-ingress/payload-processing-e2e-inf-b-cc6328-6fd658c9b8-vf7tb" Oct 29 23:20:08.029997 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:08.029912 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-45shh\" (UniqueName: \"kubernetes.io/projected/fb9f22eb-eef1-4b1d-8fec-3fa6a2555222-kube-api-access-45shh\") pod \"payload-processing-e2e-inf-b-cc6328-6fd658c9b8-vf7tb\" (UID: \"fb9f22eb-eef1-4b1d-8fec-3fa6a2555222\") " pod="openshift-ingress/payload-processing-e2e-inf-b-cc6328-6fd658c9b8-vf7tb" Oct 29 23:20:08.060351 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:08.060307 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6lp6g\" (UniqueName: \"kubernetes.io/projected/3eb0219c-cf7b-4f66-b484-a78a2a5514a1-kube-api-access-6lp6g\") pod \"payload-pre-processing-e2e-inf-b-cc6328-64879d8c8c-62gfk\" (UID: \"3eb0219c-cf7b-4f66-b484-a78a2a5514a1\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cc6328-64879d8c8c-62gfk" Oct 29 23:20:08.060591 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:08.060576 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3eb0219c-cf7b-4f66-b484-a78a2a5514a1-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-cc6328-64879d8c8c-62gfk\" (UID: \"3eb0219c-cf7b-4f66-b484-a78a2a5514a1\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cc6328-64879d8c8c-62gfk" Oct 29 23:20:08.061817 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:08.061785 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3eb0219c-cf7b-4f66-b484-a78a2a5514a1-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-cc6328-64879d8c8c-62gfk\" (UID: \"3eb0219c-cf7b-4f66-b484-a78a2a5514a1\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cc6328-64879d8c8c-62gfk" Oct 29 23:20:08.083750 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:08.083717 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6lp6g\" (UniqueName: \"kubernetes.io/projected/3eb0219c-cf7b-4f66-b484-a78a2a5514a1-kube-api-access-6lp6g\") pod \"payload-pre-processing-e2e-inf-b-cc6328-64879d8c8c-62gfk\" (UID: \"3eb0219c-cf7b-4f66-b484-a78a2a5514a1\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cc6328-64879d8c8c-62gfk" Oct 29 23:20:08.225473 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:08.225443 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-cc6328-6fd658c9b8-vf7tb" Oct 29 23:20:08.324390 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:08.324109 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cc6328-64879d8c8c-62gfk" Oct 29 23:20:08.579159 ip-10-0-140-74 kubenswrapper[5057]: W1029 23:20:08.579097 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb9f22eb_eef1_4b1d_8fec_3fa6a2555222.slice/crio-f464a7236ed44aa03557c37afaedfed1970188a4098754b95fe5f52226cba0c2 WatchSource:0}: Error finding container f464a7236ed44aa03557c37afaedfed1970188a4098754b95fe5f52226cba0c2: Status 404 returned error can't find the container with id f464a7236ed44aa03557c37afaedfed1970188a4098754b95fe5f52226cba0c2 Oct 29 23:20:08.714672 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:08.714410 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-cc6328-6fd658c9b8-vf7tb"] Oct 29 23:20:08.725663 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:08.725629 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-cc6328-64879d8c8c-62gfk"] Oct 29 23:20:09.151540 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:09.151456 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:20:09.158544 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:09.158516 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cc6328-64879d8c8c-62gfk" event={"ID":"3eb0219c-cf7b-4f66-b484-a78a2a5514a1","Type":"ContainerStarted","Data":"adc446a0e96784a34afcefe74ac6ed13c71698783c9d1dd2d33149600489864d"} Oct 29 23:20:09.158662 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:09.158573 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cc6328-64879d8c8c-62gfk" event={"ID":"3eb0219c-cf7b-4f66-b484-a78a2a5514a1","Type":"ContainerStarted","Data":"11f5c4498fff2580c43c3747076241b60bfde742327f922dd283265e44eef89d"} Oct 29 23:20:09.158662 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:09.158602 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cc6328-64879d8c8c-62gfk" Oct 29 23:20:09.160395 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:09.160369 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-cc6328-6fd658c9b8-vf7tb" event={"ID":"fb9f22eb-eef1-4b1d-8fec-3fa6a2555222","Type":"ContainerStarted","Data":"c5ba1c3001ad84d2ecd235d7945e11feaad470f254b82f776d97a94a40408b4c"} Oct 29 23:20:09.160528 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:09.160408 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-cc6328-6fd658c9b8-vf7tb" event={"ID":"fb9f22eb-eef1-4b1d-8fec-3fa6a2555222","Type":"ContainerStarted","Data":"f464a7236ed44aa03557c37afaedfed1970188a4098754b95fe5f52226cba0c2"} Oct 29 23:20:09.160528 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:09.160428 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-cc6328-6fd658c9b8-vf7tb" Oct 29 23:20:09.240013 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:09.239947 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cc6328-64879d8c8c-62gfk" podStartSLOduration=2.239928087 podStartE2EDuration="2.239928087s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:20:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:20:09.222500475 +0000 UTC m=+2790.005713902" watchObservedRunningTime="2026-10-29 23:20:09.239928087 +0000 UTC m=+2790.023141514" Oct 29 23:20:09.267534 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:09.267471 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-cc6328-6fd658c9b8-vf7tb" podStartSLOduration=2.267452532 podStartE2EDuration="2.267452532s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:20:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:20:09.261261147 +0000 UTC m=+2790.044474574" watchObservedRunningTime="2026-10-29 23:20:09.267452532 +0000 UTC m=+2790.050665958" Oct 29 23:20:10.839090 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:10.839048 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:20:11.073595 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:11.073557 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:20:11.624704 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:11.624671 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:20:11.736316 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:11.736271 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:20:11.939148 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:11.939067 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-fba83e-openshift-default-7544fcb869-hrggn"] Oct 29 23:20:11.939682 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:11.939267 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-fba83e-openshift-default-7544fcb869-hrggn" podUID="5d01c437-34ee-4d97-a1cc-83037a6fbdaa" containerName="istio-proxy" containerID="cri-o://2797dddcf89763789ef39efda61992dbd84a9d6d5fe7e069d8bc3255c7c317c0" gracePeriod=30 Oct 29 23:20:11.972001 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:11.971971 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:20:12.611436 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:12.611377 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-49d87f-5zs8r"] Oct 29 23:20:12.618080 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:12.618052 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-49d87f-5zs8r" Oct 29 23:20:12.632555 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:12.632525 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:20:12.636082 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:12.636054 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-49d87f-5zs8r"] Oct 29 23:20:12.644387 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:12.644318 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dmx57\"" Oct 29 23:20:12.677062 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:12.677026 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26a5c4a4-486d-4497-8a65-fffc110c9a45-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-49d87f-5zs8r\" (UID: \"26a5c4a4-486d-4497-8a65-fffc110c9a45\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-49d87f-5zs8r" Oct 29 23:20:12.677225 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:12.677205 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26a5c4a4-486d-4497-8a65-fffc110c9a45-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-49d87f-5zs8r\" (UID: \"26a5c4a4-486d-4497-8a65-fffc110c9a45\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-49d87f-5zs8r" Oct 29 23:20:12.677556 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:12.677529 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75lvz\" (UniqueName: \"kubernetes.io/projected/26a5c4a4-486d-4497-8a65-fffc110c9a45-kube-api-access-75lvz\") pod \"maas-api-revoke-keys-e2e-inf-b-49d87f-5zs8r\" (UID: \"26a5c4a4-486d-4497-8a65-fffc110c9a45\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-49d87f-5zs8r" Oct 29 23:20:12.779964 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:12.779923 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75lvz\" (UniqueName: \"kubernetes.io/projected/26a5c4a4-486d-4497-8a65-fffc110c9a45-kube-api-access-75lvz\") pod \"maas-api-revoke-keys-e2e-inf-b-49d87f-5zs8r\" (UID: \"26a5c4a4-486d-4497-8a65-fffc110c9a45\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-49d87f-5zs8r" Oct 29 23:20:12.780134 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:12.780047 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26a5c4a4-486d-4497-8a65-fffc110c9a45-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-49d87f-5zs8r\" (UID: \"26a5c4a4-486d-4497-8a65-fffc110c9a45\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-49d87f-5zs8r" Oct 29 23:20:12.780217 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:12.780201 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26a5c4a4-486d-4497-8a65-fffc110c9a45-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-49d87f-5zs8r\" (UID: \"26a5c4a4-486d-4497-8a65-fffc110c9a45\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-49d87f-5zs8r" Oct 29 23:20:12.780864 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:12.780835 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26a5c4a4-486d-4497-8a65-fffc110c9a45-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-49d87f-5zs8r\" (UID: \"26a5c4a4-486d-4497-8a65-fffc110c9a45\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-49d87f-5zs8r" Oct 29 23:20:12.782480 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:12.782457 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26a5c4a4-486d-4497-8a65-fffc110c9a45-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-49d87f-5zs8r\" (UID: \"26a5c4a4-486d-4497-8a65-fffc110c9a45\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-49d87f-5zs8r" Oct 29 23:20:12.803256 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:12.803226 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-75lvz\" (UniqueName: \"kubernetes.io/projected/26a5c4a4-486d-4497-8a65-fffc110c9a45-kube-api-access-75lvz\") pod \"maas-api-revoke-keys-e2e-inf-b-49d87f-5zs8r\" (UID: \"26a5c4a4-486d-4497-8a65-fffc110c9a45\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-49d87f-5zs8r" Oct 29 23:20:12.938745 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:12.938671 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-49d87f-5zs8r" Oct 29 23:20:13.214951 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:13.214917 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:20:13.356052 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:13.356016 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-49d87f-5zs8r"] Oct 29 23:20:14.234739 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:14.234702 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-49d87f-5zs8r" event={"ID":"26a5c4a4-486d-4497-8a65-fffc110c9a45","Type":"ContainerStarted","Data":"fb02052ef8934d6de0c89be624228de4a404489ba44daa99355046cc5a7341bb"} Oct 29 23:20:14.235159 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:14.234764 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-49d87f-5zs8r" event={"ID":"26a5c4a4-486d-4497-8a65-fffc110c9a45","Type":"ContainerStarted","Data":"3cf03c40c1d3a7deaa229f4fdf29a2b8eee24068da05ca0219ce197f0a697fa9"} Oct 29 23:20:14.279547 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:14.279489 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-49d87f-5zs8r" podStartSLOduration=2.27947596 podStartE2EDuration="2.27947596s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:20:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:20:14.273836037 +0000 UTC m=+2795.057049462" watchObservedRunningTime="2026-10-29 23:20:14.27947596 +0000 UTC m=+2795.062689386" Oct 29 23:20:15.044786 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:15.044751 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:20:15.246168 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:15.246125 5057 generic.go:356] "Generic (PLEG): container finished" podID="26a5c4a4-486d-4497-8a65-fffc110c9a45" containerID="fb02052ef8934d6de0c89be624228de4a404489ba44daa99355046cc5a7341bb" exitCode=0 Oct 29 23:20:15.246683 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:15.246175 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-49d87f-5zs8r" event={"ID":"26a5c4a4-486d-4497-8a65-fffc110c9a45","Type":"ContainerDied","Data":"fb02052ef8934d6de0c89be624228de4a404489ba44daa99355046cc5a7341bb"} Oct 29 23:20:16.496354 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:16.496315 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-49d87f-5zs8r" Oct 29 23:20:16.651771 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:16.651681 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26a5c4a4-486d-4497-8a65-fffc110c9a45-kube-api-access-75lvz\" (UniqueName: \"kubernetes.io/projected/26a5c4a4-486d-4497-8a65-fffc110c9a45-kube-api-access-75lvz\") pod \"26a5c4a4-486d-4497-8a65-fffc110c9a45\" (UID: \"26a5c4a4-486d-4497-8a65-fffc110c9a45\") " Oct 29 23:20:16.651938 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:16.651852 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26a5c4a4-486d-4497-8a65-fffc110c9a45-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26a5c4a4-486d-4497-8a65-fffc110c9a45-maas-api-openshift-ca\") pod \"26a5c4a4-486d-4497-8a65-fffc110c9a45\" (UID: \"26a5c4a4-486d-4497-8a65-fffc110c9a45\") " Oct 29 23:20:16.651938 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:16.651903 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/26a5c4a4-486d-4497-8a65-fffc110c9a45-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26a5c4a4-486d-4497-8a65-fffc110c9a45-maas-api-xks-ca\") pod \"26a5c4a4-486d-4497-8a65-fffc110c9a45\" (UID: \"26a5c4a4-486d-4497-8a65-fffc110c9a45\") " Oct 29 23:20:16.652261 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:16.652232 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26a5c4a4-486d-4497-8a65-fffc110c9a45-maas-api-openshift-ca" pod "26a5c4a4-486d-4497-8a65-fffc110c9a45" (UID: "26a5c4a4-486d-4497-8a65-fffc110c9a45"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:20:16.652859 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:16.652835 5057 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26a5c4a4-486d-4497-8a65-fffc110c9a45-maas-api-openshift-ca\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:20:16.654037 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:16.654003 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26a5c4a4-486d-4497-8a65-fffc110c9a45-maas-api-xks-ca" pod "26a5c4a4-486d-4497-8a65-fffc110c9a45" (UID: "26a5c4a4-486d-4497-8a65-fffc110c9a45"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:20:16.654142 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:16.654075 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26a5c4a4-486d-4497-8a65-fffc110c9a45-kube-api-access-75lvz" pod "26a5c4a4-486d-4497-8a65-fffc110c9a45" (UID: "26a5c4a4-486d-4497-8a65-fffc110c9a45"). InnerVolumeSpecName "kube-api-access-75lvz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:20:16.753774 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:16.753743 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-75lvz\" (UniqueName: \"kubernetes.io/projected/26a5c4a4-486d-4497-8a65-fffc110c9a45-kube-api-access-75lvz\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:20:16.753774 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:16.753771 5057 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26a5c4a4-486d-4497-8a65-fffc110c9a45-maas-api-xks-ca\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:20:16.791949 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:16.791913 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:20:17.237257 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:17.237223 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-fba83e-openshift-default-7544fcb869-hrggn" Oct 29 23:20:17.269253 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:17.269214 5057 generic.go:356] "Generic (PLEG): container finished" podID="5d01c437-34ee-4d97-a1cc-83037a6fbdaa" containerID="2797dddcf89763789ef39efda61992dbd84a9d6d5fe7e069d8bc3255c7c317c0" exitCode=0 Oct 29 23:20:17.269730 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:17.269703 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-fba83e-openshift-default-7544fcb869-hrggn" event={"ID":"5d01c437-34ee-4d97-a1cc-83037a6fbdaa","Type":"ContainerDied","Data":"2797dddcf89763789ef39efda61992dbd84a9d6d5fe7e069d8bc3255c7c317c0"} Oct 29 23:20:17.269829 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:17.269798 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-fba83e-openshift-default-7544fcb869-hrggn" event={"ID":"5d01c437-34ee-4d97-a1cc-83037a6fbdaa","Type":"ContainerDied","Data":"9dd471111f9618d2fa1b6cb867c6d9361647a559da26febf63f5145db6e71290"} Oct 29 23:20:17.269892 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:17.269830 5057 scope.go:122] "RemoveContainer" containerID="2797dddcf89763789ef39efda61992dbd84a9d6d5fe7e069d8bc3255c7c317c0" Oct 29 23:20:17.270006 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:17.269988 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-fba83e-openshift-default-7544fcb869-hrggn" Oct 29 23:20:17.271889 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:17.271845 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-49d87f-5zs8r" event={"ID":"26a5c4a4-486d-4497-8a65-fffc110c9a45","Type":"ContainerDied","Data":"3cf03c40c1d3a7deaa229f4fdf29a2b8eee24068da05ca0219ce197f0a697fa9"} Oct 29 23:20:17.271889 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:17.271865 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-49d87f-5zs8r" Oct 29 23:20:17.272027 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:17.271892 5057 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3cf03c40c1d3a7deaa229f4fdf29a2b8eee24068da05ca0219ce197f0a697fa9" Oct 29 23:20:17.285998 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:17.285980 5057 scope.go:122] "RemoveContainer" containerID="2797dddcf89763789ef39efda61992dbd84a9d6d5fe7e069d8bc3255c7c317c0" Oct 29 23:20:17.286286 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:20:17.286263 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2797dddcf89763789ef39efda61992dbd84a9d6d5fe7e069d8bc3255c7c317c0\": container with ID starting with 2797dddcf89763789ef39efda61992dbd84a9d6d5fe7e069d8bc3255c7c317c0 not found: ID does not exist" containerID="2797dddcf89763789ef39efda61992dbd84a9d6d5fe7e069d8bc3255c7c317c0" Oct 29 23:20:17.286405 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:17.286294 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2797dddcf89763789ef39efda61992dbd84a9d6d5fe7e069d8bc3255c7c317c0"} err="failed to get container status \"2797dddcf89763789ef39efda61992dbd84a9d6d5fe7e069d8bc3255c7c317c0\": rpc error: code = NotFound desc = could not find container \"2797dddcf89763789ef39efda61992dbd84a9d6d5fe7e069d8bc3255c7c317c0\": container with ID starting with 2797dddcf89763789ef39efda61992dbd84a9d6d5fe7e069d8bc3255c7c317c0 not found: ID does not exist" Oct 29 23:20:17.364890 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:17.364853 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d01c437-34ee-4d97-a1cc-83037a6fbdaa-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d01c437-34ee-4d97-a1cc-83037a6fbdaa-workload-socket\") pod \"5d01c437-34ee-4d97-a1cc-83037a6fbdaa\" (UID: \"5d01c437-34ee-4d97-a1cc-83037a6fbdaa\") " Oct 29 23:20:17.365077 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:17.364967 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d01c437-34ee-4d97-a1cc-83037a6fbdaa-kube-api-access-nqxwz\" (UniqueName: \"kubernetes.io/projected/5d01c437-34ee-4d97-a1cc-83037a6fbdaa-kube-api-access-nqxwz\") pod \"5d01c437-34ee-4d97-a1cc-83037a6fbdaa\" (UID: \"5d01c437-34ee-4d97-a1cc-83037a6fbdaa\") " Oct 29 23:20:17.365077 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:17.365032 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d01c437-34ee-4d97-a1cc-83037a6fbdaa-istio-token\" (UniqueName: \"kubernetes.io/projected/5d01c437-34ee-4d97-a1cc-83037a6fbdaa-istio-token\") pod \"5d01c437-34ee-4d97-a1cc-83037a6fbdaa\" (UID: \"5d01c437-34ee-4d97-a1cc-83037a6fbdaa\") " Oct 29 23:20:17.365214 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:17.365106 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d01c437-34ee-4d97-a1cc-83037a6fbdaa-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d01c437-34ee-4d97-a1cc-83037a6fbdaa-istio-data\") pod \"5d01c437-34ee-4d97-a1cc-83037a6fbdaa\" (UID: \"5d01c437-34ee-4d97-a1cc-83037a6fbdaa\") " Oct 29 23:20:17.365214 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:17.365207 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d01c437-34ee-4d97-a1cc-83037a6fbdaa-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d01c437-34ee-4d97-a1cc-83037a6fbdaa-istio-envoy\") pod \"5d01c437-34ee-4d97-a1cc-83037a6fbdaa\" (UID: \"5d01c437-34ee-4d97-a1cc-83037a6fbdaa\") " Oct 29 23:20:17.365385 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:17.365262 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5d01c437-34ee-4d97-a1cc-83037a6fbdaa-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d01c437-34ee-4d97-a1cc-83037a6fbdaa-istio-podinfo\") pod \"5d01c437-34ee-4d97-a1cc-83037a6fbdaa\" (UID: \"5d01c437-34ee-4d97-a1cc-83037a6fbdaa\") " Oct 29 23:20:17.365385 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:17.365312 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d01c437-34ee-4d97-a1cc-83037a6fbdaa-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d01c437-34ee-4d97-a1cc-83037a6fbdaa-credential-socket\") pod \"5d01c437-34ee-4d97-a1cc-83037a6fbdaa\" (UID: \"5d01c437-34ee-4d97-a1cc-83037a6fbdaa\") " Oct 29 23:20:17.365529 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:17.365436 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d01c437-34ee-4d97-a1cc-83037a6fbdaa-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d01c437-34ee-4d97-a1cc-83037a6fbdaa-istiod-ca-cert\") pod \"5d01c437-34ee-4d97-a1cc-83037a6fbdaa\" (UID: \"5d01c437-34ee-4d97-a1cc-83037a6fbdaa\") " Oct 29 23:20:17.365603 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:17.365534 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d01c437-34ee-4d97-a1cc-83037a6fbdaa-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d01c437-34ee-4d97-a1cc-83037a6fbdaa-workload-certs\") pod \"5d01c437-34ee-4d97-a1cc-83037a6fbdaa\" (UID: \"5d01c437-34ee-4d97-a1cc-83037a6fbdaa\") " Oct 29 23:20:17.366124 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:17.366094 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d01c437-34ee-4d97-a1cc-83037a6fbdaa-credential-socket" pod "5d01c437-34ee-4d97-a1cc-83037a6fbdaa" (UID: "5d01c437-34ee-4d97-a1cc-83037a6fbdaa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:20:17.366449 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:17.366426 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d01c437-34ee-4d97-a1cc-83037a6fbdaa-workload-socket" pod "5d01c437-34ee-4d97-a1cc-83037a6fbdaa" (UID: "5d01c437-34ee-4d97-a1cc-83037a6fbdaa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:20:17.366570 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:17.366517 5057 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d01c437-34ee-4d97-a1cc-83037a6fbdaa-credential-socket\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:20:17.366841 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:17.366817 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d01c437-34ee-4d97-a1cc-83037a6fbdaa-istiod-ca-cert" pod "5d01c437-34ee-4d97-a1cc-83037a6fbdaa" (UID: "5d01c437-34ee-4d97-a1cc-83037a6fbdaa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:20:17.366975 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:17.366956 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d01c437-34ee-4d97-a1cc-83037a6fbdaa-istio-data" pod "5d01c437-34ee-4d97-a1cc-83037a6fbdaa" (UID: "5d01c437-34ee-4d97-a1cc-83037a6fbdaa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:20:17.368639 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:17.368613 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d01c437-34ee-4d97-a1cc-83037a6fbdaa-workload-certs" pod "5d01c437-34ee-4d97-a1cc-83037a6fbdaa" (UID: "5d01c437-34ee-4d97-a1cc-83037a6fbdaa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:20:17.369600 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:17.369579 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d01c437-34ee-4d97-a1cc-83037a6fbdaa-istio-envoy" pod "5d01c437-34ee-4d97-a1cc-83037a6fbdaa" (UID: "5d01c437-34ee-4d97-a1cc-83037a6fbdaa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:20:17.371709 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:17.371678 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d01c437-34ee-4d97-a1cc-83037a6fbdaa-kube-api-access-nqxwz" pod "5d01c437-34ee-4d97-a1cc-83037a6fbdaa" (UID: "5d01c437-34ee-4d97-a1cc-83037a6fbdaa"). InnerVolumeSpecName "kube-api-access-nqxwz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:20:17.371810 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:17.371754 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5d01c437-34ee-4d97-a1cc-83037a6fbdaa-istio-podinfo" pod "5d01c437-34ee-4d97-a1cc-83037a6fbdaa" (UID: "5d01c437-34ee-4d97-a1cc-83037a6fbdaa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:20:17.372995 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:17.372972 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d01c437-34ee-4d97-a1cc-83037a6fbdaa-istio-token" pod "5d01c437-34ee-4d97-a1cc-83037a6fbdaa" (UID: "5d01c437-34ee-4d97-a1cc-83037a6fbdaa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:20:17.469184 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:17.469155 5057 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d01c437-34ee-4d97-a1cc-83037a6fbdaa-workload-socket\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:20:17.469343 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:17.469186 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nqxwz\" (UniqueName: \"kubernetes.io/projected/5d01c437-34ee-4d97-a1cc-83037a6fbdaa-kube-api-access-nqxwz\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:20:17.469343 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:17.469200 5057 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d01c437-34ee-4d97-a1cc-83037a6fbdaa-istio-token\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:20:17.469343 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:17.469212 5057 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d01c437-34ee-4d97-a1cc-83037a6fbdaa-istio-data\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:20:17.469343 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:17.469220 5057 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d01c437-34ee-4d97-a1cc-83037a6fbdaa-istio-envoy\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:20:17.469343 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:17.469229 5057 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d01c437-34ee-4d97-a1cc-83037a6fbdaa-istio-podinfo\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:20:17.469343 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:17.469236 5057 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d01c437-34ee-4d97-a1cc-83037a6fbdaa-istiod-ca-cert\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:20:17.469343 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:17.469244 5057 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d01c437-34ee-4d97-a1cc-83037a6fbdaa-workload-certs\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:20:17.626401 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:17.622687 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-fba83e-openshift-default-7544fcb869-hrggn"] Oct 29 23:20:17.628232 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:17.628205 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-fba83e-openshift-default-7544fcb869-hrggn"] Oct 29 23:20:18.149105 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:18.149077 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-024aa6-b99d57cd7-pslx4" Oct 29 23:20:18.149276 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:18.149159 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-024aa6-5599897986-xltss" Oct 29 23:20:18.506250 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:18.506207 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:20:19.029688 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:20:19.029651 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d2dnn" containerName="extractor" Oct 29 23:20:19.334684 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:19.334603 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d01c437-34ee-4d97-a1cc-83037a6fbdaa" path="/var/lib/kubelet/pods/5d01c437-34ee-4d97-a1cc-83037a6fbdaa/volumes" Oct 29 23:20:20.150109 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:20.150071 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:20:20.172643 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:20.172619 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cc6328-64879d8c8c-62gfk" Oct 29 23:20:20.172757 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:20.172693 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-cc6328-6fd658c9b8-vf7tb" Oct 29 23:20:21.803647 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:21.803606 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:20:22.697935 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:22.697896 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:20:22.807816 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:22.807775 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:20:23.038678 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:23.038603 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-49d87f-bd9d969c8-gmn7p"] Oct 29 23:20:23.074193 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:23.074158 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-49d87f-597c5cd9bc-q4vk4"] Oct 29 23:20:23.196403 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:23.196355 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:20:23.311608 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:23.311359 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:20:23.816003 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:23.815968 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:20:25.740407 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:25.740366 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:20:25.879580 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:25.879536 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:20:27.632881 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:27.632813 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:20:28.039977 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:28.039901 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-49d87f-bd9d969c8-gmn7p" podUID="831c69ad-255b-4115-8e66-5b1b9b928a5d" containerName="payload-processing" containerID="cri-o://5624298438e9776429f7a0944bf085b561245e688610cd64e19818db63aa4593" gracePeriod=25 Oct 29 23:20:28.075184 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:28.075140 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-49d87f-597c5cd9bc-q4vk4" podUID="33f2625d-10eb-432c-8958-f6768c4db656" containerName="payload-pre-processing" containerID="cri-o://d11a3d9eebeceeb7ce772b050dc2198bffc5ec253fcedc6f9c72f3b5c4e2f7fd" gracePeriod=25 Oct 29 23:20:29.352137 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:29.352096 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:20:31.038109 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:31.038075 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:20:32.711316 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:32.711281 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:20:34.406062 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:34.406018 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:20:35.157862 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:35.157825 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:20:36.101443 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:36.101398 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:20:36.689629 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:36.689586 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:20:37.037760 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:37.037724 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4265824e-x58pr"] Oct 29 23:20:37.048461 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:37.048433 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4265824e-x58pr"] Oct 29 23:20:37.306199 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:37.306103 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-49d87f-bd9d969c8-gmn7p" podUID="831c69ad-255b-4115-8e66-5b1b9b928a5d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:20:37.306647 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:37.306267 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-49d87f-597c5cd9bc-q4vk4" podUID="33f2625d-10eb-432c-8958-f6768c4db656" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:20:37.333957 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:37.333924 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ae7ef4b-9c55-41c6-bc89-1007e8fd63b1" path="/var/lib/kubelet/pods/7ae7ef4b-9c55-41c6-bc89-1007e8fd63b1/volumes" Oct 29 23:20:37.793885 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:37.793834 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:20:39.317724 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:39.317680 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:20:41.002900 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:41.002861 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:20:42.600754 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:42.600712 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:20:44.119912 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:44.119868 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:20:45.813466 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:45.813423 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:20:47.305506 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:47.305465 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-49d87f-bd9d969c8-gmn7p" podUID="831c69ad-255b-4115-8e66-5b1b9b928a5d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:20:47.306238 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:47.306207 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-49d87f-597c5cd9bc-q4vk4" podUID="33f2625d-10eb-432c-8958-f6768c4db656" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:20:47.496029 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:47.495986 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:20:48.034237 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:48.034201 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ea7eae2-2bt9w"] Oct 29 23:20:48.045087 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:48.045059 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ea7eae2-2bt9w"] Oct 29 23:20:48.438571 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:48.438542 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-49d87f-bd9d969c8-gmn7p" Oct 29 23:20:48.509734 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:48.509705 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-49d87f-597c5cd9bc-q4vk4" Oct 29 23:20:48.565527 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:48.565481 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33f2625d-10eb-432c-8958-f6768c4db656-kube-api-access-c5kr5\" (UniqueName: \"kubernetes.io/projected/33f2625d-10eb-432c-8958-f6768c4db656-kube-api-access-c5kr5\") pod \"33f2625d-10eb-432c-8958-f6768c4db656\" (UID: \"33f2625d-10eb-432c-8958-f6768c4db656\") " Oct 29 23:20:48.565749 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:48.565586 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/831c69ad-255b-4115-8e66-5b1b9b928a5d-kube-api-access-pn2tx\" (UniqueName: \"kubernetes.io/projected/831c69ad-255b-4115-8e66-5b1b9b928a5d-kube-api-access-pn2tx\") pod \"831c69ad-255b-4115-8e66-5b1b9b928a5d\" (UID: \"831c69ad-255b-4115-8e66-5b1b9b928a5d\") " Oct 29 23:20:48.566607 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:48.566552 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/831c69ad-255b-4115-8e66-5b1b9b928a5d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/831c69ad-255b-4115-8e66-5b1b9b928a5d-plugins-config-volume\") pod \"831c69ad-255b-4115-8e66-5b1b9b928a5d\" (UID: \"831c69ad-255b-4115-8e66-5b1b9b928a5d\") " Oct 29 23:20:48.566740 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:48.566693 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33f2625d-10eb-432c-8958-f6768c4db656-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33f2625d-10eb-432c-8958-f6768c4db656-plugins-config-volume\") pod \"33f2625d-10eb-432c-8958-f6768c4db656\" (UID: \"33f2625d-10eb-432c-8958-f6768c4db656\") " Oct 29 23:20:48.567317 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:48.567294 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33f2625d-10eb-432c-8958-f6768c4db656-plugins-config-volume" pod "33f2625d-10eb-432c-8958-f6768c4db656" (UID: "33f2625d-10eb-432c-8958-f6768c4db656"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:20:48.568561 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:48.568539 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/831c69ad-255b-4115-8e66-5b1b9b928a5d-plugins-config-volume" pod "831c69ad-255b-4115-8e66-5b1b9b928a5d" (UID: "831c69ad-255b-4115-8e66-5b1b9b928a5d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:20:48.575949 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:48.575893 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33f2625d-10eb-432c-8958-f6768c4db656-kube-api-access-c5kr5" pod "33f2625d-10eb-432c-8958-f6768c4db656" (UID: "33f2625d-10eb-432c-8958-f6768c4db656"). InnerVolumeSpecName "kube-api-access-c5kr5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:20:48.576037 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:48.575961 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/831c69ad-255b-4115-8e66-5b1b9b928a5d-kube-api-access-pn2tx" pod "831c69ad-255b-4115-8e66-5b1b9b928a5d" (UID: "831c69ad-255b-4115-8e66-5b1b9b928a5d"). InnerVolumeSpecName "kube-api-access-pn2tx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:20:48.645882 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:48.645851 5057 generic.go:356] "Generic (PLEG): container finished" podID="33f2625d-10eb-432c-8958-f6768c4db656" containerID="d11a3d9eebeceeb7ce772b050dc2198bffc5ec253fcedc6f9c72f3b5c4e2f7fd" exitCode=0 Oct 29 23:20:48.646049 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:48.645903 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-49d87f-597c5cd9bc-q4vk4" Oct 29 23:20:48.646049 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:48.645945 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-49d87f-597c5cd9bc-q4vk4" event={"ID":"33f2625d-10eb-432c-8958-f6768c4db656","Type":"ContainerDied","Data":"d11a3d9eebeceeb7ce772b050dc2198bffc5ec253fcedc6f9c72f3b5c4e2f7fd"} Oct 29 23:20:48.646049 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:48.646032 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-49d87f-597c5cd9bc-q4vk4" event={"ID":"33f2625d-10eb-432c-8958-f6768c4db656","Type":"ContainerDied","Data":"deecf0e1236e84f103f206261f6210b16a924cf4b3fc0c064e84bceca82b4b5d"} Oct 29 23:20:48.646225 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:48.646064 5057 scope.go:122] "RemoveContainer" containerID="d11a3d9eebeceeb7ce772b050dc2198bffc5ec253fcedc6f9c72f3b5c4e2f7fd" Oct 29 23:20:48.647596 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:48.647573 5057 generic.go:356] "Generic (PLEG): container finished" podID="831c69ad-255b-4115-8e66-5b1b9b928a5d" containerID="5624298438e9776429f7a0944bf085b561245e688610cd64e19818db63aa4593" exitCode=0 Oct 29 23:20:48.647727 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:48.647661 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-49d87f-bd9d969c8-gmn7p" Oct 29 23:20:48.647792 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:48.647660 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-49d87f-bd9d969c8-gmn7p" event={"ID":"831c69ad-255b-4115-8e66-5b1b9b928a5d","Type":"ContainerDied","Data":"5624298438e9776429f7a0944bf085b561245e688610cd64e19818db63aa4593"} Oct 29 23:20:48.647859 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:48.647813 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-49d87f-bd9d969c8-gmn7p" event={"ID":"831c69ad-255b-4115-8e66-5b1b9b928a5d","Type":"ContainerDied","Data":"cc8765b0dd21da040cebcaaa5317668433ca0000da1fe2f1a939833b2adbe2ef"} Oct 29 23:20:48.658552 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:48.658534 5057 scope.go:122] "RemoveContainer" containerID="d11a3d9eebeceeb7ce772b050dc2198bffc5ec253fcedc6f9c72f3b5c4e2f7fd" Oct 29 23:20:48.658821 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:20:48.658791 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d11a3d9eebeceeb7ce772b050dc2198bffc5ec253fcedc6f9c72f3b5c4e2f7fd\": container with ID starting with d11a3d9eebeceeb7ce772b050dc2198bffc5ec253fcedc6f9c72f3b5c4e2f7fd not found: ID does not exist" containerID="d11a3d9eebeceeb7ce772b050dc2198bffc5ec253fcedc6f9c72f3b5c4e2f7fd" Oct 29 23:20:48.658933 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:48.658831 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d11a3d9eebeceeb7ce772b050dc2198bffc5ec253fcedc6f9c72f3b5c4e2f7fd"} err="failed to get container status \"d11a3d9eebeceeb7ce772b050dc2198bffc5ec253fcedc6f9c72f3b5c4e2f7fd\": rpc error: code = NotFound desc = could not find container \"d11a3d9eebeceeb7ce772b050dc2198bffc5ec253fcedc6f9c72f3b5c4e2f7fd\": container with ID starting with d11a3d9eebeceeb7ce772b050dc2198bffc5ec253fcedc6f9c72f3b5c4e2f7fd not found: ID does not exist" Oct 29 23:20:48.658933 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:48.658854 5057 scope.go:122] "RemoveContainer" containerID="5624298438e9776429f7a0944bf085b561245e688610cd64e19818db63aa4593" Oct 29 23:20:48.668513 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:48.668491 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c5kr5\" (UniqueName: \"kubernetes.io/projected/33f2625d-10eb-432c-8958-f6768c4db656-kube-api-access-c5kr5\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:20:48.668601 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:48.668515 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pn2tx\" (UniqueName: \"kubernetes.io/projected/831c69ad-255b-4115-8e66-5b1b9b928a5d-kube-api-access-pn2tx\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:20:48.668601 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:48.668526 5057 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/831c69ad-255b-4115-8e66-5b1b9b928a5d-plugins-config-volume\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:20:48.668601 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:48.668534 5057 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33f2625d-10eb-432c-8958-f6768c4db656-plugins-config-volume\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:20:48.669856 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:48.669842 5057 scope.go:122] "RemoveContainer" containerID="5624298438e9776429f7a0944bf085b561245e688610cd64e19818db63aa4593" Oct 29 23:20:48.670141 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:20:48.670119 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5624298438e9776429f7a0944bf085b561245e688610cd64e19818db63aa4593\": container with ID starting with 5624298438e9776429f7a0944bf085b561245e688610cd64e19818db63aa4593 not found: ID does not exist" containerID="5624298438e9776429f7a0944bf085b561245e688610cd64e19818db63aa4593" Oct 29 23:20:48.670182 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:48.670151 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5624298438e9776429f7a0944bf085b561245e688610cd64e19818db63aa4593"} err="failed to get container status \"5624298438e9776429f7a0944bf085b561245e688610cd64e19818db63aa4593\": rpc error: code = NotFound desc = could not find container \"5624298438e9776429f7a0944bf085b561245e688610cd64e19818db63aa4593\": container with ID starting with 5624298438e9776429f7a0944bf085b561245e688610cd64e19818db63aa4593 not found: ID does not exist" Oct 29 23:20:48.701835 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:48.701805 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-49d87f-597c5cd9bc-q4vk4"] Oct 29 23:20:48.708110 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:48.708083 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-49d87f-597c5cd9bc-q4vk4"] Oct 29 23:20:48.738739 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:48.738710 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-49d87f-bd9d969c8-gmn7p"] Oct 29 23:20:48.744056 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:48.744033 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-49d87f-bd9d969c8-gmn7p"] Oct 29 23:20:49.030237 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:20:49.030200 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d2dnn" containerName="extractor" Oct 29 23:20:49.070250 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:49.070221 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:20:49.334185 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:49.334101 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33f2625d-10eb-432c-8958-f6768c4db656" path="/var/lib/kubelet/pods/33f2625d-10eb-432c-8958-f6768c4db656/volumes" Oct 29 23:20:49.334641 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:49.334624 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74c06cbc-5309-4045-9c17-c0fc69d05638" path="/var/lib/kubelet/pods/74c06cbc-5309-4045-9c17-c0fc69d05638/volumes" Oct 29 23:20:49.335020 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:49.335006 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="831c69ad-255b-4115-8e66-5b1b9b928a5d" path="/var/lib/kubelet/pods/831c69ad-255b-4115-8e66-5b1b9b928a5d/volumes" Oct 29 23:20:50.694994 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:50.694916 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:20:51.383926 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:51.383890 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v2tfm"] Oct 29 23:20:52.402908 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:52.402876 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-1a8107/test-model-1a8107-kserve-9bc6dcdc-z65tc"] Oct 29 23:20:52.404165 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:52.404132 5057 cpu_manager.go:414] "containerMap: removing container" podUID="5d01c437-34ee-4d97-a1cc-83037a6fbdaa" containerName="istio-proxy" Oct 29 23:20:52.404165 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:52.404156 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d01c437-34ee-4d97-a1cc-83037a6fbdaa" containerName="istio-proxy" Oct 29 23:20:52.404445 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:52.404193 5057 cpu_manager.go:414] "containerMap: removing container" podUID="831c69ad-255b-4115-8e66-5b1b9b928a5d" containerName="payload-processing" Oct 29 23:20:52.404445 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:52.404202 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="831c69ad-255b-4115-8e66-5b1b9b928a5d" containerName="payload-processing" Oct 29 23:20:52.404445 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:52.404284 5057 cpu_manager.go:414] "containerMap: removing container" podUID="33f2625d-10eb-432c-8958-f6768c4db656" containerName="payload-pre-processing" Oct 29 23:20:52.404445 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:52.404293 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33f2625d-10eb-432c-8958-f6768c4db656" containerName="payload-pre-processing" Oct 29 23:20:52.404445 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:52.404308 5057 cpu_manager.go:414] "containerMap: removing container" podUID="26a5c4a4-486d-4497-8a65-fffc110c9a45" containerName="revoke-keys" Oct 29 23:20:52.404445 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:52.404315 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26a5c4a4-486d-4497-8a65-fffc110c9a45" containerName="revoke-keys" Oct 29 23:20:52.404760 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:52.404660 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="33f2625d-10eb-432c-8958-f6768c4db656" containerName="payload-pre-processing" Oct 29 23:20:52.404760 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:52.404680 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="26a5c4a4-486d-4497-8a65-fffc110c9a45" containerName="revoke-keys" Oct 29 23:20:52.404760 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:52.404700 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d01c437-34ee-4d97-a1cc-83037a6fbdaa" containerName="istio-proxy" Oct 29 23:20:52.404760 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:52.404753 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="831c69ad-255b-4115-8e66-5b1b9b928a5d" containerName="payload-processing" Oct 29 23:20:52.409799 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:52.409765 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:20:52.410761 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:52.410734 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-1a8107/test-model-1a8107-kserve-9bc6dcdc-z65tc" Oct 29 23:20:52.444753 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:52.444712 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-1a8107\"/\"openshift-service-ca.crt\"" Oct 29 23:20:52.444915 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:52.444868 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-1a8107\"/\"test-model-1a8107-kserve-self-signed-certs\"" Oct 29 23:20:52.444985 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:52.444960 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-1a8107\"/\"default-dockercfg-xfqpd\"" Oct 29 23:20:52.445071 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:52.445048 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-1a8107\"/\"kube-root-ca.crt\"" Oct 29 23:20:52.448380 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:52.448046 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-1a8107/test-model-1a8107-kserve-9bc6dcdc-z65tc"] Oct 29 23:20:52.521162 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:52.520545 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:20:52.552291 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:52.552253 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/23b3cc65-71ec-4963-9045-175ab7a0b0d5-tmp-dir\") pod \"test-model-1a8107-kserve-9bc6dcdc-z65tc\" (UID: \"23b3cc65-71ec-4963-9045-175ab7a0b0d5\") " pod="ai-tenant-e2e-inf-b-1a8107/test-model-1a8107-kserve-9bc6dcdc-z65tc" Oct 29 23:20:52.552581 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:52.552363 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/23b3cc65-71ec-4963-9045-175ab7a0b0d5-tls-certs\") pod \"test-model-1a8107-kserve-9bc6dcdc-z65tc\" (UID: \"23b3cc65-71ec-4963-9045-175ab7a0b0d5\") " pod="ai-tenant-e2e-inf-b-1a8107/test-model-1a8107-kserve-9bc6dcdc-z65tc" Oct 29 23:20:52.552581 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:52.552545 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nfp64\" (UniqueName: \"kubernetes.io/projected/23b3cc65-71ec-4963-9045-175ab7a0b0d5-kube-api-access-nfp64\") pod \"test-model-1a8107-kserve-9bc6dcdc-z65tc\" (UID: \"23b3cc65-71ec-4963-9045-175ab7a0b0d5\") " pod="ai-tenant-e2e-inf-b-1a8107/test-model-1a8107-kserve-9bc6dcdc-z65tc" Oct 29 23:20:52.552779 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:52.552760 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/23b3cc65-71ec-4963-9045-175ab7a0b0d5-dshm\") pod \"test-model-1a8107-kserve-9bc6dcdc-z65tc\" (UID: \"23b3cc65-71ec-4963-9045-175ab7a0b0d5\") " pod="ai-tenant-e2e-inf-b-1a8107/test-model-1a8107-kserve-9bc6dcdc-z65tc" Oct 29 23:20:52.552847 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:52.552790 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/23b3cc65-71ec-4963-9045-175ab7a0b0d5-home\") pod \"test-model-1a8107-kserve-9bc6dcdc-z65tc\" (UID: \"23b3cc65-71ec-4963-9045-175ab7a0b0d5\") " pod="ai-tenant-e2e-inf-b-1a8107/test-model-1a8107-kserve-9bc6dcdc-z65tc" Oct 29 23:20:52.552907 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:52.552876 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/23b3cc65-71ec-4963-9045-175ab7a0b0d5-model-cache\") pod \"test-model-1a8107-kserve-9bc6dcdc-z65tc\" (UID: \"23b3cc65-71ec-4963-9045-175ab7a0b0d5\") " pod="ai-tenant-e2e-inf-b-1a8107/test-model-1a8107-kserve-9bc6dcdc-z65tc" Oct 29 23:20:52.655004 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:52.654917 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/23b3cc65-71ec-4963-9045-175ab7a0b0d5-dshm\") pod \"test-model-1a8107-kserve-9bc6dcdc-z65tc\" (UID: \"23b3cc65-71ec-4963-9045-175ab7a0b0d5\") " pod="ai-tenant-e2e-inf-b-1a8107/test-model-1a8107-kserve-9bc6dcdc-z65tc" Oct 29 23:20:52.655004 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:52.654959 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/23b3cc65-71ec-4963-9045-175ab7a0b0d5-home\") pod \"test-model-1a8107-kserve-9bc6dcdc-z65tc\" (UID: \"23b3cc65-71ec-4963-9045-175ab7a0b0d5\") " pod="ai-tenant-e2e-inf-b-1a8107/test-model-1a8107-kserve-9bc6dcdc-z65tc" Oct 29 23:20:52.655287 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:52.655016 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/23b3cc65-71ec-4963-9045-175ab7a0b0d5-model-cache\") pod \"test-model-1a8107-kserve-9bc6dcdc-z65tc\" (UID: \"23b3cc65-71ec-4963-9045-175ab7a0b0d5\") " pod="ai-tenant-e2e-inf-b-1a8107/test-model-1a8107-kserve-9bc6dcdc-z65tc" Oct 29 23:20:52.655287 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:52.655093 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/23b3cc65-71ec-4963-9045-175ab7a0b0d5-tmp-dir\") pod \"test-model-1a8107-kserve-9bc6dcdc-z65tc\" (UID: \"23b3cc65-71ec-4963-9045-175ab7a0b0d5\") " pod="ai-tenant-e2e-inf-b-1a8107/test-model-1a8107-kserve-9bc6dcdc-z65tc" Oct 29 23:20:52.655287 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:52.655172 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/23b3cc65-71ec-4963-9045-175ab7a0b0d5-tls-certs\") pod \"test-model-1a8107-kserve-9bc6dcdc-z65tc\" (UID: \"23b3cc65-71ec-4963-9045-175ab7a0b0d5\") " pod="ai-tenant-e2e-inf-b-1a8107/test-model-1a8107-kserve-9bc6dcdc-z65tc" Oct 29 23:20:52.655287 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:52.655277 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nfp64\" (UniqueName: \"kubernetes.io/projected/23b3cc65-71ec-4963-9045-175ab7a0b0d5-kube-api-access-nfp64\") pod \"test-model-1a8107-kserve-9bc6dcdc-z65tc\" (UID: \"23b3cc65-71ec-4963-9045-175ab7a0b0d5\") " pod="ai-tenant-e2e-inf-b-1a8107/test-model-1a8107-kserve-9bc6dcdc-z65tc" Oct 29 23:20:52.655564 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:52.655486 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/23b3cc65-71ec-4963-9045-175ab7a0b0d5-home\") pod \"test-model-1a8107-kserve-9bc6dcdc-z65tc\" (UID: \"23b3cc65-71ec-4963-9045-175ab7a0b0d5\") " pod="ai-tenant-e2e-inf-b-1a8107/test-model-1a8107-kserve-9bc6dcdc-z65tc" Oct 29 23:20:52.655564 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:52.655511 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/23b3cc65-71ec-4963-9045-175ab7a0b0d5-model-cache\") pod \"test-model-1a8107-kserve-9bc6dcdc-z65tc\" (UID: \"23b3cc65-71ec-4963-9045-175ab7a0b0d5\") " pod="ai-tenant-e2e-inf-b-1a8107/test-model-1a8107-kserve-9bc6dcdc-z65tc" Oct 29 23:20:52.655683 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:52.655661 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/23b3cc65-71ec-4963-9045-175ab7a0b0d5-tmp-dir\") pod \"test-model-1a8107-kserve-9bc6dcdc-z65tc\" (UID: \"23b3cc65-71ec-4963-9045-175ab7a0b0d5\") " pod="ai-tenant-e2e-inf-b-1a8107/test-model-1a8107-kserve-9bc6dcdc-z65tc" Oct 29 23:20:52.657790 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:52.657764 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/23b3cc65-71ec-4963-9045-175ab7a0b0d5-dshm\") pod \"test-model-1a8107-kserve-9bc6dcdc-z65tc\" (UID: \"23b3cc65-71ec-4963-9045-175ab7a0b0d5\") " pod="ai-tenant-e2e-inf-b-1a8107/test-model-1a8107-kserve-9bc6dcdc-z65tc" Oct 29 23:20:52.658013 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:52.657989 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/23b3cc65-71ec-4963-9045-175ab7a0b0d5-tls-certs\") pod \"test-model-1a8107-kserve-9bc6dcdc-z65tc\" (UID: \"23b3cc65-71ec-4963-9045-175ab7a0b0d5\") " pod="ai-tenant-e2e-inf-b-1a8107/test-model-1a8107-kserve-9bc6dcdc-z65tc" Oct 29 23:20:52.752351 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:52.751716 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nfp64\" (UniqueName: \"kubernetes.io/projected/23b3cc65-71ec-4963-9045-175ab7a0b0d5-kube-api-access-nfp64\") pod \"test-model-1a8107-kserve-9bc6dcdc-z65tc\" (UID: \"23b3cc65-71ec-4963-9045-175ab7a0b0d5\") " pod="ai-tenant-e2e-inf-b-1a8107/test-model-1a8107-kserve-9bc6dcdc-z65tc" Oct 29 23:20:53.048732 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:53.048699 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-1a8107/test-model-1a8107-kserve-9bc6dcdc-z65tc" Oct 29 23:20:53.401358 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:53.400107 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-1a8107/test-model-1a8107-kserve-9bc6dcdc-z65tc"] Oct 29 23:20:53.894611 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:53.894560 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-1a8107/test-model-1a8107-kserve-9bc6dcdc-z65tc" event={"ID":"23b3cc65-71ec-4963-9045-175ab7a0b0d5","Type":"ContainerStarted","Data":"cbd1651bf8b60d7555a78a4211056fbd3acf51576bac4f5a1a7d63b08cc92389"} Oct 29 23:20:53.895114 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:53.894638 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-1a8107/test-model-1a8107-kserve-9bc6dcdc-z65tc" event={"ID":"23b3cc65-71ec-4963-9045-175ab7a0b0d5","Type":"ContainerStarted","Data":"ef1c9511b95015ea2226cd37bdb378d0c98976527721eb5262e9ff7a8ef36ae3"} Oct 29 23:20:53.941178 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:53.941132 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-1a8107/test-model-1a8107-kserve-9bc6dcdc-z65tc" podStartSLOduration=1.9411177240000002 podStartE2EDuration="1.941117724s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:20:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:20:53.936659796 +0000 UTC m=+2834.719873219" watchObservedRunningTime="2026-10-29 23:20:53.941117724 +0000 UTC m=+2834.724331149" Oct 29 23:20:54.049714 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:54.049677 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-1a8107/test-model-1a8107-kserve-9bc6dcdc-z65tc" Oct 29 23:20:54.517101 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:54.517055 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:20:56.269340 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:56.269292 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:20:57.805238 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:57.805198 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:20:59.415436 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:20:59.415401 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:21:00.984321 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:21:00.984287 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:21:02.592195 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:21:02.592159 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:21:02.702710 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:21:02.702673 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:21:02.813120 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:21:02.813088 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:21:03.049836 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:21:03.049805 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-1a8107/test-model-1a8107-kserve-9bc6dcdc-z65tc" Oct 29 23:21:03.063861 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:21:03.063834 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-1a8107/test-model-1a8107-kserve-9bc6dcdc-z65tc" Oct 29 23:21:04.006221 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:21:04.006187 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-1a8107/test-model-1a8107-kserve-9bc6dcdc-z65tc" Oct 29 23:21:04.468447 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:21:04.468403 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-cc6328-59f5f5554d-rgnmt"] Oct 29 23:21:04.474308 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:21:04.474291 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cc6328-59f5f5554d-rgnmt" Oct 29 23:21:04.493466 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:21:04.493430 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-cc6328-serving-cert\"" Oct 29 23:21:04.502995 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:21:04.502971 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-cc6328-59f5f5554d-rgnmt"] Oct 29 23:21:04.553119 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:21:04.553088 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:21:04.629197 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:21:04.629165 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7x6dh\" (UniqueName: \"kubernetes.io/projected/04c996bb-8ee7-465f-9c69-ba118111d6f2-kube-api-access-7x6dh\") pod \"maas-api-e2e-inf-b-cc6328-59f5f5554d-rgnmt\" (UID: \"04c996bb-8ee7-465f-9c69-ba118111d6f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cc6328-59f5f5554d-rgnmt" Oct 29 23:21:04.629412 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:21:04.629235 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/04c996bb-8ee7-465f-9c69-ba118111d6f2-maas-api-tls\") pod \"maas-api-e2e-inf-b-cc6328-59f5f5554d-rgnmt\" (UID: \"04c996bb-8ee7-465f-9c69-ba118111d6f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cc6328-59f5f5554d-rgnmt" Oct 29 23:21:04.629517 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:21:04.629494 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/04c996bb-8ee7-465f-9c69-ba118111d6f2-openshift-service-ca\") pod \"maas-api-e2e-inf-b-cc6328-59f5f5554d-rgnmt\" (UID: \"04c996bb-8ee7-465f-9c69-ba118111d6f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cc6328-59f5f5554d-rgnmt" Oct 29 23:21:04.629667 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:21:04.629647 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/04c996bb-8ee7-465f-9c69-ba118111d6f2-metrics-certs\") pod \"maas-api-e2e-inf-b-cc6328-59f5f5554d-rgnmt\" (UID: \"04c996bb-8ee7-465f-9c69-ba118111d6f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cc6328-59f5f5554d-rgnmt" Oct 29 23:21:04.682357 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:21:04.671620 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:21:04.731098 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:21:04.731006 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/04c996bb-8ee7-465f-9c69-ba118111d6f2-maas-api-tls\") pod \"maas-api-e2e-inf-b-cc6328-59f5f5554d-rgnmt\" (UID: \"04c996bb-8ee7-465f-9c69-ba118111d6f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cc6328-59f5f5554d-rgnmt" Oct 29 23:21:04.731301 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:21:04.731277 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/04c996bb-8ee7-465f-9c69-ba118111d6f2-openshift-service-ca\") pod \"maas-api-e2e-inf-b-cc6328-59f5f5554d-rgnmt\" (UID: \"04c996bb-8ee7-465f-9c69-ba118111d6f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cc6328-59f5f5554d-rgnmt" Oct 29 23:21:04.731468 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:21:04.731444 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/04c996bb-8ee7-465f-9c69-ba118111d6f2-metrics-certs\") pod \"maas-api-e2e-inf-b-cc6328-59f5f5554d-rgnmt\" (UID: \"04c996bb-8ee7-465f-9c69-ba118111d6f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cc6328-59f5f5554d-rgnmt" Oct 29 23:21:04.731566 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:21:04.731514 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7x6dh\" (UniqueName: \"kubernetes.io/projected/04c996bb-8ee7-465f-9c69-ba118111d6f2-kube-api-access-7x6dh\") pod \"maas-api-e2e-inf-b-cc6328-59f5f5554d-rgnmt\" (UID: \"04c996bb-8ee7-465f-9c69-ba118111d6f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cc6328-59f5f5554d-rgnmt" Oct 29 23:21:04.734842 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:21:04.734816 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/04c996bb-8ee7-465f-9c69-ba118111d6f2-maas-api-tls\") pod \"maas-api-e2e-inf-b-cc6328-59f5f5554d-rgnmt\" (UID: \"04c996bb-8ee7-465f-9c69-ba118111d6f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cc6328-59f5f5554d-rgnmt" Oct 29 23:21:04.738974 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:21:04.738185 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/04c996bb-8ee7-465f-9c69-ba118111d6f2-openshift-service-ca\") pod \"maas-api-e2e-inf-b-cc6328-59f5f5554d-rgnmt\" (UID: \"04c996bb-8ee7-465f-9c69-ba118111d6f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cc6328-59f5f5554d-rgnmt" Oct 29 23:21:04.738974 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:21:04.738823 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/04c996bb-8ee7-465f-9c69-ba118111d6f2-metrics-certs\") pod \"maas-api-e2e-inf-b-cc6328-59f5f5554d-rgnmt\" (UID: \"04c996bb-8ee7-465f-9c69-ba118111d6f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cc6328-59f5f5554d-rgnmt" Oct 29 23:21:04.759191 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:21:04.759160 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7x6dh\" (UniqueName: \"kubernetes.io/projected/04c996bb-8ee7-465f-9c69-ba118111d6f2-kube-api-access-7x6dh\") pod \"maas-api-e2e-inf-b-cc6328-59f5f5554d-rgnmt\" (UID: \"04c996bb-8ee7-465f-9c69-ba118111d6f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cc6328-59f5f5554d-rgnmt" Oct 29 23:21:04.803694 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:21:04.803657 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cc6328-59f5f5554d-rgnmt" Oct 29 23:21:05.141210 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:21:05.139007 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-cc6328-59f5f5554d-rgnmt"] Oct 29 23:21:05.146788 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:21:05.146606 5057 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 29 23:21:06.021267 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:21:06.021228 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cc6328-59f5f5554d-rgnmt" event={"ID":"04c996bb-8ee7-465f-9c69-ba118111d6f2","Type":"ContainerStarted","Data":"e0e16310fb9a9d5d25ac6d46d902666bb483f1dad6d0a020099547eaede51c88"} Oct 29 23:21:06.021513 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:21:06.021283 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cc6328-59f5f5554d-rgnmt" event={"ID":"04c996bb-8ee7-465f-9c69-ba118111d6f2","Type":"ContainerStarted","Data":"f2f532b17efc4d19211f7fd79863e9733d2200e6dfc2930f5d612f55f1b9f47e"} Oct 29 23:21:06.021513 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:21:06.021386 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cc6328-59f5f5554d-rgnmt" Oct 29 23:21:06.070429 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:21:06.070383 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cc6328-59f5f5554d-rgnmt" podStartSLOduration=1.706069609 podStartE2EDuration="2.070368457s" totalImagesPullingTime="364.298848ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:21:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:21:06.062690199 +0000 UTC m=+2846.845903625" watchObservedRunningTime="2026-10-29 23:21:06.070368457 +0000 UTC m=+2846.853581883" Oct 29 23:21:06.593888 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:21:06.593848 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:21:08.121992 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:21:08.121954 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:21:09.646841 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:21:09.646809 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:21:11.391537 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:21:11.391499 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:21:12.040381 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:21:12.040323 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cc6328-59f5f5554d-rgnmt" Oct 29 23:21:12.922223 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:21:12.922179 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:21:14.522170 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:21:14.522132 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:21:15.909493 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:21:15.909453 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:21:16.568034 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:21:16.567999 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:21:16.807214 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:21:16.807182 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:21:16.974390 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:21:16.965987 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-49d87f-5zs8r"] Oct 29 23:21:17.013467 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:21:17.013428 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-49d87f-5zs8r"] Oct 29 23:21:17.164590 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:21:17.164550 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:21:17.337173 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:21:17.337086 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26a5c4a4-486d-4497-8a65-fffc110c9a45" path="/var/lib/kubelet/pods/26a5c4a4-486d-4497-8a65-fffc110c9a45/volumes" Oct 29 23:21:17.411260 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:21:17.411221 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:21:18.906161 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:21:18.906125 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:21:19.055990 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:21:19.055946 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d2dnn" containerName="extractor" Oct 29 23:21:20.520457 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:21:20.520410 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:21:22.088455 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:21:22.088417 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:21:23.093589 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:21:23.093554 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-qpb7q"] Oct 29 23:21:23.094066 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:21:23.093753 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-qpb7q" podUID="708063fe-d579-4b1b-982d-63faeb84dac7" containerName="istio-proxy" containerID="cri-o://439ca738d78e30ca61c85ad476b4595de3d3bb044711cc2b9f212752831d4580" gracePeriod=30 Oct 29 23:21:23.709069 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:21:23.709031 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:21:23.888724 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:21:23.888684 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:21:24.001468 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:21:24.000998 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:21:24.216920 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:21:24.216884 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:21:25.184665 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:21:25.184615 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:21:25.298308 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:21:25.298273 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:21:25.407383 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:21:25.407309 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:21:25.514998 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:21:25.514960 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:21:27.075213 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:21:27.075180 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-058e69ab-ngh6t"] Oct 29 23:21:27.082078 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:21:27.082056 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-058e69ab-ngh6t" Oct 29 23:21:27.099567 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:21:27.099538 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:21:27.102739 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:21:27.102710 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-058e69ab-ngh6t"] Oct 29 23:21:27.107530 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:21:27.107511 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dmx57\"" Oct 29 23:21:27.131375 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:21:27.130398 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:21:27.260226 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:21:27.260186 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kr4bn\" (UniqueName: \"kubernetes.io/projected/c9e1f7d1-fb93-44da-9b30-8c27769dbbc9-kube-api-access-kr4bn\") pod \"maas-api-revoke-sub-058e69ab-ngh6t\" (UID: \"c9e1f7d1-fb93-44da-9b30-8c27769dbbc9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-058e69ab-ngh6t" Oct 29 23:21:27.260437 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:21:27.260385 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9e1f7d1-fb93-44da-9b30-8c27769dbbc9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-058e69ab-ngh6t\" (UID: \"c9e1f7d1-fb93-44da-9b30-8c27769dbbc9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-058e69ab-ngh6t" Oct 29 23:21:27.260532 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:21:27.260485 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9e1f7d1-fb93-44da-9b30-8c27769dbbc9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-058e69ab-ngh6t\" (UID: \"c9e1f7d1-fb93-44da-9b30-8c27769dbbc9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-058e69ab-ngh6t" Oct 29 23:21:27.362098 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:21:27.362017 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kr4bn\" (UniqueName: \"kubernetes.io/projected/c9e1f7d1-fb93-44da-9b30-8c27769dbbc9-kube-api-access-kr4bn\") pod \"maas-api-revoke-sub-058e69ab-ngh6t\" (UID: \"c9e1f7d1-fb93-44da-9b30-8c27769dbbc9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-058e69ab-ngh6t" Oct 29 23:21:27.362274 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:21:27.362131 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9e1f7d1-fb93-44da-9b30-8c27769dbbc9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-058e69ab-ngh6t\" (UID: \"c9e1f7d1-fb93-44da-9b30-8c27769dbbc9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-058e69ab-ngh6t" Oct 29 23:21:27.362274 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:21:27.362179 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9e1f7d1-fb93-44da-9b30-8c27769dbbc9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-058e69ab-ngh6t\" (UID: \"c9e1f7d1-fb93-44da-9b30-8c27769dbbc9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-058e69ab-ngh6t" Oct 29 23:21:27.363526 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:21:27.363497 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9e1f7d1-fb93-44da-9b30-8c27769dbbc9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-058e69ab-ngh6t\" (UID: \"c9e1f7d1-fb93-44da-9b30-8c27769dbbc9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-058e69ab-ngh6t" Oct 29 23:21:27.365196 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:21:27.365170 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9e1f7d1-fb93-44da-9b30-8c27769dbbc9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-058e69ab-ngh6t\" (UID: \"c9e1f7d1-fb93-44da-9b30-8c27769dbbc9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-058e69ab-ngh6t" Oct 29 23:21:27.385791 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:21:27.385767 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kr4bn\" (UniqueName: \"kubernetes.io/projected/c9e1f7d1-fb93-44da-9b30-8c27769dbbc9-kube-api-access-kr4bn\") pod \"maas-api-revoke-sub-058e69ab-ngh6t\" (UID: \"c9e1f7d1-fb93-44da-9b30-8c27769dbbc9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-058e69ab-ngh6t" Oct 29 23:21:27.400606 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:21:27.400580 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-058e69ab-ngh6t" Oct 29 23:21:27.712549 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:21:27.712522 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-058e69ab-ngh6t"] Oct 29 23:21:27.713246 ip-10-0-140-74 kubenswrapper[5057]: W1029 23:21:27.713214 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9e1f7d1_fb93_44da_9b30_8c27769dbbc9.slice/crio-f3409922075baadf1b43b390490a5fba7743e766279314603bfcd48ce5222964 WatchSource:0}: Error finding container f3409922075baadf1b43b390490a5fba7743e766279314603bfcd48ce5222964: Status 404 returned error can't find the container with id f3409922075baadf1b43b390490a5fba7743e766279314603bfcd48ce5222964 Oct 29 23:21:28.300504 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:21:28.298880 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-058e69ab-ngh6t" event={"ID":"c9e1f7d1-fb93-44da-9b30-8c27769dbbc9","Type":"ContainerStarted","Data":"d66683bb06b93b6757593cd6e707ffd3602aeb34fcbc66e6876ad26bea6d04a0"} Oct 29 23:21:28.300504 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:21:28.298950 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-058e69ab-ngh6t" event={"ID":"c9e1f7d1-fb93-44da-9b30-8c27769dbbc9","Type":"ContainerStarted","Data":"f3409922075baadf1b43b390490a5fba7743e766279314603bfcd48ce5222964"} Oct 29 23:21:28.301571 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:21:28.301537 5057 generic.go:356] "Generic (PLEG): container finished" podID="708063fe-d579-4b1b-982d-63faeb84dac7" containerID="439ca738d78e30ca61c85ad476b4595de3d3bb044711cc2b9f212752831d4580" exitCode=0 Oct 29 23:21:28.301705 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:21:28.301674 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-qpb7q" event={"ID":"708063fe-d579-4b1b-982d-63faeb84dac7","Type":"ContainerDied","Data":"439ca738d78e30ca61c85ad476b4595de3d3bb044711cc2b9f212752831d4580"} Oct 29 23:21:28.342402 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:21:28.342258 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-058e69ab-ngh6t" podStartSLOduration=1.34224392 podStartE2EDuration="1.34224392s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:21:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:21:28.33850639 +0000 UTC m=+2869.121719818" watchObservedRunningTime="2026-10-29 23:21:28.34224392 +0000 UTC m=+2869.125457346" Oct 29 23:21:28.400106 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:21:28.400082 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-qpb7q" Oct 29 23:21:28.483593 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:21:28.483556 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/708063fe-d579-4b1b-982d-63faeb84dac7-istio-token\" (UniqueName: \"kubernetes.io/projected/708063fe-d579-4b1b-982d-63faeb84dac7-istio-token\") pod \"708063fe-d579-4b1b-982d-63faeb84dac7\" (UID: \"708063fe-d579-4b1b-982d-63faeb84dac7\") " Oct 29 23:21:28.483789 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:21:28.483631 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/708063fe-d579-4b1b-982d-63faeb84dac7-kube-api-access-w4ps9\" (UniqueName: \"kubernetes.io/projected/708063fe-d579-4b1b-982d-63faeb84dac7-kube-api-access-w4ps9\") pod \"708063fe-d579-4b1b-982d-63faeb84dac7\" (UID: \"708063fe-d579-4b1b-982d-63faeb84dac7\") " Oct 29 23:21:28.483789 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:21:28.483682 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/708063fe-d579-4b1b-982d-63faeb84dac7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/708063fe-d579-4b1b-982d-63faeb84dac7-istiod-ca-cert\") pod \"708063fe-d579-4b1b-982d-63faeb84dac7\" (UID: \"708063fe-d579-4b1b-982d-63faeb84dac7\") " Oct 29 23:21:28.483789 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:21:28.483723 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/708063fe-d579-4b1b-982d-63faeb84dac7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/708063fe-d579-4b1b-982d-63faeb84dac7-workload-socket\") pod \"708063fe-d579-4b1b-982d-63faeb84dac7\" (UID: \"708063fe-d579-4b1b-982d-63faeb84dac7\") " Oct 29 23:21:28.483980 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:21:28.483904 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/708063fe-d579-4b1b-982d-63faeb84dac7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/708063fe-d579-4b1b-982d-63faeb84dac7-istio-envoy\") pod \"708063fe-d579-4b1b-982d-63faeb84dac7\" (UID: \"708063fe-d579-4b1b-982d-63faeb84dac7\") " Oct 29 23:21:28.484052 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:21:28.483976 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/708063fe-d579-4b1b-982d-63faeb84dac7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/708063fe-d579-4b1b-982d-63faeb84dac7-credential-socket\") pod \"708063fe-d579-4b1b-982d-63faeb84dac7\" (UID: \"708063fe-d579-4b1b-982d-63faeb84dac7\") " Oct 29 23:21:28.484135 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:21:28.484109 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/708063fe-d579-4b1b-982d-63faeb84dac7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/708063fe-d579-4b1b-982d-63faeb84dac7-istio-podinfo\") pod \"708063fe-d579-4b1b-982d-63faeb84dac7\" (UID: \"708063fe-d579-4b1b-982d-63faeb84dac7\") " Oct 29 23:21:28.484198 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:21:28.484171 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/708063fe-d579-4b1b-982d-63faeb84dac7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/708063fe-d579-4b1b-982d-63faeb84dac7-workload-certs\") pod \"708063fe-d579-4b1b-982d-63faeb84dac7\" (UID: \"708063fe-d579-4b1b-982d-63faeb84dac7\") " Oct 29 23:21:28.484278 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:21:28.484261 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/708063fe-d579-4b1b-982d-63faeb84dac7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/708063fe-d579-4b1b-982d-63faeb84dac7-istio-data\") pod \"708063fe-d579-4b1b-982d-63faeb84dac7\" (UID: \"708063fe-d579-4b1b-982d-63faeb84dac7\") " Oct 29 23:21:28.484506 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:21:28.484478 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/708063fe-d579-4b1b-982d-63faeb84dac7-credential-socket" pod "708063fe-d579-4b1b-982d-63faeb84dac7" (UID: "708063fe-d579-4b1b-982d-63faeb84dac7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:21:28.484667 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:21:28.484641 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/708063fe-d579-4b1b-982d-63faeb84dac7-workload-certs" pod "708063fe-d579-4b1b-982d-63faeb84dac7" (UID: "708063fe-d579-4b1b-982d-63faeb84dac7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:21:28.484731 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:21:28.484697 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/708063fe-d579-4b1b-982d-63faeb84dac7-istio-data" pod "708063fe-d579-4b1b-982d-63faeb84dac7" (UID: "708063fe-d579-4b1b-982d-63faeb84dac7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:21:28.484946 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:21:28.484924 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/708063fe-d579-4b1b-982d-63faeb84dac7-workload-socket" pod "708063fe-d579-4b1b-982d-63faeb84dac7" (UID: "708063fe-d579-4b1b-982d-63faeb84dac7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:21:28.485187 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:21:28.485164 5057 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/708063fe-d579-4b1b-982d-63faeb84dac7-credential-socket\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:21:28.485370 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:21:28.485190 5057 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/708063fe-d579-4b1b-982d-63faeb84dac7-workload-certs\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:21:28.485370 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:21:28.485204 5057 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/708063fe-d579-4b1b-982d-63faeb84dac7-istio-data\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:21:28.485370 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:21:28.485215 5057 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/708063fe-d579-4b1b-982d-63faeb84dac7-workload-socket\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:21:28.485370 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:21:28.485240 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/708063fe-d579-4b1b-982d-63faeb84dac7-istiod-ca-cert" pod "708063fe-d579-4b1b-982d-63faeb84dac7" (UID: "708063fe-d579-4b1b-982d-63faeb84dac7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:21:28.486698 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:21:28.486658 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/708063fe-d579-4b1b-982d-63faeb84dac7-istio-token" pod "708063fe-d579-4b1b-982d-63faeb84dac7" (UID: "708063fe-d579-4b1b-982d-63faeb84dac7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:21:28.487115 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:21:28.487089 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/708063fe-d579-4b1b-982d-63faeb84dac7-kube-api-access-w4ps9" pod "708063fe-d579-4b1b-982d-63faeb84dac7" (UID: "708063fe-d579-4b1b-982d-63faeb84dac7"). InnerVolumeSpecName "kube-api-access-w4ps9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:21:28.487377 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:21:28.487317 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/708063fe-d579-4b1b-982d-63faeb84dac7-istio-envoy" pod "708063fe-d579-4b1b-982d-63faeb84dac7" (UID: "708063fe-d579-4b1b-982d-63faeb84dac7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:21:28.488109 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:21:28.488085 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/708063fe-d579-4b1b-982d-63faeb84dac7-istio-podinfo" pod "708063fe-d579-4b1b-982d-63faeb84dac7" (UID: "708063fe-d579-4b1b-982d-63faeb84dac7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:21:28.587007 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:21:28.586975 5057 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/708063fe-d579-4b1b-982d-63faeb84dac7-istio-podinfo\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:21:28.587007 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:21:28.587005 5057 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/708063fe-d579-4b1b-982d-63faeb84dac7-istio-token\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:21:28.587249 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:21:28.587017 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w4ps9\" (UniqueName: \"kubernetes.io/projected/708063fe-d579-4b1b-982d-63faeb84dac7-kube-api-access-w4ps9\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:21:28.587249 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:21:28.587032 5057 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/708063fe-d579-4b1b-982d-63faeb84dac7-istiod-ca-cert\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:21:28.587249 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:21:28.587044 5057 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/708063fe-d579-4b1b-982d-63faeb84dac7-istio-envoy\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:21:28.885988 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:21:28.885947 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:21:29.314408 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:21:29.314380 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-qpb7q" Oct 29 23:21:29.314408 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:21:29.314392 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-qpb7q" event={"ID":"708063fe-d579-4b1b-982d-63faeb84dac7","Type":"ContainerDied","Data":"e2011d07395a149958f8faab372c0e8f52f2b933c37191c1824191e413b48bc8"} Oct 29 23:21:29.314915 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:21:29.314462 5057 scope.go:122] "RemoveContainer" containerID="439ca738d78e30ca61c85ad476b4595de3d3bb044711cc2b9f212752831d4580" Oct 29 23:21:29.316922 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:21:29.316893 5057 generic.go:356] "Generic (PLEG): container finished" podID="c9e1f7d1-fb93-44da-9b30-8c27769dbbc9" containerID="d66683bb06b93b6757593cd6e707ffd3602aeb34fcbc66e6876ad26bea6d04a0" exitCode=0 Oct 29 23:21:29.317076 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:21:29.316964 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-058e69ab-ngh6t" event={"ID":"c9e1f7d1-fb93-44da-9b30-8c27769dbbc9","Type":"ContainerDied","Data":"d66683bb06b93b6757593cd6e707ffd3602aeb34fcbc66e6876ad26bea6d04a0"} Oct 29 23:21:29.372707 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:21:29.372678 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-qpb7q"] Oct 29 23:21:29.383060 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:21:29.383036 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-qpb7q"] Oct 29 23:21:30.490815 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:21:30.490775 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:21:30.534827 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:21:30.534799 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-058e69ab-ngh6t" Oct 29 23:21:30.621767 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:21:30.621673 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c9e1f7d1-fb93-44da-9b30-8c27769dbbc9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9e1f7d1-fb93-44da-9b30-8c27769dbbc9-maas-api-xks-ca\") pod \"c9e1f7d1-fb93-44da-9b30-8c27769dbbc9\" (UID: \"c9e1f7d1-fb93-44da-9b30-8c27769dbbc9\") " Oct 29 23:21:30.621952 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:21:30.621886 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9e1f7d1-fb93-44da-9b30-8c27769dbbc9-kube-api-access-kr4bn\" (UniqueName: \"kubernetes.io/projected/c9e1f7d1-fb93-44da-9b30-8c27769dbbc9-kube-api-access-kr4bn\") pod \"c9e1f7d1-fb93-44da-9b30-8c27769dbbc9\" (UID: \"c9e1f7d1-fb93-44da-9b30-8c27769dbbc9\") " Oct 29 23:21:30.621952 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:21:30.621926 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9e1f7d1-fb93-44da-9b30-8c27769dbbc9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9e1f7d1-fb93-44da-9b30-8c27769dbbc9-maas-api-openshift-ca\") pod \"c9e1f7d1-fb93-44da-9b30-8c27769dbbc9\" (UID: \"c9e1f7d1-fb93-44da-9b30-8c27769dbbc9\") " Oct 29 23:21:30.623482 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:21:30.623449 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9e1f7d1-fb93-44da-9b30-8c27769dbbc9-maas-api-openshift-ca" pod "c9e1f7d1-fb93-44da-9b30-8c27769dbbc9" (UID: "c9e1f7d1-fb93-44da-9b30-8c27769dbbc9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:21:30.633581 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:21:30.627834 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9e1f7d1-fb93-44da-9b30-8c27769dbbc9-kube-api-access-kr4bn" pod "c9e1f7d1-fb93-44da-9b30-8c27769dbbc9" (UID: "c9e1f7d1-fb93-44da-9b30-8c27769dbbc9"). InnerVolumeSpecName "kube-api-access-kr4bn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:21:30.634535 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:21:30.634509 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9e1f7d1-fb93-44da-9b30-8c27769dbbc9-maas-api-xks-ca" pod "c9e1f7d1-fb93-44da-9b30-8c27769dbbc9" (UID: "c9e1f7d1-fb93-44da-9b30-8c27769dbbc9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:21:30.724258 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:21:30.724226 5057 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9e1f7d1-fb93-44da-9b30-8c27769dbbc9-maas-api-xks-ca\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:21:30.724258 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:21:30.724256 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kr4bn\" (UniqueName: \"kubernetes.io/projected/c9e1f7d1-fb93-44da-9b30-8c27769dbbc9-kube-api-access-kr4bn\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:21:30.724258 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:21:30.724265 5057 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9e1f7d1-fb93-44da-9b30-8c27769dbbc9-maas-api-openshift-ca\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:21:31.335199 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:21:31.335164 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="708063fe-d579-4b1b-982d-63faeb84dac7" path="/var/lib/kubelet/pods/708063fe-d579-4b1b-982d-63faeb84dac7/volumes" Oct 29 23:21:31.351454 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:21:31.351419 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-058e69ab-ngh6t" event={"ID":"c9e1f7d1-fb93-44da-9b30-8c27769dbbc9","Type":"ContainerDied","Data":"f3409922075baadf1b43b390490a5fba7743e766279314603bfcd48ce5222964"} Oct 29 23:21:31.351614 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:21:31.351522 5057 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f3409922075baadf1b43b390490a5fba7743e766279314603bfcd48ce5222964" Oct 29 23:21:31.351614 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:21:31.351605 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-058e69ab-ngh6t" Oct 29 23:21:32.087167 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:21:32.087126 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:21:33.642394 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:21:33.642357 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:21:35.235400 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:21:35.235360 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:21:35.348961 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:21:35.348919 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:21:36.818708 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:21:36.818661 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:21:37.398430 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:21:37.398390 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:21:38.418099 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:21:38.417868 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:21:40.047297 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:21:40.047261 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:21:41.716030 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:21:41.715979 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:21:43.373273 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:21:43.373236 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:21:43.376529 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:21:43.376503 5057 scope.go:122] "RemoveContainer" containerID="cb9df601f3b58829063cb9ae2a55f6af59ad9c9cda67697f7b84636564b43690" Oct 29 23:21:43.389855 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:21:43.389819 5057 scope.go:122] "RemoveContainer" containerID="d510894b22f912db5a0119141c30dc848c0a216c90906143f4d9e835f21a5af7" Oct 29 23:21:43.758459 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:21:43.758424 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:21:44.929019 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:21:44.928985 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:21:46.575945 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:21:46.575902 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:21:48.309779 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:21:48.309733 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:21:49.035219 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:21:49.035181 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d2dnn" containerName="extractor" Oct 29 23:21:50.016932 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:21:50.016890 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:21:51.561935 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:21:51.561898 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:21:53.972834 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:21:53.972799 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:21:56.072595 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:21:56.072555 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-67363b-openshift-default-745d499d87-z5d9k"] Oct 29 23:21:56.073847 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:21:56.073761 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-67363b-openshift-default-745d499d87-z5d9k" podUID="76be25ad-e827-41d0-975d-a6bf7a1dcdee" containerName="istio-proxy" containerID="cri-o://93d5677e2d0a1bc0250b19d36222d04ede6231167ead1db5731bb56bf2c9bddf" gracePeriod=30 Oct 29 23:21:57.162655 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:21:57.162615 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:21:59.023445 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:21:59.023407 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:22:00.801965 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:00.801880 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:22:01.528405 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:01.528374 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-67363b-openshift-default-745d499d87-z5d9k" Oct 29 23:22:01.664485 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:01.664453 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76be25ad-e827-41d0-975d-a6bf7a1dcdee-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76be25ad-e827-41d0-975d-a6bf7a1dcdee-workload-certs\") pod \"76be25ad-e827-41d0-975d-a6bf7a1dcdee\" (UID: \"76be25ad-e827-41d0-975d-a6bf7a1dcdee\") " Oct 29 23:22:01.664686 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:01.664562 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76be25ad-e827-41d0-975d-a6bf7a1dcdee-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76be25ad-e827-41d0-975d-a6bf7a1dcdee-istio-envoy\") pod \"76be25ad-e827-41d0-975d-a6bf7a1dcdee\" (UID: \"76be25ad-e827-41d0-975d-a6bf7a1dcdee\") " Oct 29 23:22:01.664686 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:01.664613 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/76be25ad-e827-41d0-975d-a6bf7a1dcdee-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76be25ad-e827-41d0-975d-a6bf7a1dcdee-istiod-ca-cert\") pod \"76be25ad-e827-41d0-975d-a6bf7a1dcdee\" (UID: \"76be25ad-e827-41d0-975d-a6bf7a1dcdee\") " Oct 29 23:22:01.664686 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:01.664631 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76be25ad-e827-41d0-975d-a6bf7a1dcdee-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76be25ad-e827-41d0-975d-a6bf7a1dcdee-workload-socket\") pod \"76be25ad-e827-41d0-975d-a6bf7a1dcdee\" (UID: \"76be25ad-e827-41d0-975d-a6bf7a1dcdee\") " Oct 29 23:22:01.664876 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:01.664713 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76be25ad-e827-41d0-975d-a6bf7a1dcdee-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76be25ad-e827-41d0-975d-a6bf7a1dcdee-istio-data\") pod \"76be25ad-e827-41d0-975d-a6bf7a1dcdee\" (UID: \"76be25ad-e827-41d0-975d-a6bf7a1dcdee\") " Oct 29 23:22:01.664876 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:01.664760 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76be25ad-e827-41d0-975d-a6bf7a1dcdee-istio-token\" (UniqueName: \"kubernetes.io/projected/76be25ad-e827-41d0-975d-a6bf7a1dcdee-istio-token\") pod \"76be25ad-e827-41d0-975d-a6bf7a1dcdee\" (UID: \"76be25ad-e827-41d0-975d-a6bf7a1dcdee\") " Oct 29 23:22:01.664876 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:01.664770 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76be25ad-e827-41d0-975d-a6bf7a1dcdee-workload-certs" pod "76be25ad-e827-41d0-975d-a6bf7a1dcdee" (UID: "76be25ad-e827-41d0-975d-a6bf7a1dcdee"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:22:01.664876 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:01.664798 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76be25ad-e827-41d0-975d-a6bf7a1dcdee-kube-api-access-22llm\" (UniqueName: \"kubernetes.io/projected/76be25ad-e827-41d0-975d-a6bf7a1dcdee-kube-api-access-22llm\") pod \"76be25ad-e827-41d0-975d-a6bf7a1dcdee\" (UID: \"76be25ad-e827-41d0-975d-a6bf7a1dcdee\") " Oct 29 23:22:01.664876 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:01.664871 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/76be25ad-e827-41d0-975d-a6bf7a1dcdee-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76be25ad-e827-41d0-975d-a6bf7a1dcdee-istio-podinfo\") pod \"76be25ad-e827-41d0-975d-a6bf7a1dcdee\" (UID: \"76be25ad-e827-41d0-975d-a6bf7a1dcdee\") " Oct 29 23:22:01.665168 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:01.664918 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76be25ad-e827-41d0-975d-a6bf7a1dcdee-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76be25ad-e827-41d0-975d-a6bf7a1dcdee-credential-socket\") pod \"76be25ad-e827-41d0-975d-a6bf7a1dcdee\" (UID: \"76be25ad-e827-41d0-975d-a6bf7a1dcdee\") " Oct 29 23:22:01.665168 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:01.664980 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76be25ad-e827-41d0-975d-a6bf7a1dcdee-workload-socket" pod "76be25ad-e827-41d0-975d-a6bf7a1dcdee" (UID: "76be25ad-e827-41d0-975d-a6bf7a1dcdee"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:22:01.665168 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:01.665010 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76be25ad-e827-41d0-975d-a6bf7a1dcdee-istio-data" pod "76be25ad-e827-41d0-975d-a6bf7a1dcdee" (UID: "76be25ad-e827-41d0-975d-a6bf7a1dcdee"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:22:01.665168 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:01.665030 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76be25ad-e827-41d0-975d-a6bf7a1dcdee-istiod-ca-cert" pod "76be25ad-e827-41d0-975d-a6bf7a1dcdee" (UID: "76be25ad-e827-41d0-975d-a6bf7a1dcdee"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:22:01.665414 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:01.665199 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76be25ad-e827-41d0-975d-a6bf7a1dcdee-credential-socket" pod "76be25ad-e827-41d0-975d-a6bf7a1dcdee" (UID: "76be25ad-e827-41d0-975d-a6bf7a1dcdee"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:22:01.666161 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:01.666111 5057 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76be25ad-e827-41d0-975d-a6bf7a1dcdee-credential-socket\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:22:01.666161 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:01.666140 5057 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76be25ad-e827-41d0-975d-a6bf7a1dcdee-workload-certs\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:22:01.666161 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:01.666155 5057 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76be25ad-e827-41d0-975d-a6bf7a1dcdee-istiod-ca-cert\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:22:01.666443 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:01.666168 5057 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76be25ad-e827-41d0-975d-a6bf7a1dcdee-workload-socket\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:22:01.666443 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:01.666181 5057 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76be25ad-e827-41d0-975d-a6bf7a1dcdee-istio-data\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:22:01.667505 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:01.667477 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76be25ad-e827-41d0-975d-a6bf7a1dcdee-istio-token" pod "76be25ad-e827-41d0-975d-a6bf7a1dcdee" (UID: "76be25ad-e827-41d0-975d-a6bf7a1dcdee"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:22:01.667608 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:01.667516 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76be25ad-e827-41d0-975d-a6bf7a1dcdee-kube-api-access-22llm" pod "76be25ad-e827-41d0-975d-a6bf7a1dcdee" (UID: "76be25ad-e827-41d0-975d-a6bf7a1dcdee"). InnerVolumeSpecName "kube-api-access-22llm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:22:01.667608 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:01.667525 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/76be25ad-e827-41d0-975d-a6bf7a1dcdee-istio-podinfo" pod "76be25ad-e827-41d0-975d-a6bf7a1dcdee" (UID: "76be25ad-e827-41d0-975d-a6bf7a1dcdee"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:22:01.667959 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:01.667940 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76be25ad-e827-41d0-975d-a6bf7a1dcdee-istio-envoy" pod "76be25ad-e827-41d0-975d-a6bf7a1dcdee" (UID: "76be25ad-e827-41d0-975d-a6bf7a1dcdee"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:22:01.767359 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:01.767260 5057 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76be25ad-e827-41d0-975d-a6bf7a1dcdee-istio-token\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:22:01.767359 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:01.767292 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-22llm\" (UniqueName: \"kubernetes.io/projected/76be25ad-e827-41d0-975d-a6bf7a1dcdee-kube-api-access-22llm\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:22:01.767359 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:01.767304 5057 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76be25ad-e827-41d0-975d-a6bf7a1dcdee-istio-podinfo\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:22:01.767359 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:01.767312 5057 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76be25ad-e827-41d0-975d-a6bf7a1dcdee-istio-envoy\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:22:02.038608 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:02.038508 5057 generic.go:356] "Generic (PLEG): container finished" podID="76be25ad-e827-41d0-975d-a6bf7a1dcdee" containerID="93d5677e2d0a1bc0250b19d36222d04ede6231167ead1db5731bb56bf2c9bddf" exitCode=0 Oct 29 23:22:02.038608 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:02.038552 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-67363b-openshift-default-745d499d87-z5d9k" event={"ID":"76be25ad-e827-41d0-975d-a6bf7a1dcdee","Type":"ContainerDied","Data":"93d5677e2d0a1bc0250b19d36222d04ede6231167ead1db5731bb56bf2c9bddf"} Oct 29 23:22:02.039129 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:02.038561 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-67363b-openshift-default-745d499d87-z5d9k" Oct 29 23:22:02.039129 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:02.038637 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-67363b-openshift-default-745d499d87-z5d9k" event={"ID":"76be25ad-e827-41d0-975d-a6bf7a1dcdee","Type":"ContainerDied","Data":"968642abf6227f964a6e964b7996dc2a1c620ba1344b9a15244602d30992c67d"} Oct 29 23:22:02.039129 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:02.038664 5057 scope.go:122] "RemoveContainer" containerID="93d5677e2d0a1bc0250b19d36222d04ede6231167ead1db5731bb56bf2c9bddf" Oct 29 23:22:02.052893 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:02.052868 5057 scope.go:122] "RemoveContainer" containerID="93d5677e2d0a1bc0250b19d36222d04ede6231167ead1db5731bb56bf2c9bddf" Oct 29 23:22:02.053163 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:22:02.053143 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93d5677e2d0a1bc0250b19d36222d04ede6231167ead1db5731bb56bf2c9bddf\": container with ID starting with 93d5677e2d0a1bc0250b19d36222d04ede6231167ead1db5731bb56bf2c9bddf not found: ID does not exist" containerID="93d5677e2d0a1bc0250b19d36222d04ede6231167ead1db5731bb56bf2c9bddf" Oct 29 23:22:02.053232 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:02.053175 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93d5677e2d0a1bc0250b19d36222d04ede6231167ead1db5731bb56bf2c9bddf"} err="failed to get container status \"93d5677e2d0a1bc0250b19d36222d04ede6231167ead1db5731bb56bf2c9bddf\": rpc error: code = NotFound desc = could not find container \"93d5677e2d0a1bc0250b19d36222d04ede6231167ead1db5731bb56bf2c9bddf\": container with ID starting with 93d5677e2d0a1bc0250b19d36222d04ede6231167ead1db5731bb56bf2c9bddf not found: ID does not exist" Oct 29 23:22:02.144782 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:02.144749 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-67363b-openshift-default-745d499d87-z5d9k"] Oct 29 23:22:02.156559 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:02.156529 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-67363b-openshift-default-745d499d87-z5d9k"] Oct 29 23:22:02.637185 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:02.637146 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:22:03.333977 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:03.333945 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76be25ad-e827-41d0-975d-a6bf7a1dcdee" path="/var/lib/kubelet/pods/76be25ad-e827-41d0-975d-a6bf7a1dcdee/volumes" Oct 29 23:22:04.327021 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:04.326970 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:22:05.143466 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:05.143435 5057 patch_prober.go:29] interesting pod/e2e-inf-b-cc6328-openshift-default-84c4dfccc9-8k9hr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.219:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 23:22:05.161827 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:05.143505 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-cc6328-openshift-default-84c4dfccc9-8k9hr" podUID="268ae24d-3a42-472a-97d7-aeacd9989632" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.219:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 23:22:06.099099 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:06.099064 5057 generic.go:356] "Generic (PLEG): container finished" podID="268ae24d-3a42-472a-97d7-aeacd9989632" containerID="060923ec45c1ee57ef9922829f4a5166a251b06042203ba607f48932a190b092" exitCode=0 Oct 29 23:22:06.099308 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:06.099142 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-cc6328-openshift-default-84c4dfccc9-8k9hr" event={"ID":"268ae24d-3a42-472a-97d7-aeacd9989632","Type":"ContainerDied","Data":"060923ec45c1ee57ef9922829f4a5166a251b06042203ba607f48932a190b092"} Oct 29 23:22:06.099655 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:06.099632 5057 scope.go:122] "RemoveContainer" containerID="060923ec45c1ee57ef9922829f4a5166a251b06042203ba607f48932a190b092" Oct 29 23:22:06.102145 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:06.102109 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 23:22:06.102262 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:06.102201 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 23:22:06.102262 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:06.102248 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 23:22:06.206233 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:06.206181 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:22:06.762844 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:06.762757 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:22:07.113703 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:07.113619 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-cc6328-openshift-default-84c4dfccc9-8k9hr" event={"ID":"268ae24d-3a42-472a-97d7-aeacd9989632","Type":"ContainerStarted","Data":"64462bcdeb77a27dc04b5f2652ad8fcb2fd0f9841fd47f70ac2fa2eded568cf4"} Oct 29 23:22:07.447596 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:07.447572 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-cc6328-openshift-default-84c4dfccc9-8k9hr" Oct 29 23:22:07.448018 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:07.447835 5057 patch_prober.go:29] interesting pod/e2e-inf-b-cc6328-openshift-default-84c4dfccc9-8k9hr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" start-of-body= Oct 29 23:22:07.448018 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:07.447902 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-cc6328-openshift-default-84c4dfccc9-8k9hr" podUID="268ae24d-3a42-472a-97d7-aeacd9989632" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" Oct 29 23:22:07.503063 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:07.503023 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:22:08.446169 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:08.446135 5057 patch_prober.go:29] interesting pod/e2e-inf-b-cc6328-openshift-default-84c4dfccc9-8k9hr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" start-of-body= Oct 29 23:22:08.446419 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:08.446190 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-cc6328-openshift-default-84c4dfccc9-8k9hr" podUID="268ae24d-3a42-472a-97d7-aeacd9989632" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" Oct 29 23:22:08.853985 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:08.853947 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:22:09.445784 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:09.445749 5057 patch_prober.go:29] interesting pod/e2e-inf-b-cc6328-openshift-default-84c4dfccc9-8k9hr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" start-of-body= Oct 29 23:22:09.445994 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:09.445817 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-cc6328-openshift-default-84c4dfccc9-8k9hr" podUID="268ae24d-3a42-472a-97d7-aeacd9989632" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" Oct 29 23:22:10.445909 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:10.445873 5057 patch_prober.go:29] interesting pod/e2e-inf-b-cc6328-openshift-default-84c4dfccc9-8k9hr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" start-of-body= Oct 29 23:22:10.446311 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:10.445923 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-cc6328-openshift-default-84c4dfccc9-8k9hr" podUID="268ae24d-3a42-472a-97d7-aeacd9989632" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" Oct 29 23:22:10.530106 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:10.530040 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-1056cc-openshift-default-7b4b9768c9-vsbgm"] Oct 29 23:22:10.531996 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:10.531968 5057 cpu_manager.go:414] "containerMap: removing container" podUID="708063fe-d579-4b1b-982d-63faeb84dac7" containerName="istio-proxy" Oct 29 23:22:10.532163 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:10.532150 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="708063fe-d579-4b1b-982d-63faeb84dac7" containerName="istio-proxy" Oct 29 23:22:10.532295 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:10.532285 5057 cpu_manager.go:414] "containerMap: removing container" podUID="76be25ad-e827-41d0-975d-a6bf7a1dcdee" containerName="istio-proxy" Oct 29 23:22:10.532391 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:10.532380 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76be25ad-e827-41d0-975d-a6bf7a1dcdee" containerName="istio-proxy" Oct 29 23:22:10.532520 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:10.532510 5057 cpu_manager.go:414] "containerMap: removing container" podUID="c9e1f7d1-fb93-44da-9b30-8c27769dbbc9" containerName="revoke-keys" Oct 29 23:22:10.532594 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:10.532584 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9e1f7d1-fb93-44da-9b30-8c27769dbbc9" containerName="revoke-keys" Oct 29 23:22:10.532897 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:10.532885 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="708063fe-d579-4b1b-982d-63faeb84dac7" containerName="istio-proxy" Oct 29 23:22:10.532998 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:10.532989 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="76be25ad-e827-41d0-975d-a6bf7a1dcdee" containerName="istio-proxy" Oct 29 23:22:10.533081 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:10.533073 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9e1f7d1-fb93-44da-9b30-8c27769dbbc9" containerName="revoke-keys" Oct 29 23:22:10.539390 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:10.539236 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1056cc-openshift-default-7b4b9768c9-vsbgm" Oct 29 23:22:10.543632 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:10.543605 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-1056cc-openshift-default-7b4b9768c9-vsbgm"] Oct 29 23:22:10.556372 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:10.554674 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-1056cc-openshift-default-dockercfg-6p8ms\"" Oct 29 23:22:10.621744 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:10.621707 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:22:10.698407 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:10.681781 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21034c72-9803-4050-b5b5-df014d3668e4-istio-envoy\") pod \"e2e-inf-a-1056cc-openshift-default-7b4b9768c9-vsbgm\" (UID: \"21034c72-9803-4050-b5b5-df014d3668e4\") " pod="openshift-ingress/e2e-inf-a-1056cc-openshift-default-7b4b9768c9-vsbgm" Oct 29 23:22:10.698407 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:10.681840 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhx4l\" (UniqueName: \"kubernetes.io/projected/21034c72-9803-4050-b5b5-df014d3668e4-kube-api-access-vhx4l\") pod \"e2e-inf-a-1056cc-openshift-default-7b4b9768c9-vsbgm\" (UID: \"21034c72-9803-4050-b5b5-df014d3668e4\") " pod="openshift-ingress/e2e-inf-a-1056cc-openshift-default-7b4b9768c9-vsbgm" Oct 29 23:22:10.698407 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:10.681910 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21034c72-9803-4050-b5b5-df014d3668e4-workload-socket\") pod \"e2e-inf-a-1056cc-openshift-default-7b4b9768c9-vsbgm\" (UID: \"21034c72-9803-4050-b5b5-df014d3668e4\") " pod="openshift-ingress/e2e-inf-a-1056cc-openshift-default-7b4b9768c9-vsbgm" Oct 29 23:22:10.698407 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:10.681967 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21034c72-9803-4050-b5b5-df014d3668e4-workload-certs\") pod \"e2e-inf-a-1056cc-openshift-default-7b4b9768c9-vsbgm\" (UID: \"21034c72-9803-4050-b5b5-df014d3668e4\") " pod="openshift-ingress/e2e-inf-a-1056cc-openshift-default-7b4b9768c9-vsbgm" Oct 29 23:22:10.698407 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:10.682025 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21034c72-9803-4050-b5b5-df014d3668e4-istiod-ca-cert\") pod \"e2e-inf-a-1056cc-openshift-default-7b4b9768c9-vsbgm\" (UID: \"21034c72-9803-4050-b5b5-df014d3668e4\") " pod="openshift-ingress/e2e-inf-a-1056cc-openshift-default-7b4b9768c9-vsbgm" Oct 29 23:22:10.698407 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:10.682109 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21034c72-9803-4050-b5b5-df014d3668e4-istio-data\") pod \"e2e-inf-a-1056cc-openshift-default-7b4b9768c9-vsbgm\" (UID: \"21034c72-9803-4050-b5b5-df014d3668e4\") " pod="openshift-ingress/e2e-inf-a-1056cc-openshift-default-7b4b9768c9-vsbgm" Oct 29 23:22:10.698407 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:10.682253 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21034c72-9803-4050-b5b5-df014d3668e4-credential-socket\") pod \"e2e-inf-a-1056cc-openshift-default-7b4b9768c9-vsbgm\" (UID: \"21034c72-9803-4050-b5b5-df014d3668e4\") " pod="openshift-ingress/e2e-inf-a-1056cc-openshift-default-7b4b9768c9-vsbgm" Oct 29 23:22:10.698407 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:10.682358 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21034c72-9803-4050-b5b5-df014d3668e4-istio-token\") pod \"e2e-inf-a-1056cc-openshift-default-7b4b9768c9-vsbgm\" (UID: \"21034c72-9803-4050-b5b5-df014d3668e4\") " pod="openshift-ingress/e2e-inf-a-1056cc-openshift-default-7b4b9768c9-vsbgm" Oct 29 23:22:10.698407 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:10.682466 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21034c72-9803-4050-b5b5-df014d3668e4-istio-podinfo\") pod \"e2e-inf-a-1056cc-openshift-default-7b4b9768c9-vsbgm\" (UID: \"21034c72-9803-4050-b5b5-df014d3668e4\") " pod="openshift-ingress/e2e-inf-a-1056cc-openshift-default-7b4b9768c9-vsbgm" Oct 29 23:22:10.760956 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:10.760926 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:22:10.802779 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:10.802505 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21034c72-9803-4050-b5b5-df014d3668e4-istio-podinfo\") pod \"e2e-inf-a-1056cc-openshift-default-7b4b9768c9-vsbgm\" (UID: \"21034c72-9803-4050-b5b5-df014d3668e4\") " pod="openshift-ingress/e2e-inf-a-1056cc-openshift-default-7b4b9768c9-vsbgm" Oct 29 23:22:10.802779 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:10.802612 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21034c72-9803-4050-b5b5-df014d3668e4-istio-envoy\") pod \"e2e-inf-a-1056cc-openshift-default-7b4b9768c9-vsbgm\" (UID: \"21034c72-9803-4050-b5b5-df014d3668e4\") " pod="openshift-ingress/e2e-inf-a-1056cc-openshift-default-7b4b9768c9-vsbgm" Oct 29 23:22:10.802779 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:10.802653 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhx4l\" (UniqueName: \"kubernetes.io/projected/21034c72-9803-4050-b5b5-df014d3668e4-kube-api-access-vhx4l\") pod \"e2e-inf-a-1056cc-openshift-default-7b4b9768c9-vsbgm\" (UID: \"21034c72-9803-4050-b5b5-df014d3668e4\") " pod="openshift-ingress/e2e-inf-a-1056cc-openshift-default-7b4b9768c9-vsbgm" Oct 29 23:22:10.802779 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:10.802719 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21034c72-9803-4050-b5b5-df014d3668e4-workload-socket\") pod \"e2e-inf-a-1056cc-openshift-default-7b4b9768c9-vsbgm\" (UID: \"21034c72-9803-4050-b5b5-df014d3668e4\") " pod="openshift-ingress/e2e-inf-a-1056cc-openshift-default-7b4b9768c9-vsbgm" Oct 29 23:22:10.803187 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:10.803156 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21034c72-9803-4050-b5b5-df014d3668e4-workload-socket\") pod \"e2e-inf-a-1056cc-openshift-default-7b4b9768c9-vsbgm\" (UID: \"21034c72-9803-4050-b5b5-df014d3668e4\") " pod="openshift-ingress/e2e-inf-a-1056cc-openshift-default-7b4b9768c9-vsbgm" Oct 29 23:22:10.803269 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:10.803252 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21034c72-9803-4050-b5b5-df014d3668e4-workload-certs\") pod \"e2e-inf-a-1056cc-openshift-default-7b4b9768c9-vsbgm\" (UID: \"21034c72-9803-4050-b5b5-df014d3668e4\") " pod="openshift-ingress/e2e-inf-a-1056cc-openshift-default-7b4b9768c9-vsbgm" Oct 29 23:22:10.803350 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:10.803312 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21034c72-9803-4050-b5b5-df014d3668e4-istiod-ca-cert\") pod \"e2e-inf-a-1056cc-openshift-default-7b4b9768c9-vsbgm\" (UID: \"21034c72-9803-4050-b5b5-df014d3668e4\") " pod="openshift-ingress/e2e-inf-a-1056cc-openshift-default-7b4b9768c9-vsbgm" Oct 29 23:22:10.811411 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:10.803414 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21034c72-9803-4050-b5b5-df014d3668e4-istio-data\") pod \"e2e-inf-a-1056cc-openshift-default-7b4b9768c9-vsbgm\" (UID: \"21034c72-9803-4050-b5b5-df014d3668e4\") " pod="openshift-ingress/e2e-inf-a-1056cc-openshift-default-7b4b9768c9-vsbgm" Oct 29 23:22:10.811411 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:10.803555 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21034c72-9803-4050-b5b5-df014d3668e4-credential-socket\") pod \"e2e-inf-a-1056cc-openshift-default-7b4b9768c9-vsbgm\" (UID: \"21034c72-9803-4050-b5b5-df014d3668e4\") " pod="openshift-ingress/e2e-inf-a-1056cc-openshift-default-7b4b9768c9-vsbgm" Oct 29 23:22:10.811411 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:10.803637 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21034c72-9803-4050-b5b5-df014d3668e4-istio-token\") pod \"e2e-inf-a-1056cc-openshift-default-7b4b9768c9-vsbgm\" (UID: \"21034c72-9803-4050-b5b5-df014d3668e4\") " pod="openshift-ingress/e2e-inf-a-1056cc-openshift-default-7b4b9768c9-vsbgm" Oct 29 23:22:10.811411 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:10.808286 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21034c72-9803-4050-b5b5-df014d3668e4-istiod-ca-cert\") pod \"e2e-inf-a-1056cc-openshift-default-7b4b9768c9-vsbgm\" (UID: \"21034c72-9803-4050-b5b5-df014d3668e4\") " pod="openshift-ingress/e2e-inf-a-1056cc-openshift-default-7b4b9768c9-vsbgm" Oct 29 23:22:10.811411 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:10.808565 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21034c72-9803-4050-b5b5-df014d3668e4-workload-certs\") pod \"e2e-inf-a-1056cc-openshift-default-7b4b9768c9-vsbgm\" (UID: \"21034c72-9803-4050-b5b5-df014d3668e4\") " pod="openshift-ingress/e2e-inf-a-1056cc-openshift-default-7b4b9768c9-vsbgm" Oct 29 23:22:10.811411 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:10.808793 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21034c72-9803-4050-b5b5-df014d3668e4-istio-data\") pod \"e2e-inf-a-1056cc-openshift-default-7b4b9768c9-vsbgm\" (UID: \"21034c72-9803-4050-b5b5-df014d3668e4\") " pod="openshift-ingress/e2e-inf-a-1056cc-openshift-default-7b4b9768c9-vsbgm" Oct 29 23:22:10.811411 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:10.809021 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21034c72-9803-4050-b5b5-df014d3668e4-credential-socket\") pod \"e2e-inf-a-1056cc-openshift-default-7b4b9768c9-vsbgm\" (UID: \"21034c72-9803-4050-b5b5-df014d3668e4\") " pod="openshift-ingress/e2e-inf-a-1056cc-openshift-default-7b4b9768c9-vsbgm" Oct 29 23:22:10.814578 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:10.814550 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21034c72-9803-4050-b5b5-df014d3668e4-istio-podinfo\") pod \"e2e-inf-a-1056cc-openshift-default-7b4b9768c9-vsbgm\" (UID: \"21034c72-9803-4050-b5b5-df014d3668e4\") " pod="openshift-ingress/e2e-inf-a-1056cc-openshift-default-7b4b9768c9-vsbgm" Oct 29 23:22:10.815086 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:10.815059 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21034c72-9803-4050-b5b5-df014d3668e4-istio-envoy\") pod \"e2e-inf-a-1056cc-openshift-default-7b4b9768c9-vsbgm\" (UID: \"21034c72-9803-4050-b5b5-df014d3668e4\") " pod="openshift-ingress/e2e-inf-a-1056cc-openshift-default-7b4b9768c9-vsbgm" Oct 29 23:22:10.836431 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:10.836396 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21034c72-9803-4050-b5b5-df014d3668e4-istio-token\") pod \"e2e-inf-a-1056cc-openshift-default-7b4b9768c9-vsbgm\" (UID: \"21034c72-9803-4050-b5b5-df014d3668e4\") " pod="openshift-ingress/e2e-inf-a-1056cc-openshift-default-7b4b9768c9-vsbgm" Oct 29 23:22:10.838420 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:10.838398 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhx4l\" (UniqueName: \"kubernetes.io/projected/21034c72-9803-4050-b5b5-df014d3668e4-kube-api-access-vhx4l\") pod \"e2e-inf-a-1056cc-openshift-default-7b4b9768c9-vsbgm\" (UID: \"21034c72-9803-4050-b5b5-df014d3668e4\") " pod="openshift-ingress/e2e-inf-a-1056cc-openshift-default-7b4b9768c9-vsbgm" Oct 29 23:22:10.869035 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:10.869015 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1056cc-openshift-default-7b4b9768c9-vsbgm" Oct 29 23:22:11.423195 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:11.423076 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-1056cc-openshift-default-7b4b9768c9-vsbgm"] Oct 29 23:22:11.425000 ip-10-0-140-74 kubenswrapper[5057]: W1029 23:22:11.424962 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21034c72_9803_4050_b5b5_df014d3668e4.slice/crio-32746f43afcb0588619144ef6bd702aac8f9665448a3281ab7c5eab24023eb84 WatchSource:0}: Error finding container 32746f43afcb0588619144ef6bd702aac8f9665448a3281ab7c5eab24023eb84: Status 404 returned error can't find the container with id 32746f43afcb0588619144ef6bd702aac8f9665448a3281ab7c5eab24023eb84 Oct 29 23:22:11.428683 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:11.428646 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 23:22:11.428787 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:11.428717 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 23:22:11.428787 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:11.428748 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 23:22:11.445946 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:11.445920 5057 patch_prober.go:29] interesting pod/e2e-inf-b-cc6328-openshift-default-84c4dfccc9-8k9hr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" start-of-body= Oct 29 23:22:11.446374 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:11.445974 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-cc6328-openshift-default-84c4dfccc9-8k9hr" podUID="268ae24d-3a42-472a-97d7-aeacd9989632" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" Oct 29 23:22:12.180356 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:12.178202 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1056cc-openshift-default-7b4b9768c9-vsbgm" event={"ID":"21034c72-9803-4050-b5b5-df014d3668e4","Type":"ContainerStarted","Data":"8483f8d3504552b3c392e1b0198deb2044ee95f156ee1c04f18c4f050688ae69"} Oct 29 23:22:12.180356 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:12.178265 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1056cc-openshift-default-7b4b9768c9-vsbgm" event={"ID":"21034c72-9803-4050-b5b5-df014d3668e4","Type":"ContainerStarted","Data":"32746f43afcb0588619144ef6bd702aac8f9665448a3281ab7c5eab24023eb84"} Oct 29 23:22:12.254621 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:12.254560 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-1056cc-openshift-default-7b4b9768c9-vsbgm" podStartSLOduration=2.254543374 podStartE2EDuration="2.254543374s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:22:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:22:12.236355873 +0000 UTC m=+2913.019569313" watchObservedRunningTime="2026-10-29 23:22:12.254543374 +0000 UTC m=+2913.037756800" Oct 29 23:22:12.337856 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:12.337802 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:22:12.454432 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:12.449203 5057 patch_prober.go:29] interesting pod/e2e-inf-b-cc6328-openshift-default-84c4dfccc9-8k9hr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" start-of-body= Oct 29 23:22:12.454432 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:12.449259 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-cc6328-openshift-default-84c4dfccc9-8k9hr" podUID="268ae24d-3a42-472a-97d7-aeacd9989632" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" Oct 29 23:22:12.870176 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:12.870084 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-1056cc-openshift-default-7b4b9768c9-vsbgm" Oct 29 23:22:12.871688 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:12.871644 5057 patch_prober.go:29] interesting pod/e2e-inf-a-1056cc-openshift-default-7b4b9768c9-vsbgm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused" start-of-body= Oct 29 23:22:12.871874 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:12.871699 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1056cc-openshift-default-7b4b9768c9-vsbgm" podUID="21034c72-9803-4050-b5b5-df014d3668e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused" Oct 29 23:22:13.450761 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:13.450732 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-cc6328-openshift-default-84c4dfccc9-8k9hr" Oct 29 23:22:13.451242 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:13.451227 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-cc6328-openshift-default-84c4dfccc9-8k9hr" Oct 29 23:22:13.451829 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:13.451795 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-cc6328-openshift-default-84c4dfccc9-8k9hr" Oct 29 23:22:13.870369 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:13.870264 5057 patch_prober.go:29] interesting pod/e2e-inf-a-1056cc-openshift-default-7b4b9768c9-vsbgm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused" start-of-body= Oct 29 23:22:13.870369 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:13.870317 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1056cc-openshift-default-7b4b9768c9-vsbgm" podUID="21034c72-9803-4050-b5b5-df014d3668e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused" Oct 29 23:22:14.019541 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:14.019503 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:22:14.869998 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:14.869963 5057 patch_prober.go:29] interesting pod/e2e-inf-a-1056cc-openshift-default-7b4b9768c9-vsbgm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused" start-of-body= Oct 29 23:22:14.870160 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:14.870010 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1056cc-openshift-default-7b4b9768c9-vsbgm" podUID="21034c72-9803-4050-b5b5-df014d3668e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused" Oct 29 23:22:15.627455 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:15.627417 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:22:15.869664 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:15.869623 5057 patch_prober.go:29] interesting pod/e2e-inf-a-1056cc-openshift-default-7b4b9768c9-vsbgm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused" start-of-body= Oct 29 23:22:15.869860 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:15.869692 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1056cc-openshift-default-7b4b9768c9-vsbgm" podUID="21034c72-9803-4050-b5b5-df014d3668e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused" Oct 29 23:22:16.869441 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:16.869406 5057 patch_prober.go:29] interesting pod/e2e-inf-a-1056cc-openshift-default-7b4b9768c9-vsbgm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused" start-of-body= Oct 29 23:22:16.869834 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:16.869454 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1056cc-openshift-default-7b4b9768c9-vsbgm" podUID="21034c72-9803-4050-b5b5-df014d3668e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused" Oct 29 23:22:17.155140 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:17.155056 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vm822"] Oct 29 23:22:17.176140 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:17.175941 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vm822" Oct 29 23:22:17.233498 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:17.233469 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vm822"] Oct 29 23:22:17.315233 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:17.315197 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:22:17.321010 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:17.320974 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef4e2b67-0954-4fe1-a2b4-223771c24e79-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-vm822\" (UID: \"ef4e2b67-0954-4fe1-a2b4-223771c24e79\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vm822" Oct 29 23:22:17.321149 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:17.321077 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef4e2b67-0954-4fe1-a2b4-223771c24e79-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-vm822\" (UID: \"ef4e2b67-0954-4fe1-a2b4-223771c24e79\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vm822" Oct 29 23:22:17.321214 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:17.321200 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef4e2b67-0954-4fe1-a2b4-223771c24e79-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-vm822\" (UID: \"ef4e2b67-0954-4fe1-a2b4-223771c24e79\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vm822" Oct 29 23:22:17.321253 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:17.321240 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef4e2b67-0954-4fe1-a2b4-223771c24e79-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-vm822\" (UID: \"ef4e2b67-0954-4fe1-a2b4-223771c24e79\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vm822" Oct 29 23:22:17.321290 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:17.321279 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef4e2b67-0954-4fe1-a2b4-223771c24e79-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-vm822\" (UID: \"ef4e2b67-0954-4fe1-a2b4-223771c24e79\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vm822" Oct 29 23:22:17.321324 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:17.321307 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef4e2b67-0954-4fe1-a2b4-223771c24e79-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-vm822\" (UID: \"ef4e2b67-0954-4fe1-a2b4-223771c24e79\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vm822" Oct 29 23:22:17.321436 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:17.321361 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef4e2b67-0954-4fe1-a2b4-223771c24e79-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-vm822\" (UID: \"ef4e2b67-0954-4fe1-a2b4-223771c24e79\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vm822" Oct 29 23:22:17.321436 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:17.321427 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9djc\" (UniqueName: \"kubernetes.io/projected/ef4e2b67-0954-4fe1-a2b4-223771c24e79-kube-api-access-b9djc\") pod \"maas-default-gateway-openshift-default-6db586dd6c-vm822\" (UID: \"ef4e2b67-0954-4fe1-a2b4-223771c24e79\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vm822" Oct 29 23:22:17.321522 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:17.321471 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef4e2b67-0954-4fe1-a2b4-223771c24e79-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-vm822\" (UID: \"ef4e2b67-0954-4fe1-a2b4-223771c24e79\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vm822" Oct 29 23:22:17.436349 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:17.428845 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9djc\" (UniqueName: \"kubernetes.io/projected/ef4e2b67-0954-4fe1-a2b4-223771c24e79-kube-api-access-b9djc\") pod \"maas-default-gateway-openshift-default-6db586dd6c-vm822\" (UID: \"ef4e2b67-0954-4fe1-a2b4-223771c24e79\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vm822" Oct 29 23:22:17.436349 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:17.428915 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef4e2b67-0954-4fe1-a2b4-223771c24e79-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-vm822\" (UID: \"ef4e2b67-0954-4fe1-a2b4-223771c24e79\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vm822" Oct 29 23:22:17.436349 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:17.429118 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef4e2b67-0954-4fe1-a2b4-223771c24e79-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-vm822\" (UID: \"ef4e2b67-0954-4fe1-a2b4-223771c24e79\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vm822" Oct 29 23:22:17.436349 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:17.429203 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef4e2b67-0954-4fe1-a2b4-223771c24e79-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-vm822\" (UID: \"ef4e2b67-0954-4fe1-a2b4-223771c24e79\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vm822" Oct 29 23:22:17.436349 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:17.429325 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef4e2b67-0954-4fe1-a2b4-223771c24e79-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-vm822\" (UID: \"ef4e2b67-0954-4fe1-a2b4-223771c24e79\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vm822" Oct 29 23:22:17.436349 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:17.429384 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef4e2b67-0954-4fe1-a2b4-223771c24e79-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-vm822\" (UID: \"ef4e2b67-0954-4fe1-a2b4-223771c24e79\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vm822" Oct 29 23:22:17.436349 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:17.429425 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef4e2b67-0954-4fe1-a2b4-223771c24e79-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-vm822\" (UID: \"ef4e2b67-0954-4fe1-a2b4-223771c24e79\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vm822" Oct 29 23:22:17.436349 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:17.429457 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef4e2b67-0954-4fe1-a2b4-223771c24e79-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-vm822\" (UID: \"ef4e2b67-0954-4fe1-a2b4-223771c24e79\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vm822" Oct 29 23:22:17.436349 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:17.429500 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef4e2b67-0954-4fe1-a2b4-223771c24e79-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-vm822\" (UID: \"ef4e2b67-0954-4fe1-a2b4-223771c24e79\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vm822" Oct 29 23:22:17.436349 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:17.429986 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef4e2b67-0954-4fe1-a2b4-223771c24e79-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-vm822\" (UID: \"ef4e2b67-0954-4fe1-a2b4-223771c24e79\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vm822" Oct 29 23:22:17.436349 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:17.433275 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef4e2b67-0954-4fe1-a2b4-223771c24e79-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-vm822\" (UID: \"ef4e2b67-0954-4fe1-a2b4-223771c24e79\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vm822" Oct 29 23:22:17.436349 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:17.433716 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef4e2b67-0954-4fe1-a2b4-223771c24e79-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-vm822\" (UID: \"ef4e2b67-0954-4fe1-a2b4-223771c24e79\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vm822" Oct 29 23:22:17.436349 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:17.434399 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef4e2b67-0954-4fe1-a2b4-223771c24e79-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-vm822\" (UID: \"ef4e2b67-0954-4fe1-a2b4-223771c24e79\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vm822" Oct 29 23:22:17.436349 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:17.435026 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef4e2b67-0954-4fe1-a2b4-223771c24e79-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-vm822\" (UID: \"ef4e2b67-0954-4fe1-a2b4-223771c24e79\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vm822" Oct 29 23:22:17.439706 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:17.439436 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef4e2b67-0954-4fe1-a2b4-223771c24e79-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-vm822\" (UID: \"ef4e2b67-0954-4fe1-a2b4-223771c24e79\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vm822" Oct 29 23:22:17.444505 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:17.444454 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef4e2b67-0954-4fe1-a2b4-223771c24e79-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-vm822\" (UID: \"ef4e2b67-0954-4fe1-a2b4-223771c24e79\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vm822" Oct 29 23:22:17.454025 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:17.453307 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9djc\" (UniqueName: \"kubernetes.io/projected/ef4e2b67-0954-4fe1-a2b4-223771c24e79-kube-api-access-b9djc\") pod \"maas-default-gateway-openshift-default-6db586dd6c-vm822\" (UID: \"ef4e2b67-0954-4fe1-a2b4-223771c24e79\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vm822" Oct 29 23:22:17.455978 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:17.455942 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef4e2b67-0954-4fe1-a2b4-223771c24e79-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-vm822\" (UID: \"ef4e2b67-0954-4fe1-a2b4-223771c24e79\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vm822" Oct 29 23:22:17.533544 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:17.533507 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vm822" Oct 29 23:22:17.869963 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:17.869927 5057 patch_prober.go:29] interesting pod/e2e-inf-a-1056cc-openshift-default-7b4b9768c9-vsbgm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused" start-of-body= Oct 29 23:22:17.870384 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:17.869988 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1056cc-openshift-default-7b4b9768c9-vsbgm" podUID="21034c72-9803-4050-b5b5-df014d3668e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused" Oct 29 23:22:17.926387 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:17.925324 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vm822"] Oct 29 23:22:17.932673 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:17.932635 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 23:22:17.932748 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:17.932731 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 23:22:17.932786 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:17.932771 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 23:22:18.277625 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:18.277581 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vm822" event={"ID":"ef4e2b67-0954-4fe1-a2b4-223771c24e79","Type":"ContainerStarted","Data":"6e37281a47dce4d2ee8de518de5bce7c1165b6b4bb963b3b0e3d8fe1829c1aa7"} Oct 29 23:22:18.277832 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:18.277635 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vm822" event={"ID":"ef4e2b67-0954-4fe1-a2b4-223771c24e79","Type":"ContainerStarted","Data":"5ae8db59974be3a373dfa94f0d808a4b13dd3948c328c066dcbbc779ae6f8f55"} Oct 29 23:22:18.325884 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:18.324607 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vm822" podStartSLOduration=1.324588975 podStartE2EDuration="1.324588975s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:22:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:22:18.322251517 +0000 UTC m=+2919.105464947" watchObservedRunningTime="2026-10-29 23:22:18.324588975 +0000 UTC m=+2919.107802440" Oct 29 23:22:18.534451 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:18.534374 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vm822" Oct 29 23:22:18.874667 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:18.874590 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-1056cc-openshift-default-7b4b9768c9-vsbgm" Oct 29 23:22:18.875105 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:18.875083 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-1056cc-openshift-default-7b4b9768c9-vsbgm" Oct 29 23:22:18.875739 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:18.875717 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-1056cc-openshift-default-7b4b9768c9-vsbgm" Oct 29 23:22:19.047523 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:22:19.047481 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d2dnn" containerName="extractor" Oct 29 23:22:19.258376 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:19.258340 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:22:19.535224 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:19.535144 5057 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-vm822 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.229:15021/healthz/ready\": dial tcp 10.132.0.229:15021: connect: connection refused" start-of-body= Oct 29 23:22:19.535224 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:19.535200 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vm822" podUID="ef4e2b67-0954-4fe1-a2b4-223771c24e79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.229:15021/healthz/ready\": dial tcp 10.132.0.229:15021: connect: connection refused" Oct 29 23:22:19.837565 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:19.837484 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-1a8107-openshift-default-7fff599fbf-rtjrx"] Oct 29 23:22:19.842479 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:19.842456 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-1a8107-openshift-default-7fff599fbf-rtjrx" Oct 29 23:22:19.857058 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:19.857036 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-1a8107-openshift-default-dockercfg-9g58t\"" Oct 29 23:22:19.869453 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:19.869428 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-1a8107-openshift-default-7fff599fbf-rtjrx"] Oct 29 23:22:20.014112 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:19.987592 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a1427f2-b35c-4355-9997-cde832700d7f-istio-podinfo\") pod \"e2e-inf-b-1a8107-openshift-default-7fff599fbf-rtjrx\" (UID: \"1a1427f2-b35c-4355-9997-cde832700d7f\") " pod="openshift-ingress/e2e-inf-b-1a8107-openshift-default-7fff599fbf-rtjrx" Oct 29 23:22:20.017202 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:20.017172 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a1427f2-b35c-4355-9997-cde832700d7f-istio-envoy\") pod \"e2e-inf-b-1a8107-openshift-default-7fff599fbf-rtjrx\" (UID: \"1a1427f2-b35c-4355-9997-cde832700d7f\") " pod="openshift-ingress/e2e-inf-b-1a8107-openshift-default-7fff599fbf-rtjrx" Oct 29 23:22:20.017509 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:20.017484 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a1427f2-b35c-4355-9997-cde832700d7f-istiod-ca-cert\") pod \"e2e-inf-b-1a8107-openshift-default-7fff599fbf-rtjrx\" (UID: \"1a1427f2-b35c-4355-9997-cde832700d7f\") " pod="openshift-ingress/e2e-inf-b-1a8107-openshift-default-7fff599fbf-rtjrx" Oct 29 23:22:20.017684 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:20.017667 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a1427f2-b35c-4355-9997-cde832700d7f-workload-socket\") pod \"e2e-inf-b-1a8107-openshift-default-7fff599fbf-rtjrx\" (UID: \"1a1427f2-b35c-4355-9997-cde832700d7f\") " pod="openshift-ingress/e2e-inf-b-1a8107-openshift-default-7fff599fbf-rtjrx" Oct 29 23:22:20.017865 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:20.017848 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a1427f2-b35c-4355-9997-cde832700d7f-istio-token\") pod \"e2e-inf-b-1a8107-openshift-default-7fff599fbf-rtjrx\" (UID: \"1a1427f2-b35c-4355-9997-cde832700d7f\") " pod="openshift-ingress/e2e-inf-b-1a8107-openshift-default-7fff599fbf-rtjrx" Oct 29 23:22:20.018061 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:20.018043 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a1427f2-b35c-4355-9997-cde832700d7f-workload-certs\") pod \"e2e-inf-b-1a8107-openshift-default-7fff599fbf-rtjrx\" (UID: \"1a1427f2-b35c-4355-9997-cde832700d7f\") " pod="openshift-ingress/e2e-inf-b-1a8107-openshift-default-7fff599fbf-rtjrx" Oct 29 23:22:20.018223 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:20.018204 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9nz8t\" (UniqueName: \"kubernetes.io/projected/1a1427f2-b35c-4355-9997-cde832700d7f-kube-api-access-9nz8t\") pod \"e2e-inf-b-1a8107-openshift-default-7fff599fbf-rtjrx\" (UID: \"1a1427f2-b35c-4355-9997-cde832700d7f\") " pod="openshift-ingress/e2e-inf-b-1a8107-openshift-default-7fff599fbf-rtjrx" Oct 29 23:22:20.018488 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:20.018465 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a1427f2-b35c-4355-9997-cde832700d7f-credential-socket\") pod \"e2e-inf-b-1a8107-openshift-default-7fff599fbf-rtjrx\" (UID: \"1a1427f2-b35c-4355-9997-cde832700d7f\") " pod="openshift-ingress/e2e-inf-b-1a8107-openshift-default-7fff599fbf-rtjrx" Oct 29 23:22:20.018797 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:20.018773 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a1427f2-b35c-4355-9997-cde832700d7f-istio-data\") pod \"e2e-inf-b-1a8107-openshift-default-7fff599fbf-rtjrx\" (UID: \"1a1427f2-b35c-4355-9997-cde832700d7f\") " pod="openshift-ingress/e2e-inf-b-1a8107-openshift-default-7fff599fbf-rtjrx" Oct 29 23:22:20.120844 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:20.120575 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a1427f2-b35c-4355-9997-cde832700d7f-istio-data\") pod \"e2e-inf-b-1a8107-openshift-default-7fff599fbf-rtjrx\" (UID: \"1a1427f2-b35c-4355-9997-cde832700d7f\") " pod="openshift-ingress/e2e-inf-b-1a8107-openshift-default-7fff599fbf-rtjrx" Oct 29 23:22:20.120844 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:20.120674 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a1427f2-b35c-4355-9997-cde832700d7f-istio-podinfo\") pod \"e2e-inf-b-1a8107-openshift-default-7fff599fbf-rtjrx\" (UID: \"1a1427f2-b35c-4355-9997-cde832700d7f\") " pod="openshift-ingress/e2e-inf-b-1a8107-openshift-default-7fff599fbf-rtjrx" Oct 29 23:22:20.120844 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:20.120734 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a1427f2-b35c-4355-9997-cde832700d7f-istio-envoy\") pod \"e2e-inf-b-1a8107-openshift-default-7fff599fbf-rtjrx\" (UID: \"1a1427f2-b35c-4355-9997-cde832700d7f\") " pod="openshift-ingress/e2e-inf-b-1a8107-openshift-default-7fff599fbf-rtjrx" Oct 29 23:22:20.120844 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:20.120821 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a1427f2-b35c-4355-9997-cde832700d7f-istiod-ca-cert\") pod \"e2e-inf-b-1a8107-openshift-default-7fff599fbf-rtjrx\" (UID: \"1a1427f2-b35c-4355-9997-cde832700d7f\") " pod="openshift-ingress/e2e-inf-b-1a8107-openshift-default-7fff599fbf-rtjrx" Oct 29 23:22:20.121252 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:20.120870 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a1427f2-b35c-4355-9997-cde832700d7f-workload-socket\") pod \"e2e-inf-b-1a8107-openshift-default-7fff599fbf-rtjrx\" (UID: \"1a1427f2-b35c-4355-9997-cde832700d7f\") " pod="openshift-ingress/e2e-inf-b-1a8107-openshift-default-7fff599fbf-rtjrx" Oct 29 23:22:20.121252 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:20.120934 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a1427f2-b35c-4355-9997-cde832700d7f-istio-token\") pod \"e2e-inf-b-1a8107-openshift-default-7fff599fbf-rtjrx\" (UID: \"1a1427f2-b35c-4355-9997-cde832700d7f\") " pod="openshift-ingress/e2e-inf-b-1a8107-openshift-default-7fff599fbf-rtjrx" Oct 29 23:22:20.121252 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:20.121058 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a1427f2-b35c-4355-9997-cde832700d7f-workload-certs\") pod \"e2e-inf-b-1a8107-openshift-default-7fff599fbf-rtjrx\" (UID: \"1a1427f2-b35c-4355-9997-cde832700d7f\") " pod="openshift-ingress/e2e-inf-b-1a8107-openshift-default-7fff599fbf-rtjrx" Oct 29 23:22:20.121252 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:20.121102 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9nz8t\" (UniqueName: \"kubernetes.io/projected/1a1427f2-b35c-4355-9997-cde832700d7f-kube-api-access-9nz8t\") pod \"e2e-inf-b-1a8107-openshift-default-7fff599fbf-rtjrx\" (UID: \"1a1427f2-b35c-4355-9997-cde832700d7f\") " pod="openshift-ingress/e2e-inf-b-1a8107-openshift-default-7fff599fbf-rtjrx" Oct 29 23:22:20.121252 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:20.121181 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a1427f2-b35c-4355-9997-cde832700d7f-credential-socket\") pod \"e2e-inf-b-1a8107-openshift-default-7fff599fbf-rtjrx\" (UID: \"1a1427f2-b35c-4355-9997-cde832700d7f\") " pod="openshift-ingress/e2e-inf-b-1a8107-openshift-default-7fff599fbf-rtjrx" Oct 29 23:22:20.121559 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:20.121525 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a1427f2-b35c-4355-9997-cde832700d7f-istio-data\") pod \"e2e-inf-b-1a8107-openshift-default-7fff599fbf-rtjrx\" (UID: \"1a1427f2-b35c-4355-9997-cde832700d7f\") " pod="openshift-ingress/e2e-inf-b-1a8107-openshift-default-7fff599fbf-rtjrx" Oct 29 23:22:20.121755 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:20.121728 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a1427f2-b35c-4355-9997-cde832700d7f-credential-socket\") pod \"e2e-inf-b-1a8107-openshift-default-7fff599fbf-rtjrx\" (UID: \"1a1427f2-b35c-4355-9997-cde832700d7f\") " pod="openshift-ingress/e2e-inf-b-1a8107-openshift-default-7fff599fbf-rtjrx" Oct 29 23:22:20.121854 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:20.121823 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a1427f2-b35c-4355-9997-cde832700d7f-workload-socket\") pod \"e2e-inf-b-1a8107-openshift-default-7fff599fbf-rtjrx\" (UID: \"1a1427f2-b35c-4355-9997-cde832700d7f\") " pod="openshift-ingress/e2e-inf-b-1a8107-openshift-default-7fff599fbf-rtjrx" Oct 29 23:22:20.122474 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:20.122451 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a1427f2-b35c-4355-9997-cde832700d7f-workload-certs\") pod \"e2e-inf-b-1a8107-openshift-default-7fff599fbf-rtjrx\" (UID: \"1a1427f2-b35c-4355-9997-cde832700d7f\") " pod="openshift-ingress/e2e-inf-b-1a8107-openshift-default-7fff599fbf-rtjrx" Oct 29 23:22:20.122632 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:20.122610 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a1427f2-b35c-4355-9997-cde832700d7f-istiod-ca-cert\") pod \"e2e-inf-b-1a8107-openshift-default-7fff599fbf-rtjrx\" (UID: \"1a1427f2-b35c-4355-9997-cde832700d7f\") " pod="openshift-ingress/e2e-inf-b-1a8107-openshift-default-7fff599fbf-rtjrx" Oct 29 23:22:20.125017 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:20.124993 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a1427f2-b35c-4355-9997-cde832700d7f-istio-podinfo\") pod \"e2e-inf-b-1a8107-openshift-default-7fff599fbf-rtjrx\" (UID: \"1a1427f2-b35c-4355-9997-cde832700d7f\") " pod="openshift-ingress/e2e-inf-b-1a8107-openshift-default-7fff599fbf-rtjrx" Oct 29 23:22:20.125494 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:20.125475 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a1427f2-b35c-4355-9997-cde832700d7f-istio-envoy\") pod \"e2e-inf-b-1a8107-openshift-default-7fff599fbf-rtjrx\" (UID: \"1a1427f2-b35c-4355-9997-cde832700d7f\") " pod="openshift-ingress/e2e-inf-b-1a8107-openshift-default-7fff599fbf-rtjrx" Oct 29 23:22:20.161394 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:20.161362 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9nz8t\" (UniqueName: \"kubernetes.io/projected/1a1427f2-b35c-4355-9997-cde832700d7f-kube-api-access-9nz8t\") pod \"e2e-inf-b-1a8107-openshift-default-7fff599fbf-rtjrx\" (UID: \"1a1427f2-b35c-4355-9997-cde832700d7f\") " pod="openshift-ingress/e2e-inf-b-1a8107-openshift-default-7fff599fbf-rtjrx" Oct 29 23:22:20.163459 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:20.163436 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a1427f2-b35c-4355-9997-cde832700d7f-istio-token\") pod \"e2e-inf-b-1a8107-openshift-default-7fff599fbf-rtjrx\" (UID: \"1a1427f2-b35c-4355-9997-cde832700d7f\") " pod="openshift-ingress/e2e-inf-b-1a8107-openshift-default-7fff599fbf-rtjrx" Oct 29 23:22:20.459447 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:20.459413 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-1a8107-openshift-default-7fff599fbf-rtjrx" Oct 29 23:22:20.534215 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:20.534182 5057 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-vm822 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.229:15021/healthz/ready\": dial tcp 10.132.0.229:15021: connect: connection refused" start-of-body= Oct 29 23:22:20.534379 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:20.534269 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vm822" podUID="ef4e2b67-0954-4fe1-a2b4-223771c24e79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.229:15021/healthz/ready\": dial tcp 10.132.0.229:15021: connect: connection refused" Oct 29 23:22:20.828571 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:20.828542 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-1a8107-openshift-default-7fff599fbf-rtjrx"] Oct 29 23:22:20.832631 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:20.832598 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 23:22:20.832741 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:20.832664 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 23:22:20.832741 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:20.832695 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 23:22:21.345744 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:21.345707 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-1a8107-openshift-default-7fff599fbf-rtjrx" event={"ID":"1a1427f2-b35c-4355-9997-cde832700d7f","Type":"ContainerStarted","Data":"21b23ab4ce637a39eb4a94f9a00935ce39aaeca6fc3c2da9178ee83903e3ebd2"} Oct 29 23:22:21.346247 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:21.345754 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-1a8107-openshift-default-7fff599fbf-rtjrx" event={"ID":"1a1427f2-b35c-4355-9997-cde832700d7f","Type":"ContainerStarted","Data":"af491a46d2a7721a8a51903b8fc682e4358302dcbc22e1c8948b63d7d2c6d7c0"} Oct 29 23:22:21.348397 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:21.348365 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:22:21.446354 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:21.439672 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-1a8107-openshift-default-7fff599fbf-rtjrx" podStartSLOduration=2.439630449 podStartE2EDuration="2.439630449s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:22:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:22:21.393554086 +0000 UTC m=+2922.176767536" watchObservedRunningTime="2026-10-29 23:22:21.439630449 +0000 UTC m=+2922.222843876" Oct 29 23:22:21.467378 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:21.465505 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-1a8107-openshift-default-7fff599fbf-rtjrx" Oct 29 23:22:21.467378 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:21.465621 5057 patch_prober.go:29] interesting pod/e2e-inf-b-1a8107-openshift-default-7fff599fbf-rtjrx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused" start-of-body= Oct 29 23:22:21.467378 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:21.465663 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-1a8107-openshift-default-7fff599fbf-rtjrx" podUID="1a1427f2-b35c-4355-9997-cde832700d7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused" Oct 29 23:22:21.541350 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:21.536989 5057 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-vm822 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.229:15021/healthz/ready\": dial tcp 10.132.0.229:15021: connect: connection refused" start-of-body= Oct 29 23:22:21.541350 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:21.537047 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vm822" podUID="ef4e2b67-0954-4fe1-a2b4-223771c24e79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.229:15021/healthz/ready\": dial tcp 10.132.0.229:15021: connect: connection refused" Oct 29 23:22:22.460465 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:22.460432 5057 patch_prober.go:29] interesting pod/e2e-inf-b-1a8107-openshift-default-7fff599fbf-rtjrx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused" start-of-body= Oct 29 23:22:22.460900 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:22.460876 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-1a8107-openshift-default-7fff599fbf-rtjrx" podUID="1a1427f2-b35c-4355-9997-cde832700d7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused" Oct 29 23:22:22.536498 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:22.536417 5057 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-vm822 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.229:15021/healthz/ready\": dial tcp 10.132.0.229:15021: connect: connection refused" start-of-body= Oct 29 23:22:22.536498 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:22.536478 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vm822" podUID="ef4e2b67-0954-4fe1-a2b4-223771c24e79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.229:15021/healthz/ready\": dial tcp 10.132.0.229:15021: connect: connection refused" Oct 29 23:22:23.396355 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:23.376317 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:22:23.465370 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:23.459736 5057 patch_prober.go:29] interesting pod/e2e-inf-b-1a8107-openshift-default-7fff599fbf-rtjrx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused" start-of-body= Oct 29 23:22:23.465370 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:23.459805 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-1a8107-openshift-default-7fff599fbf-rtjrx" podUID="1a1427f2-b35c-4355-9997-cde832700d7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused" Oct 29 23:22:23.553355 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:23.541702 5057 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-vm822 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.229:15021/healthz/ready\": dial tcp 10.132.0.229:15021: connect: connection refused" start-of-body= Oct 29 23:22:23.553355 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:23.541775 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vm822" podUID="ef4e2b67-0954-4fe1-a2b4-223771c24e79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.229:15021/healthz/ready\": dial tcp 10.132.0.229:15021: connect: connection refused" Oct 29 23:22:24.460770 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:24.460735 5057 patch_prober.go:29] interesting pod/e2e-inf-b-1a8107-openshift-default-7fff599fbf-rtjrx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused" start-of-body= Oct 29 23:22:24.460960 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:24.460788 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-1a8107-openshift-default-7fff599fbf-rtjrx" podUID="1a1427f2-b35c-4355-9997-cde832700d7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused" Oct 29 23:22:24.534294 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:24.534241 5057 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-vm822 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.229:15021/healthz/ready\": dial tcp 10.132.0.229:15021: connect: connection refused" start-of-body= Oct 29 23:22:24.534810 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:24.534311 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vm822" podUID="ef4e2b67-0954-4fe1-a2b4-223771c24e79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.229:15021/healthz/ready\": dial tcp 10.132.0.229:15021: connect: connection refused" Oct 29 23:22:25.116678 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:25.116641 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:22:25.459883 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:25.459855 5057 patch_prober.go:29] interesting pod/e2e-inf-b-1a8107-openshift-default-7fff599fbf-rtjrx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused" start-of-body= Oct 29 23:22:25.460030 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:25.459904 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-1a8107-openshift-default-7fff599fbf-rtjrx" podUID="1a1427f2-b35c-4355-9997-cde832700d7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused" Oct 29 23:22:25.534126 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:25.534090 5057 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-vm822 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.229:15021/healthz/ready\": dial tcp 10.132.0.229:15021: connect: connection refused" start-of-body= Oct 29 23:22:25.534342 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:25.534156 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vm822" podUID="ef4e2b67-0954-4fe1-a2b4-223771c24e79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.229:15021/healthz/ready\": dial tcp 10.132.0.229:15021: connect: connection refused" Oct 29 23:22:26.462922 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:26.462881 5057 patch_prober.go:29] interesting pod/e2e-inf-b-1a8107-openshift-default-7fff599fbf-rtjrx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused" start-of-body= Oct 29 23:22:26.463223 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:26.463188 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-1a8107-openshift-default-7fff599fbf-rtjrx" podUID="1a1427f2-b35c-4355-9997-cde832700d7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused" Oct 29 23:22:26.534075 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:26.534034 5057 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-vm822 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.229:15021/healthz/ready\": dial tcp 10.132.0.229:15021: connect: connection refused" start-of-body= Oct 29 23:22:26.534303 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:26.534281 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vm822" podUID="ef4e2b67-0954-4fe1-a2b4-223771c24e79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.229:15021/healthz/ready\": dial tcp 10.132.0.229:15021: connect: connection refused" Oct 29 23:22:26.718032 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:26.717925 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:22:27.460318 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:27.460282 5057 patch_prober.go:29] interesting pod/e2e-inf-b-1a8107-openshift-default-7fff599fbf-rtjrx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused" start-of-body= Oct 29 23:22:27.460547 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:27.460368 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-1a8107-openshift-default-7fff599fbf-rtjrx" podUID="1a1427f2-b35c-4355-9997-cde832700d7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused" Oct 29 23:22:27.919322 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:27.919235 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vm822" Oct 29 23:22:27.919828 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:27.919579 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vm822" Oct 29 23:22:27.920501 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:27.920469 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vm822" Oct 29 23:22:28.465520 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:28.465492 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-1a8107-openshift-default-7fff599fbf-rtjrx" Oct 29 23:22:28.465912 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:28.465887 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-1a8107-openshift-default-7fff599fbf-rtjrx" Oct 29 23:22:28.466656 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:28.466633 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-1a8107-openshift-default-7fff599fbf-rtjrx" Oct 29 23:22:28.514309 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:28.514245 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:22:30.077163 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:30.077129 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:22:31.613672 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:31.613615 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:22:33.384475 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:33.384437 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:22:35.147113 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:35.147072 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:22:35.484595 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:35.484561 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:22:37.004146 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:37.004109 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:22:38.909005 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:38.908966 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:22:40.652530 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:40.652490 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:22:42.335640 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:42.335603 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:22:42.487603 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:42.487567 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:22:43.495701 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:43.495669 5057 scope.go:122] "RemoveContainer" containerID="154ad77663a91b2fb80ce9d97ee4ab0e1a40096d46078821d41a53aebff654b7" Oct 29 23:22:44.392287 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:44.392250 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:22:44.566355 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:44.563957 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:22:46.168848 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:46.168813 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:22:47.388934 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:47.388898 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:22:48.126129 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:48.126087 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:22:49.030164 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:22:49.030124 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d2dnn" containerName="extractor" Oct 29 23:22:49.818168 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:49.818130 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:22:51.366847 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:51.366811 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:22:53.108211 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:53.108168 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:22:54.699080 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:54.699023 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:22:56.324412 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:56.324377 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:22:56.507102 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:56.507059 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v2tfm"] Oct 29 23:22:57.830436 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:57.830386 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-hchzr"] Oct 29 23:22:57.830928 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:57.830652 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-hchzr" podUID="c0fe6ba5-aa29-48d1-955d-e8e063d8e77f" containerName="istio-proxy" containerID="cri-o://448ae9294e937a80008216ac8aea75ebed0a6da9d3caa4292cdc432ee1bf3060" gracePeriod=30 Oct 29 23:22:58.968713 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:22:58.968675 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:23:01.936356 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:01.935299 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:23:03.015522 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:03.015482 5057 generic.go:356] "Generic (PLEG): container finished" podID="c0fe6ba5-aa29-48d1-955d-e8e063d8e77f" containerID="448ae9294e937a80008216ac8aea75ebed0a6da9d3caa4292cdc432ee1bf3060" exitCode=0 Oct 29 23:23:03.016038 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:03.015522 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-hchzr" event={"ID":"c0fe6ba5-aa29-48d1-955d-e8e063d8e77f","Type":"ContainerDied","Data":"448ae9294e937a80008216ac8aea75ebed0a6da9d3caa4292cdc432ee1bf3060"} Oct 29 23:23:03.160545 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:03.160520 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-hchzr" Oct 29 23:23:03.206724 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:03.206692 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0fe6ba5-aa29-48d1-955d-e8e063d8e77f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0fe6ba5-aa29-48d1-955d-e8e063d8e77f-istio-envoy\") pod \"c0fe6ba5-aa29-48d1-955d-e8e063d8e77f\" (UID: \"c0fe6ba5-aa29-48d1-955d-e8e063d8e77f\") " Oct 29 23:23:03.206922 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:03.206899 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0fe6ba5-aa29-48d1-955d-e8e063d8e77f-istio-token\" (UniqueName: \"kubernetes.io/projected/c0fe6ba5-aa29-48d1-955d-e8e063d8e77f-istio-token\") pod \"c0fe6ba5-aa29-48d1-955d-e8e063d8e77f\" (UID: \"c0fe6ba5-aa29-48d1-955d-e8e063d8e77f\") " Oct 29 23:23:03.207045 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:03.206986 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0fe6ba5-aa29-48d1-955d-e8e063d8e77f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0fe6ba5-aa29-48d1-955d-e8e063d8e77f-workload-socket\") pod \"c0fe6ba5-aa29-48d1-955d-e8e063d8e77f\" (UID: \"c0fe6ba5-aa29-48d1-955d-e8e063d8e77f\") " Oct 29 23:23:03.207124 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:03.207061 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0fe6ba5-aa29-48d1-955d-e8e063d8e77f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0fe6ba5-aa29-48d1-955d-e8e063d8e77f-credential-socket\") pod \"c0fe6ba5-aa29-48d1-955d-e8e063d8e77f\" (UID: \"c0fe6ba5-aa29-48d1-955d-e8e063d8e77f\") " Oct 29 23:23:03.207124 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:03.207088 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0fe6ba5-aa29-48d1-955d-e8e063d8e77f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0fe6ba5-aa29-48d1-955d-e8e063d8e77f-workload-certs\") pod \"c0fe6ba5-aa29-48d1-955d-e8e063d8e77f\" (UID: \"c0fe6ba5-aa29-48d1-955d-e8e063d8e77f\") " Oct 29 23:23:03.207668 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:03.207344 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c0fe6ba5-aa29-48d1-955d-e8e063d8e77f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0fe6ba5-aa29-48d1-955d-e8e063d8e77f-istio-podinfo\") pod \"c0fe6ba5-aa29-48d1-955d-e8e063d8e77f\" (UID: \"c0fe6ba5-aa29-48d1-955d-e8e063d8e77f\") " Oct 29 23:23:03.207668 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:03.207367 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0fe6ba5-aa29-48d1-955d-e8e063d8e77f-credential-socket" pod "c0fe6ba5-aa29-48d1-955d-e8e063d8e77f" (UID: "c0fe6ba5-aa29-48d1-955d-e8e063d8e77f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:23:03.207668 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:03.207324 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0fe6ba5-aa29-48d1-955d-e8e063d8e77f-workload-socket" pod "c0fe6ba5-aa29-48d1-955d-e8e063d8e77f" (UID: "c0fe6ba5-aa29-48d1-955d-e8e063d8e77f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:23:03.207668 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:03.207397 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0fe6ba5-aa29-48d1-955d-e8e063d8e77f-workload-certs" pod "c0fe6ba5-aa29-48d1-955d-e8e063d8e77f" (UID: "c0fe6ba5-aa29-48d1-955d-e8e063d8e77f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:23:03.207668 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:03.207435 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0fe6ba5-aa29-48d1-955d-e8e063d8e77f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0fe6ba5-aa29-48d1-955d-e8e063d8e77f-istiod-ca-cert\") pod \"c0fe6ba5-aa29-48d1-955d-e8e063d8e77f\" (UID: \"c0fe6ba5-aa29-48d1-955d-e8e063d8e77f\") " Oct 29 23:23:03.207668 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:03.207506 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0fe6ba5-aa29-48d1-955d-e8e063d8e77f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0fe6ba5-aa29-48d1-955d-e8e063d8e77f-istio-data\") pod \"c0fe6ba5-aa29-48d1-955d-e8e063d8e77f\" (UID: \"c0fe6ba5-aa29-48d1-955d-e8e063d8e77f\") " Oct 29 23:23:03.207668 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:03.207544 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0fe6ba5-aa29-48d1-955d-e8e063d8e77f-kube-api-access-x56mg\" (UniqueName: \"kubernetes.io/projected/c0fe6ba5-aa29-48d1-955d-e8e063d8e77f-kube-api-access-x56mg\") pod \"c0fe6ba5-aa29-48d1-955d-e8e063d8e77f\" (UID: \"c0fe6ba5-aa29-48d1-955d-e8e063d8e77f\") " Oct 29 23:23:03.208157 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:03.207740 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0fe6ba5-aa29-48d1-955d-e8e063d8e77f-istio-data" pod "c0fe6ba5-aa29-48d1-955d-e8e063d8e77f" (UID: "c0fe6ba5-aa29-48d1-955d-e8e063d8e77f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:23:03.208157 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:03.207758 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0fe6ba5-aa29-48d1-955d-e8e063d8e77f-istiod-ca-cert" pod "c0fe6ba5-aa29-48d1-955d-e8e063d8e77f" (UID: "c0fe6ba5-aa29-48d1-955d-e8e063d8e77f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:23:03.208605 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:03.208581 5057 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0fe6ba5-aa29-48d1-955d-e8e063d8e77f-workload-socket\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:23:03.208731 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:03.208608 5057 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0fe6ba5-aa29-48d1-955d-e8e063d8e77f-credential-socket\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:23:03.208731 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:03.208623 5057 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0fe6ba5-aa29-48d1-955d-e8e063d8e77f-workload-certs\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:23:03.208731 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:03.208637 5057 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0fe6ba5-aa29-48d1-955d-e8e063d8e77f-istiod-ca-cert\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:23:03.208731 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:03.208650 5057 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0fe6ba5-aa29-48d1-955d-e8e063d8e77f-istio-data\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:23:03.209640 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:03.209611 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0fe6ba5-aa29-48d1-955d-e8e063d8e77f-istio-token" pod "c0fe6ba5-aa29-48d1-955d-e8e063d8e77f" (UID: "c0fe6ba5-aa29-48d1-955d-e8e063d8e77f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:23:03.209981 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:03.209957 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0fe6ba5-aa29-48d1-955d-e8e063d8e77f-istio-envoy" pod "c0fe6ba5-aa29-48d1-955d-e8e063d8e77f" (UID: "c0fe6ba5-aa29-48d1-955d-e8e063d8e77f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:23:03.210095 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:03.210028 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0fe6ba5-aa29-48d1-955d-e8e063d8e77f-kube-api-access-x56mg" pod "c0fe6ba5-aa29-48d1-955d-e8e063d8e77f" (UID: "c0fe6ba5-aa29-48d1-955d-e8e063d8e77f"). InnerVolumeSpecName "kube-api-access-x56mg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:23:03.210227 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:03.210204 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c0fe6ba5-aa29-48d1-955d-e8e063d8e77f-istio-podinfo" pod "c0fe6ba5-aa29-48d1-955d-e8e063d8e77f" (UID: "c0fe6ba5-aa29-48d1-955d-e8e063d8e77f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:23:03.310662 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:03.310626 5057 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0fe6ba5-aa29-48d1-955d-e8e063d8e77f-istio-token\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:23:03.310952 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:03.310921 5057 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0fe6ba5-aa29-48d1-955d-e8e063d8e77f-istio-podinfo\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:23:03.310952 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:03.310950 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x56mg\" (UniqueName: \"kubernetes.io/projected/c0fe6ba5-aa29-48d1-955d-e8e063d8e77f-kube-api-access-x56mg\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:23:03.311123 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:03.310967 5057 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0fe6ba5-aa29-48d1-955d-e8e063d8e77f-istio-envoy\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:23:04.027539 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:04.027506 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-hchzr" Oct 29 23:23:04.027539 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:04.027528 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-hchzr" event={"ID":"c0fe6ba5-aa29-48d1-955d-e8e063d8e77f","Type":"ContainerDied","Data":"8695b21af758fc7ea39c8bc5f1efe2e54ababa2d0791867b02422b6d15775888"} Oct 29 23:23:04.028063 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:04.027607 5057 scope.go:122] "RemoveContainer" containerID="448ae9294e937a80008216ac8aea75ebed0a6da9d3caa4292cdc432ee1bf3060" Oct 29 23:23:04.102072 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:04.102039 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-hchzr"] Oct 29 23:23:04.110792 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:04.110763 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-hchzr"] Oct 29 23:23:04.187214 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:04.187182 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:23:05.332184 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:05.332152 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0fe6ba5-aa29-48d1-955d-e8e063d8e77f" path="/var/lib/kubelet/pods/c0fe6ba5-aa29-48d1-955d-e8e063d8e77f/volumes" Oct 29 23:23:05.825781 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:05.825740 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:23:07.393218 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:07.393136 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:23:07.569812 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:07.569776 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:23:07.681724 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:07.681648 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:23:07.803242 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:07.803191 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:23:08.429423 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:08.429388 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:23:09.134326 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:09.134284 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:23:10.125776 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:10.125736 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d836f867-m6fqr"] Oct 29 23:23:10.126877 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:10.126853 5057 cpu_manager.go:414] "containerMap: removing container" podUID="c0fe6ba5-aa29-48d1-955d-e8e063d8e77f" containerName="istio-proxy" Oct 29 23:23:10.126877 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:10.126874 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0fe6ba5-aa29-48d1-955d-e8e063d8e77f" containerName="istio-proxy" Oct 29 23:23:10.127069 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:10.127043 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0fe6ba5-aa29-48d1-955d-e8e063d8e77f" containerName="istio-proxy" Oct 29 23:23:10.131286 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:10.131266 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d836f867-m6fqr" Oct 29 23:23:10.150832 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:10.150805 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d836f867-m6fqr"] Oct 29 23:23:10.154572 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:10.154547 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:23:10.154673 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:10.154584 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dmx57\"" Oct 29 23:23:10.232387 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:10.232310 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zzbjd\" (UniqueName: \"kubernetes.io/projected/6c1683db-89f4-4cff-ae7a-5170fc9645e4-kube-api-access-zzbjd\") pod \"maas-api-revoke-sub-d836f867-m6fqr\" (UID: \"6c1683db-89f4-4cff-ae7a-5170fc9645e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d836f867-m6fqr" Oct 29 23:23:10.232612 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:10.232513 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c1683db-89f4-4cff-ae7a-5170fc9645e4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d836f867-m6fqr\" (UID: \"6c1683db-89f4-4cff-ae7a-5170fc9645e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d836f867-m6fqr" Oct 29 23:23:10.232672 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:10.232629 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c1683db-89f4-4cff-ae7a-5170fc9645e4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d836f867-m6fqr\" (UID: \"6c1683db-89f4-4cff-ae7a-5170fc9645e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d836f867-m6fqr" Oct 29 23:23:10.333978 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:10.333942 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c1683db-89f4-4cff-ae7a-5170fc9645e4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d836f867-m6fqr\" (UID: \"6c1683db-89f4-4cff-ae7a-5170fc9645e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d836f867-m6fqr" Oct 29 23:23:10.334182 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:10.334074 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c1683db-89f4-4cff-ae7a-5170fc9645e4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d836f867-m6fqr\" (UID: \"6c1683db-89f4-4cff-ae7a-5170fc9645e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d836f867-m6fqr" Oct 29 23:23:10.334245 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:10.334230 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzbjd\" (UniqueName: \"kubernetes.io/projected/6c1683db-89f4-4cff-ae7a-5170fc9645e4-kube-api-access-zzbjd\") pod \"maas-api-revoke-sub-d836f867-m6fqr\" (UID: \"6c1683db-89f4-4cff-ae7a-5170fc9645e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d836f867-m6fqr" Oct 29 23:23:10.334770 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:10.334746 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c1683db-89f4-4cff-ae7a-5170fc9645e4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d836f867-m6fqr\" (UID: \"6c1683db-89f4-4cff-ae7a-5170fc9645e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d836f867-m6fqr" Oct 29 23:23:10.336688 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:10.336658 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c1683db-89f4-4cff-ae7a-5170fc9645e4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d836f867-m6fqr\" (UID: \"6c1683db-89f4-4cff-ae7a-5170fc9645e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d836f867-m6fqr" Oct 29 23:23:10.356982 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:10.356954 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzbjd\" (UniqueName: \"kubernetes.io/projected/6c1683db-89f4-4cff-ae7a-5170fc9645e4-kube-api-access-zzbjd\") pod \"maas-api-revoke-sub-d836f867-m6fqr\" (UID: \"6c1683db-89f4-4cff-ae7a-5170fc9645e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d836f867-m6fqr" Oct 29 23:23:10.447991 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:10.447966 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d836f867-m6fqr" Oct 29 23:23:10.708146 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:10.708050 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:23:10.775562 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:10.772027 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d836f867-m6fqr"] Oct 29 23:23:11.106892 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:11.106856 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d836f867-m6fqr" event={"ID":"6c1683db-89f4-4cff-ae7a-5170fc9645e4","Type":"ContainerStarted","Data":"c8a86105272ee1c8bd7f8551d18826d5c1fb4c85d929ecbc7a4ee9e5202d7687"} Oct 29 23:23:12.119196 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:12.119160 5057 generic.go:356] "Generic (PLEG): container finished" podID="6c1683db-89f4-4cff-ae7a-5170fc9645e4" containerID="15df27fea49b6e79c662c9f1b6a1cf7db5244546ea096e7c0602723233d0c503" exitCode=0 Oct 29 23:23:12.119684 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:12.119247 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d836f867-m6fqr" event={"ID":"6c1683db-89f4-4cff-ae7a-5170fc9645e4","Type":"ContainerDied","Data":"15df27fea49b6e79c662c9f1b6a1cf7db5244546ea096e7c0602723233d0c503"} Oct 29 23:23:12.277176 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:12.277081 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:23:13.330677 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:13.330639 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d836f867-m6fqr" Oct 29 23:23:13.495064 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:13.495036 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c1683db-89f4-4cff-ae7a-5170fc9645e4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c1683db-89f4-4cff-ae7a-5170fc9645e4-maas-api-openshift-ca\") pod \"6c1683db-89f4-4cff-ae7a-5170fc9645e4\" (UID: \"6c1683db-89f4-4cff-ae7a-5170fc9645e4\") " Oct 29 23:23:13.495064 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:13.495069 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6c1683db-89f4-4cff-ae7a-5170fc9645e4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c1683db-89f4-4cff-ae7a-5170fc9645e4-maas-api-xks-ca\") pod \"6c1683db-89f4-4cff-ae7a-5170fc9645e4\" (UID: \"6c1683db-89f4-4cff-ae7a-5170fc9645e4\") " Oct 29 23:23:13.495326 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:13.495279 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c1683db-89f4-4cff-ae7a-5170fc9645e4-kube-api-access-zzbjd\" (UniqueName: \"kubernetes.io/projected/6c1683db-89f4-4cff-ae7a-5170fc9645e4-kube-api-access-zzbjd\") pod \"6c1683db-89f4-4cff-ae7a-5170fc9645e4\" (UID: \"6c1683db-89f4-4cff-ae7a-5170fc9645e4\") " Oct 29 23:23:13.495481 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:13.495451 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c1683db-89f4-4cff-ae7a-5170fc9645e4-maas-api-openshift-ca" pod "6c1683db-89f4-4cff-ae7a-5170fc9645e4" (UID: "6c1683db-89f4-4cff-ae7a-5170fc9645e4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:23:13.496008 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:13.495982 5057 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c1683db-89f4-4cff-ae7a-5170fc9645e4-maas-api-openshift-ca\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:23:13.497923 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:13.497898 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c1683db-89f4-4cff-ae7a-5170fc9645e4-maas-api-xks-ca" pod "6c1683db-89f4-4cff-ae7a-5170fc9645e4" (UID: "6c1683db-89f4-4cff-ae7a-5170fc9645e4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:23:13.498005 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:13.497975 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c1683db-89f4-4cff-ae7a-5170fc9645e4-kube-api-access-zzbjd" pod "6c1683db-89f4-4cff-ae7a-5170fc9645e4" (UID: "6c1683db-89f4-4cff-ae7a-5170fc9645e4"). InnerVolumeSpecName "kube-api-access-zzbjd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:23:13.597490 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:13.597460 5057 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c1683db-89f4-4cff-ae7a-5170fc9645e4-maas-api-xks-ca\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:23:13.597620 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:13.597493 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zzbjd\" (UniqueName: \"kubernetes.io/projected/6c1683db-89f4-4cff-ae7a-5170fc9645e4-kube-api-access-zzbjd\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:23:14.000676 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:14.000639 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:23:14.185860 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:14.185827 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d836f867-m6fqr" event={"ID":"6c1683db-89f4-4cff-ae7a-5170fc9645e4","Type":"ContainerDied","Data":"c8a86105272ee1c8bd7f8551d18826d5c1fb4c85d929ecbc7a4ee9e5202d7687"} Oct 29 23:23:14.186047 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:14.185878 5057 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c8a86105272ee1c8bd7f8551d18826d5c1fb4c85d929ecbc7a4ee9e5202d7687" Oct 29 23:23:14.186047 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:14.185936 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d836f867-m6fqr" Oct 29 23:23:15.902401 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:15.902362 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:23:17.515839 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:17.515798 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:23:19.030246 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:23:19.030203 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d2dnn" containerName="extractor" Oct 29 23:23:19.239581 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:19.239537 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:23:20.711615 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:20.711571 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:23:21.055528 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:21.055373 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:23:21.605382 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:21.605321 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ffd0336a-kz9q8"] Oct 29 23:23:21.606487 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:21.606465 5057 cpu_manager.go:414] "containerMap: removing container" podUID="6c1683db-89f4-4cff-ae7a-5170fc9645e4" containerName="revoke-keys" Oct 29 23:23:21.606643 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:21.606500 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c1683db-89f4-4cff-ae7a-5170fc9645e4" containerName="revoke-keys" Oct 29 23:23:21.606643 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:21.606636 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c1683db-89f4-4cff-ae7a-5170fc9645e4" containerName="revoke-keys" Oct 29 23:23:21.612541 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:21.612513 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ffd0336a-kz9q8" Oct 29 23:23:21.628302 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:21.627822 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:23:21.628302 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:21.628069 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ffd0336a-kz9q8"] Oct 29 23:23:21.630956 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:21.630929 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dmx57\"" Oct 29 23:23:21.757951 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:21.757904 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b944d\" (UniqueName: \"kubernetes.io/projected/9570ca18-a7cc-4041-b505-e11161579302-kube-api-access-b944d\") pod \"maas-api-revoke-sub-ffd0336a-kz9q8\" (UID: \"9570ca18-a7cc-4041-b505-e11161579302\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ffd0336a-kz9q8" Oct 29 23:23:21.758405 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:21.758066 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9570ca18-a7cc-4041-b505-e11161579302-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ffd0336a-kz9q8\" (UID: \"9570ca18-a7cc-4041-b505-e11161579302\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ffd0336a-kz9q8" Oct 29 23:23:21.758510 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:21.758487 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9570ca18-a7cc-4041-b505-e11161579302-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ffd0336a-kz9q8\" (UID: \"9570ca18-a7cc-4041-b505-e11161579302\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ffd0336a-kz9q8" Oct 29 23:23:21.860428 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:21.860323 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9570ca18-a7cc-4041-b505-e11161579302-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ffd0336a-kz9q8\" (UID: \"9570ca18-a7cc-4041-b505-e11161579302\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ffd0336a-kz9q8" Oct 29 23:23:21.860606 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:21.860435 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b944d\" (UniqueName: \"kubernetes.io/projected/9570ca18-a7cc-4041-b505-e11161579302-kube-api-access-b944d\") pod \"maas-api-revoke-sub-ffd0336a-kz9q8\" (UID: \"9570ca18-a7cc-4041-b505-e11161579302\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ffd0336a-kz9q8" Oct 29 23:23:21.860606 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:21.860484 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9570ca18-a7cc-4041-b505-e11161579302-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ffd0336a-kz9q8\" (UID: \"9570ca18-a7cc-4041-b505-e11161579302\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ffd0336a-kz9q8" Oct 29 23:23:21.861143 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:21.861120 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9570ca18-a7cc-4041-b505-e11161579302-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ffd0336a-kz9q8\" (UID: \"9570ca18-a7cc-4041-b505-e11161579302\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ffd0336a-kz9q8" Oct 29 23:23:21.862697 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:21.862673 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9570ca18-a7cc-4041-b505-e11161579302-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ffd0336a-kz9q8\" (UID: \"9570ca18-a7cc-4041-b505-e11161579302\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ffd0336a-kz9q8" Oct 29 23:23:21.883360 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:21.883302 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b944d\" (UniqueName: \"kubernetes.io/projected/9570ca18-a7cc-4041-b505-e11161579302-kube-api-access-b944d\") pod \"maas-api-revoke-sub-ffd0336a-kz9q8\" (UID: \"9570ca18-a7cc-4041-b505-e11161579302\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ffd0336a-kz9q8" Oct 29 23:23:21.936431 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:21.936405 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ffd0336a-kz9q8" Oct 29 23:23:22.256026 ip-10-0-140-74 kubenswrapper[5057]: W1029 23:23:22.255998 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9570ca18_a7cc_4041_b505_e11161579302.slice/crio-a27d3ccdcfdc5dc41993d8cbac00f76406f5f51cfaa62966ebf1bafcaef083d0 WatchSource:0}: Error finding container a27d3ccdcfdc5dc41993d8cbac00f76406f5f51cfaa62966ebf1bafcaef083d0: Status 404 returned error can't find the container with id a27d3ccdcfdc5dc41993d8cbac00f76406f5f51cfaa62966ebf1bafcaef083d0 Oct 29 23:23:22.256695 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:22.256673 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ffd0336a-kz9q8"] Oct 29 23:23:22.285096 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:22.285061 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ffd0336a-kz9q8" event={"ID":"9570ca18-a7cc-4041-b505-e11161579302","Type":"ContainerStarted","Data":"a27d3ccdcfdc5dc41993d8cbac00f76406f5f51cfaa62966ebf1bafcaef083d0"} Oct 29 23:23:22.388167 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:22.388124 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:23:23.083605 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:23.083570 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b6acce1-qzn47"] Oct 29 23:23:23.090891 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:23.090866 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b6acce1-qzn47" Oct 29 23:23:23.123297 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:23.117977 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b6acce1-qzn47"] Oct 29 23:23:23.182227 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:23.182193 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mgkw5\" (UniqueName: \"kubernetes.io/projected/4fab3853-8396-4cf1-af41-96c367c1e4f4-kube-api-access-mgkw5\") pod \"maas-api-revoke-sub-1b6acce1-qzn47\" (UID: \"4fab3853-8396-4cf1-af41-96c367c1e4f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b6acce1-qzn47" Oct 29 23:23:23.182421 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:23.182309 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4fab3853-8396-4cf1-af41-96c367c1e4f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1b6acce1-qzn47\" (UID: \"4fab3853-8396-4cf1-af41-96c367c1e4f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b6acce1-qzn47" Oct 29 23:23:23.182497 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:23.182479 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4fab3853-8396-4cf1-af41-96c367c1e4f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1b6acce1-qzn47\" (UID: \"4fab3853-8396-4cf1-af41-96c367c1e4f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b6acce1-qzn47" Oct 29 23:23:23.238596 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:23.238556 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c5febaf3-x49n2"] Oct 29 23:23:23.244603 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:23.244577 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5febaf3-x49n2" Oct 29 23:23:23.268490 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:23.268464 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c5febaf3-x49n2"] Oct 29 23:23:23.284386 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:23.284157 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4fab3853-8396-4cf1-af41-96c367c1e4f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1b6acce1-qzn47\" (UID: \"4fab3853-8396-4cf1-af41-96c367c1e4f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b6acce1-qzn47" Oct 29 23:23:23.285525 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:23.284685 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgkw5\" (UniqueName: \"kubernetes.io/projected/4fab3853-8396-4cf1-af41-96c367c1e4f4-kube-api-access-mgkw5\") pod \"maas-api-revoke-sub-1b6acce1-qzn47\" (UID: \"4fab3853-8396-4cf1-af41-96c367c1e4f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b6acce1-qzn47" Oct 29 23:23:23.285525 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:23.284775 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4fab3853-8396-4cf1-af41-96c367c1e4f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1b6acce1-qzn47\" (UID: \"4fab3853-8396-4cf1-af41-96c367c1e4f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b6acce1-qzn47" Oct 29 23:23:23.286474 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:23.286061 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4fab3853-8396-4cf1-af41-96c367c1e4f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1b6acce1-qzn47\" (UID: \"4fab3853-8396-4cf1-af41-96c367c1e4f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b6acce1-qzn47" Oct 29 23:23:23.290973 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:23.290943 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4fab3853-8396-4cf1-af41-96c367c1e4f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1b6acce1-qzn47\" (UID: \"4fab3853-8396-4cf1-af41-96c367c1e4f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b6acce1-qzn47" Oct 29 23:23:23.304392 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:23.304317 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ffd0336a-kz9q8" event={"ID":"9570ca18-a7cc-4041-b505-e11161579302","Type":"ContainerStarted","Data":"328cd3316a84326fd27c3b4cf8da90944a69f676f7485700daf2198325cb050e"} Oct 29 23:23:23.326134 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:23.326103 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgkw5\" (UniqueName: \"kubernetes.io/projected/4fab3853-8396-4cf1-af41-96c367c1e4f4-kube-api-access-mgkw5\") pod \"maas-api-revoke-sub-1b6acce1-qzn47\" (UID: \"4fab3853-8396-4cf1-af41-96c367c1e4f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b6acce1-qzn47" Oct 29 23:23:23.353728 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:23.353638 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ffd0336a-kz9q8" podStartSLOduration=2.353621743 podStartE2EDuration="2.353621743s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:23:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:23:23.352268867 +0000 UTC m=+2984.135482305" watchObservedRunningTime="2026-10-29 23:23:23.353621743 +0000 UTC m=+2984.136835172" Oct 29 23:23:23.386126 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:23.386093 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gl8h6\" (UniqueName: \"kubernetes.io/projected/8eb9f5be-be7c-4469-a840-92fe5a4600fd-kube-api-access-gl8h6\") pod \"maas-api-revoke-sub-c5febaf3-x49n2\" (UID: \"8eb9f5be-be7c-4469-a840-92fe5a4600fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5febaf3-x49n2" Oct 29 23:23:23.386398 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:23.386367 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8eb9f5be-be7c-4469-a840-92fe5a4600fd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c5febaf3-x49n2\" (UID: \"8eb9f5be-be7c-4469-a840-92fe5a4600fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5febaf3-x49n2" Oct 29 23:23:23.386550 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:23.386490 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8eb9f5be-be7c-4469-a840-92fe5a4600fd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c5febaf3-x49n2\" (UID: \"8eb9f5be-be7c-4469-a840-92fe5a4600fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5febaf3-x49n2" Oct 29 23:23:23.419478 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:23.419447 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b6acce1-qzn47" Oct 29 23:23:23.489261 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:23.489210 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gl8h6\" (UniqueName: \"kubernetes.io/projected/8eb9f5be-be7c-4469-a840-92fe5a4600fd-kube-api-access-gl8h6\") pod \"maas-api-revoke-sub-c5febaf3-x49n2\" (UID: \"8eb9f5be-be7c-4469-a840-92fe5a4600fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5febaf3-x49n2" Oct 29 23:23:23.489411 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:23.489270 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8eb9f5be-be7c-4469-a840-92fe5a4600fd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c5febaf3-x49n2\" (UID: \"8eb9f5be-be7c-4469-a840-92fe5a4600fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5febaf3-x49n2" Oct 29 23:23:23.489551 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:23.489467 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8eb9f5be-be7c-4469-a840-92fe5a4600fd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c5febaf3-x49n2\" (UID: \"8eb9f5be-be7c-4469-a840-92fe5a4600fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5febaf3-x49n2" Oct 29 23:23:23.490131 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:23.490092 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8eb9f5be-be7c-4469-a840-92fe5a4600fd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c5febaf3-x49n2\" (UID: \"8eb9f5be-be7c-4469-a840-92fe5a4600fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5febaf3-x49n2" Oct 29 23:23:23.492906 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:23.492881 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8eb9f5be-be7c-4469-a840-92fe5a4600fd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c5febaf3-x49n2\" (UID: \"8eb9f5be-be7c-4469-a840-92fe5a4600fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5febaf3-x49n2" Oct 29 23:23:23.515144 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:23.515087 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gl8h6\" (UniqueName: \"kubernetes.io/projected/8eb9f5be-be7c-4469-a840-92fe5a4600fd-kube-api-access-gl8h6\") pod \"maas-api-revoke-sub-c5febaf3-x49n2\" (UID: \"8eb9f5be-be7c-4469-a840-92fe5a4600fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5febaf3-x49n2" Oct 29 23:23:23.561493 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:23.561457 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5febaf3-x49n2" Oct 29 23:23:23.740079 ip-10-0-140-74 kubenswrapper[5057]: W1029 23:23:23.740026 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fab3853_8396_4cf1_af41_96c367c1e4f4.slice/crio-7ed2520cf0340c0c382062292086fbd75a9dde585d1aaadc8f76b58a8abd66a4 WatchSource:0}: Error finding container 7ed2520cf0340c0c382062292086fbd75a9dde585d1aaadc8f76b58a8abd66a4: Status 404 returned error can't find the container with id 7ed2520cf0340c0c382062292086fbd75a9dde585d1aaadc8f76b58a8abd66a4 Oct 29 23:23:23.740237 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:23.740162 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b6acce1-qzn47"] Oct 29 23:23:23.907885 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:23.907847 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c5febaf3-x49n2"] Oct 29 23:23:24.053598 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:24.053557 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:23:24.319680 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:24.319640 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b6acce1-qzn47" event={"ID":"4fab3853-8396-4cf1-af41-96c367c1e4f4","Type":"ContainerStarted","Data":"646ae53e0347d12b4f917965dd68d7ac502e13417b3fa374db8b92de7af9ec22"} Oct 29 23:23:24.320134 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:24.319693 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b6acce1-qzn47" event={"ID":"4fab3853-8396-4cf1-af41-96c367c1e4f4","Type":"ContainerStarted","Data":"7ed2520cf0340c0c382062292086fbd75a9dde585d1aaadc8f76b58a8abd66a4"} Oct 29 23:23:24.323010 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:24.322968 5057 generic.go:356] "Generic (PLEG): container finished" podID="9570ca18-a7cc-4041-b505-e11161579302" containerID="328cd3316a84326fd27c3b4cf8da90944a69f676f7485700daf2198325cb050e" exitCode=0 Oct 29 23:23:24.323137 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:24.323049 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ffd0336a-kz9q8" event={"ID":"9570ca18-a7cc-4041-b505-e11161579302","Type":"ContainerDied","Data":"328cd3316a84326fd27c3b4cf8da90944a69f676f7485700daf2198325cb050e"} Oct 29 23:23:24.324901 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:24.324731 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5febaf3-x49n2" event={"ID":"8eb9f5be-be7c-4469-a840-92fe5a4600fd","Type":"ContainerStarted","Data":"125215fe0828e64a3764787fc7345e545a5a318f74d1dda811c97e7424639742"} Oct 29 23:23:24.367504 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:24.367452 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b6acce1-qzn47" podStartSLOduration=1.367433843 podStartE2EDuration="1.367433843s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:23:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:23:24.358904048 +0000 UTC m=+2985.142117473" watchObservedRunningTime="2026-10-29 23:23:24.367433843 +0000 UTC m=+2985.150647273" Oct 29 23:23:25.337729 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:25.337697 5057 generic.go:356] "Generic (PLEG): container finished" podID="8eb9f5be-be7c-4469-a840-92fe5a4600fd" containerID="30961bbe0bfe08fd7af701cc7e8ca63e2cfe05ef2ca534e53704101b18fc37cf" exitCode=0 Oct 29 23:23:25.338174 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:25.337774 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5febaf3-x49n2" event={"ID":"8eb9f5be-be7c-4469-a840-92fe5a4600fd","Type":"ContainerDied","Data":"30961bbe0bfe08fd7af701cc7e8ca63e2cfe05ef2ca534e53704101b18fc37cf"} Oct 29 23:23:25.339624 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:25.339602 5057 generic.go:356] "Generic (PLEG): container finished" podID="4fab3853-8396-4cf1-af41-96c367c1e4f4" containerID="646ae53e0347d12b4f917965dd68d7ac502e13417b3fa374db8b92de7af9ec22" exitCode=0 Oct 29 23:23:25.339749 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:25.339679 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b6acce1-qzn47" event={"ID":"4fab3853-8396-4cf1-af41-96c367c1e4f4","Type":"ContainerDied","Data":"646ae53e0347d12b4f917965dd68d7ac502e13417b3fa374db8b92de7af9ec22"} Oct 29 23:23:25.544690 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:25.544665 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ffd0336a-kz9q8" Oct 29 23:23:25.640948 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:25.640865 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9570ca18-a7cc-4041-b505-e11161579302-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9570ca18-a7cc-4041-b505-e11161579302-maas-api-xks-ca\") pod \"9570ca18-a7cc-4041-b505-e11161579302\" (UID: \"9570ca18-a7cc-4041-b505-e11161579302\") " Oct 29 23:23:25.641637 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:25.640959 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9570ca18-a7cc-4041-b505-e11161579302-kube-api-access-b944d\" (UniqueName: \"kubernetes.io/projected/9570ca18-a7cc-4041-b505-e11161579302-kube-api-access-b944d\") pod \"9570ca18-a7cc-4041-b505-e11161579302\" (UID: \"9570ca18-a7cc-4041-b505-e11161579302\") " Oct 29 23:23:25.641637 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:25.641311 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9570ca18-a7cc-4041-b505-e11161579302-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9570ca18-a7cc-4041-b505-e11161579302-maas-api-openshift-ca\") pod \"9570ca18-a7cc-4041-b505-e11161579302\" (UID: \"9570ca18-a7cc-4041-b505-e11161579302\") " Oct 29 23:23:25.642022 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:25.641991 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9570ca18-a7cc-4041-b505-e11161579302-maas-api-openshift-ca" pod "9570ca18-a7cc-4041-b505-e11161579302" (UID: "9570ca18-a7cc-4041-b505-e11161579302"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:23:25.644155 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:25.644127 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9570ca18-a7cc-4041-b505-e11161579302-maas-api-xks-ca" pod "9570ca18-a7cc-4041-b505-e11161579302" (UID: "9570ca18-a7cc-4041-b505-e11161579302"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:23:25.646475 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:25.646444 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9570ca18-a7cc-4041-b505-e11161579302-kube-api-access-b944d" pod "9570ca18-a7cc-4041-b505-e11161579302" (UID: "9570ca18-a7cc-4041-b505-e11161579302"). InnerVolumeSpecName "kube-api-access-b944d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:23:25.710942 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:25.710904 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:23:25.744027 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:25.744001 5057 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9570ca18-a7cc-4041-b505-e11161579302-maas-api-openshift-ca\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:23:25.744027 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:25.744024 5057 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9570ca18-a7cc-4041-b505-e11161579302-maas-api-xks-ca\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:23:25.744179 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:25.744034 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b944d\" (UniqueName: \"kubernetes.io/projected/9570ca18-a7cc-4041-b505-e11161579302-kube-api-access-b944d\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:23:26.351051 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:26.351018 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ffd0336a-kz9q8" Oct 29 23:23:26.351482 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:26.351059 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ffd0336a-kz9q8" event={"ID":"9570ca18-a7cc-4041-b505-e11161579302","Type":"ContainerDied","Data":"a27d3ccdcfdc5dc41993d8cbac00f76406f5f51cfaa62966ebf1bafcaef083d0"} Oct 29 23:23:26.351482 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:26.351107 5057 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a27d3ccdcfdc5dc41993d8cbac00f76406f5f51cfaa62966ebf1bafcaef083d0" Oct 29 23:23:26.644965 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:26.644935 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5febaf3-x49n2" Oct 29 23:23:26.647597 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:26.647578 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b6acce1-qzn47" Oct 29 23:23:26.659534 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:26.659510 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4fab3853-8396-4cf1-af41-96c367c1e4f4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4fab3853-8396-4cf1-af41-96c367c1e4f4-maas-api-openshift-ca\") pod \"4fab3853-8396-4cf1-af41-96c367c1e4f4\" (UID: \"4fab3853-8396-4cf1-af41-96c367c1e4f4\") " Oct 29 23:23:26.659673 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:26.659659 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8eb9f5be-be7c-4469-a840-92fe5a4600fd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8eb9f5be-be7c-4469-a840-92fe5a4600fd-maas-api-xks-ca\") pod \"8eb9f5be-be7c-4469-a840-92fe5a4600fd\" (UID: \"8eb9f5be-be7c-4469-a840-92fe5a4600fd\") " Oct 29 23:23:26.659724 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:26.659708 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4fab3853-8396-4cf1-af41-96c367c1e4f4-kube-api-access-mgkw5\" (UniqueName: \"kubernetes.io/projected/4fab3853-8396-4cf1-af41-96c367c1e4f4-kube-api-access-mgkw5\") pod \"4fab3853-8396-4cf1-af41-96c367c1e4f4\" (UID: \"4fab3853-8396-4cf1-af41-96c367c1e4f4\") " Oct 29 23:23:26.659784 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:26.659756 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4fab3853-8396-4cf1-af41-96c367c1e4f4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4fab3853-8396-4cf1-af41-96c367c1e4f4-maas-api-xks-ca\") pod \"4fab3853-8396-4cf1-af41-96c367c1e4f4\" (UID: \"4fab3853-8396-4cf1-af41-96c367c1e4f4\") " Oct 29 23:23:26.659846 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:26.659815 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8eb9f5be-be7c-4469-a840-92fe5a4600fd-kube-api-access-gl8h6\" (UniqueName: \"kubernetes.io/projected/8eb9f5be-be7c-4469-a840-92fe5a4600fd-kube-api-access-gl8h6\") pod \"8eb9f5be-be7c-4469-a840-92fe5a4600fd\" (UID: \"8eb9f5be-be7c-4469-a840-92fe5a4600fd\") " Oct 29 23:23:26.659906 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:26.659850 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8eb9f5be-be7c-4469-a840-92fe5a4600fd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8eb9f5be-be7c-4469-a840-92fe5a4600fd-maas-api-openshift-ca\") pod \"8eb9f5be-be7c-4469-a840-92fe5a4600fd\" (UID: \"8eb9f5be-be7c-4469-a840-92fe5a4600fd\") " Oct 29 23:23:26.659906 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:26.659853 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4fab3853-8396-4cf1-af41-96c367c1e4f4-maas-api-openshift-ca" pod "4fab3853-8396-4cf1-af41-96c367c1e4f4" (UID: "4fab3853-8396-4cf1-af41-96c367c1e4f4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:23:26.660310 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:26.660283 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8eb9f5be-be7c-4469-a840-92fe5a4600fd-maas-api-openshift-ca" pod "8eb9f5be-be7c-4469-a840-92fe5a4600fd" (UID: "8eb9f5be-be7c-4469-a840-92fe5a4600fd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:23:26.660859 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:26.660834 5057 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8eb9f5be-be7c-4469-a840-92fe5a4600fd-maas-api-openshift-ca\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:23:26.660967 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:26.660866 5057 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4fab3853-8396-4cf1-af41-96c367c1e4f4-maas-api-openshift-ca\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:23:26.662120 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:26.662098 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8eb9f5be-be7c-4469-a840-92fe5a4600fd-maas-api-xks-ca" pod "8eb9f5be-be7c-4469-a840-92fe5a4600fd" (UID: "8eb9f5be-be7c-4469-a840-92fe5a4600fd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:23:26.662120 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:26.662107 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4fab3853-8396-4cf1-af41-96c367c1e4f4-maas-api-xks-ca" pod "4fab3853-8396-4cf1-af41-96c367c1e4f4" (UID: "4fab3853-8396-4cf1-af41-96c367c1e4f4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:23:26.662244 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:26.662146 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fab3853-8396-4cf1-af41-96c367c1e4f4-kube-api-access-mgkw5" pod "4fab3853-8396-4cf1-af41-96c367c1e4f4" (UID: "4fab3853-8396-4cf1-af41-96c367c1e4f4"). InnerVolumeSpecName "kube-api-access-mgkw5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:23:26.662244 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:26.662167 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8eb9f5be-be7c-4469-a840-92fe5a4600fd-kube-api-access-gl8h6" pod "8eb9f5be-be7c-4469-a840-92fe5a4600fd" (UID: "8eb9f5be-be7c-4469-a840-92fe5a4600fd"). InnerVolumeSpecName "kube-api-access-gl8h6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:23:26.762259 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:26.762227 5057 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4fab3853-8396-4cf1-af41-96c367c1e4f4-maas-api-xks-ca\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:23:26.762259 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:26.762255 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gl8h6\" (UniqueName: \"kubernetes.io/projected/8eb9f5be-be7c-4469-a840-92fe5a4600fd-kube-api-access-gl8h6\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:23:26.762259 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:26.762266 5057 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8eb9f5be-be7c-4469-a840-92fe5a4600fd-maas-api-xks-ca\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:23:26.762608 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:26.762274 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mgkw5\" (UniqueName: \"kubernetes.io/projected/4fab3853-8396-4cf1-af41-96c367c1e4f4-kube-api-access-mgkw5\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:23:27.318391 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:27.318350 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:23:27.424234 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:27.424201 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b6acce1-qzn47" Oct 29 23:23:27.425178 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:27.425147 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b6acce1-qzn47" event={"ID":"4fab3853-8396-4cf1-af41-96c367c1e4f4","Type":"ContainerDied","Data":"7ed2520cf0340c0c382062292086fbd75a9dde585d1aaadc8f76b58a8abd66a4"} Oct 29 23:23:27.425312 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:27.425203 5057 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7ed2520cf0340c0c382062292086fbd75a9dde585d1aaadc8f76b58a8abd66a4" Oct 29 23:23:27.428552 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:27.427151 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5febaf3-x49n2" event={"ID":"8eb9f5be-be7c-4469-a840-92fe5a4600fd","Type":"ContainerDied","Data":"125215fe0828e64a3764787fc7345e545a5a318f74d1dda811c97e7424639742"} Oct 29 23:23:27.428552 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:27.427194 5057 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="125215fe0828e64a3764787fc7345e545a5a318f74d1dda811c97e7424639742" Oct 29 23:23:27.428552 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:27.427249 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5febaf3-x49n2" Oct 29 23:23:29.004893 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:29.004851 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:23:30.544820 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:30.544780 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:23:32.191697 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:32.191659 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:23:32.295580 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:32.295538 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:23:33.920774 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:33.920737 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:23:34.730116 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:34.730080 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:23:35.596875 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:35.596839 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:23:36.071356 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:36.071303 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:23:36.437297 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:36.437203 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-462fb2-tlzsg"] Oct 29 23:23:36.438596 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:36.438566 5057 cpu_manager.go:414] "containerMap: removing container" podUID="8eb9f5be-be7c-4469-a840-92fe5a4600fd" containerName="revoke-keys" Oct 29 23:23:36.438596 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:36.438594 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8eb9f5be-be7c-4469-a840-92fe5a4600fd" containerName="revoke-keys" Oct 29 23:23:36.438779 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:36.438627 5057 cpu_manager.go:414] "containerMap: removing container" podUID="4fab3853-8396-4cf1-af41-96c367c1e4f4" containerName="revoke-keys" Oct 29 23:23:36.438779 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:36.438636 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4fab3853-8396-4cf1-af41-96c367c1e4f4" containerName="revoke-keys" Oct 29 23:23:36.438779 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:36.438736 5057 cpu_manager.go:414] "containerMap: removing container" podUID="9570ca18-a7cc-4041-b505-e11161579302" containerName="revoke-keys" Oct 29 23:23:36.438779 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:36.438746 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9570ca18-a7cc-4041-b505-e11161579302" containerName="revoke-keys" Oct 29 23:23:36.438995 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:36.438949 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="8eb9f5be-be7c-4469-a840-92fe5a4600fd" containerName="revoke-keys" Oct 29 23:23:36.438995 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:36.438963 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="4fab3853-8396-4cf1-af41-96c367c1e4f4" containerName="revoke-keys" Oct 29 23:23:36.438995 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:36.438978 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="9570ca18-a7cc-4041-b505-e11161579302" containerName="revoke-keys" Oct 29 23:23:36.445516 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:36.445496 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-462fb2-tlzsg" Oct 29 23:23:36.460092 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:36.460069 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dmx57\"" Oct 29 23:23:36.460208 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:36.460134 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:23:36.463058 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:36.463034 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-462fb2-tlzsg"] Oct 29 23:23:36.495669 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:36.495624 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8tjg7\" (UniqueName: \"kubernetes.io/projected/d5e27ce5-3b2b-41d1-ac96-e82780ff1c4e-kube-api-access-8tjg7\") pod \"maas-api-revoke-keys-e2e-inf-a-462fb2-tlzsg\" (UID: \"d5e27ce5-3b2b-41d1-ac96-e82780ff1c4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-462fb2-tlzsg" Oct 29 23:23:36.495841 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:36.495807 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5e27ce5-3b2b-41d1-ac96-e82780ff1c4e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-462fb2-tlzsg\" (UID: \"d5e27ce5-3b2b-41d1-ac96-e82780ff1c4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-462fb2-tlzsg" Oct 29 23:23:36.495940 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:36.495841 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5e27ce5-3b2b-41d1-ac96-e82780ff1c4e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-462fb2-tlzsg\" (UID: \"d5e27ce5-3b2b-41d1-ac96-e82780ff1c4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-462fb2-tlzsg" Oct 29 23:23:36.598175 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:36.598133 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8tjg7\" (UniqueName: \"kubernetes.io/projected/d5e27ce5-3b2b-41d1-ac96-e82780ff1c4e-kube-api-access-8tjg7\") pod \"maas-api-revoke-keys-e2e-inf-a-462fb2-tlzsg\" (UID: \"d5e27ce5-3b2b-41d1-ac96-e82780ff1c4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-462fb2-tlzsg" Oct 29 23:23:36.598775 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:36.598409 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5e27ce5-3b2b-41d1-ac96-e82780ff1c4e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-462fb2-tlzsg\" (UID: \"d5e27ce5-3b2b-41d1-ac96-e82780ff1c4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-462fb2-tlzsg" Oct 29 23:23:36.598775 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:36.598452 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5e27ce5-3b2b-41d1-ac96-e82780ff1c4e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-462fb2-tlzsg\" (UID: \"d5e27ce5-3b2b-41d1-ac96-e82780ff1c4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-462fb2-tlzsg" Oct 29 23:23:36.599199 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:36.599169 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5e27ce5-3b2b-41d1-ac96-e82780ff1c4e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-462fb2-tlzsg\" (UID: \"d5e27ce5-3b2b-41d1-ac96-e82780ff1c4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-462fb2-tlzsg" Oct 29 23:23:36.601180 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:36.601149 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5e27ce5-3b2b-41d1-ac96-e82780ff1c4e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-462fb2-tlzsg\" (UID: \"d5e27ce5-3b2b-41d1-ac96-e82780ff1c4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-462fb2-tlzsg" Oct 29 23:23:36.621306 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:36.621278 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8tjg7\" (UniqueName: \"kubernetes.io/projected/d5e27ce5-3b2b-41d1-ac96-e82780ff1c4e-kube-api-access-8tjg7\") pod \"maas-api-revoke-keys-e2e-inf-a-462fb2-tlzsg\" (UID: \"d5e27ce5-3b2b-41d1-ac96-e82780ff1c4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-462fb2-tlzsg" Oct 29 23:23:36.764079 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:36.764046 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-462fb2-tlzsg" Oct 29 23:23:36.915291 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:36.915245 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:23:37.137419 ip-10-0-140-74 kubenswrapper[5057]: W1029 23:23:37.137387 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5e27ce5_3b2b_41d1_ac96_e82780ff1c4e.slice/crio-e156174a0fece04cc5407b2c463050e216fd01bb00e1e4629b18f5769aee5a42 WatchSource:0}: Error finding container e156174a0fece04cc5407b2c463050e216fd01bb00e1e4629b18f5769aee5a42: Status 404 returned error can't find the container with id e156174a0fece04cc5407b2c463050e216fd01bb00e1e4629b18f5769aee5a42 Oct 29 23:23:37.138693 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:37.138668 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-462fb2-tlzsg"] Oct 29 23:23:37.555989 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:37.555955 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-462fb2-tlzsg" event={"ID":"d5e27ce5-3b2b-41d1-ac96-e82780ff1c4e","Type":"ContainerStarted","Data":"81e3ca73979b9dcd5daef0ec7d2f3ac632c25c49434e19b8afe04dfa20fe307e"} Oct 29 23:23:37.556185 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:37.556008 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-462fb2-tlzsg" event={"ID":"d5e27ce5-3b2b-41d1-ac96-e82780ff1c4e","Type":"ContainerStarted","Data":"e156174a0fece04cc5407b2c463050e216fd01bb00e1e4629b18f5769aee5a42"} Oct 29 23:23:37.597657 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:37.597607 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-462fb2-tlzsg" podStartSLOduration=1.5975913579999999 podStartE2EDuration="1.597591358s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:23:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:23:37.595054042 +0000 UTC m=+2998.378267467" watchObservedRunningTime="2026-10-29 23:23:37.597591358 +0000 UTC m=+2998.380804793" Oct 29 23:23:38.567621 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:38.567539 5057 generic.go:356] "Generic (PLEG): container finished" podID="d5e27ce5-3b2b-41d1-ac96-e82780ff1c4e" containerID="81e3ca73979b9dcd5daef0ec7d2f3ac632c25c49434e19b8afe04dfa20fe307e" exitCode=0 Oct 29 23:23:38.567996 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:38.567619 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-462fb2-tlzsg" event={"ID":"d5e27ce5-3b2b-41d1-ac96-e82780ff1c4e","Type":"ContainerDied","Data":"81e3ca73979b9dcd5daef0ec7d2f3ac632c25c49434e19b8afe04dfa20fe307e"} Oct 29 23:23:38.669939 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:38.669902 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:23:39.335200 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:39.335155 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:23:39.821585 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:39.821555 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-462fb2-tlzsg" Oct 29 23:23:39.855615 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:39.855585 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e587514-z6mqr"] Oct 29 23:23:39.856658 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:39.856640 5057 cpu_manager.go:414] "containerMap: removing container" podUID="d5e27ce5-3b2b-41d1-ac96-e82780ff1c4e" containerName="revoke-keys" Oct 29 23:23:39.856717 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:39.856660 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5e27ce5-3b2b-41d1-ac96-e82780ff1c4e" containerName="revoke-keys" Oct 29 23:23:39.856821 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:39.856811 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5e27ce5-3b2b-41d1-ac96-e82780ff1c4e" containerName="revoke-keys" Oct 29 23:23:39.861184 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:39.860988 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e587514-z6mqr" Oct 29 23:23:39.895652 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:39.888022 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5e27ce5-3b2b-41d1-ac96-e82780ff1c4e-kube-api-access-8tjg7\" (UniqueName: \"kubernetes.io/projected/d5e27ce5-3b2b-41d1-ac96-e82780ff1c4e-kube-api-access-8tjg7\") pod \"d5e27ce5-3b2b-41d1-ac96-e82780ff1c4e\" (UID: \"d5e27ce5-3b2b-41d1-ac96-e82780ff1c4e\") " Oct 29 23:23:39.895652 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:39.888082 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5e27ce5-3b2b-41d1-ac96-e82780ff1c4e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5e27ce5-3b2b-41d1-ac96-e82780ff1c4e-maas-api-openshift-ca\") pod \"d5e27ce5-3b2b-41d1-ac96-e82780ff1c4e\" (UID: \"d5e27ce5-3b2b-41d1-ac96-e82780ff1c4e\") " Oct 29 23:23:39.895652 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:39.888147 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d5e27ce5-3b2b-41d1-ac96-e82780ff1c4e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5e27ce5-3b2b-41d1-ac96-e82780ff1c4e-maas-api-xks-ca\") pod \"d5e27ce5-3b2b-41d1-ac96-e82780ff1c4e\" (UID: \"d5e27ce5-3b2b-41d1-ac96-e82780ff1c4e\") " Oct 29 23:23:39.901394 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:39.896078 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5e27ce5-3b2b-41d1-ac96-e82780ff1c4e-maas-api-xks-ca" pod "d5e27ce5-3b2b-41d1-ac96-e82780ff1c4e" (UID: "d5e27ce5-3b2b-41d1-ac96-e82780ff1c4e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:23:39.915139 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:39.913826 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5e27ce5-3b2b-41d1-ac96-e82780ff1c4e-maas-api-openshift-ca" pod "d5e27ce5-3b2b-41d1-ac96-e82780ff1c4e" (UID: "d5e27ce5-3b2b-41d1-ac96-e82780ff1c4e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:23:39.917544 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:39.917506 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5e27ce5-3b2b-41d1-ac96-e82780ff1c4e-kube-api-access-8tjg7" pod "d5e27ce5-3b2b-41d1-ac96-e82780ff1c4e" (UID: "d5e27ce5-3b2b-41d1-ac96-e82780ff1c4e"). InnerVolumeSpecName "kube-api-access-8tjg7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:23:39.945074 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:39.944714 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e587514-z6mqr"] Oct 29 23:23:39.990310 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:39.990263 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfqfs\" (UniqueName: \"kubernetes.io/projected/b2061553-dc6a-44aa-bfa1-02390bd63b8c-kube-api-access-kfqfs\") pod \"maas-api-revoke-sub-7e587514-z6mqr\" (UID: \"b2061553-dc6a-44aa-bfa1-02390bd63b8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e587514-z6mqr" Oct 29 23:23:39.990517 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:39.990356 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2061553-dc6a-44aa-bfa1-02390bd63b8c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7e587514-z6mqr\" (UID: \"b2061553-dc6a-44aa-bfa1-02390bd63b8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e587514-z6mqr" Oct 29 23:23:39.990517 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:39.990465 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2061553-dc6a-44aa-bfa1-02390bd63b8c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7e587514-z6mqr\" (UID: \"b2061553-dc6a-44aa-bfa1-02390bd63b8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e587514-z6mqr" Oct 29 23:23:39.990791 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:39.990770 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8tjg7\" (UniqueName: \"kubernetes.io/projected/d5e27ce5-3b2b-41d1-ac96-e82780ff1c4e-kube-api-access-8tjg7\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:23:39.990892 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:39.990794 5057 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5e27ce5-3b2b-41d1-ac96-e82780ff1c4e-maas-api-openshift-ca\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:23:39.990892 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:39.990810 5057 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5e27ce5-3b2b-41d1-ac96-e82780ff1c4e-maas-api-xks-ca\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:23:40.017418 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:40.017394 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-vj6md_aa6a190e-3283-458a-9dfd-2de7cb193fc5/console-operator/2.log" Oct 29 23:23:40.026670 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:40.026647 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-vj6md_aa6a190e-3283-458a-9dfd-2de7cb193fc5/console-operator/2.log" Oct 29 23:23:40.092437 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:40.092398 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2061553-dc6a-44aa-bfa1-02390bd63b8c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7e587514-z6mqr\" (UID: \"b2061553-dc6a-44aa-bfa1-02390bd63b8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e587514-z6mqr" Oct 29 23:23:40.092795 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:40.092773 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfqfs\" (UniqueName: \"kubernetes.io/projected/b2061553-dc6a-44aa-bfa1-02390bd63b8c-kube-api-access-kfqfs\") pod \"maas-api-revoke-sub-7e587514-z6mqr\" (UID: \"b2061553-dc6a-44aa-bfa1-02390bd63b8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e587514-z6mqr" Oct 29 23:23:40.092859 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:40.092844 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2061553-dc6a-44aa-bfa1-02390bd63b8c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7e587514-z6mqr\" (UID: \"b2061553-dc6a-44aa-bfa1-02390bd63b8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e587514-z6mqr" Oct 29 23:23:40.093537 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:40.093507 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2061553-dc6a-44aa-bfa1-02390bd63b8c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7e587514-z6mqr\" (UID: \"b2061553-dc6a-44aa-bfa1-02390bd63b8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e587514-z6mqr" Oct 29 23:23:40.095046 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:40.095021 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2061553-dc6a-44aa-bfa1-02390bd63b8c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7e587514-z6mqr\" (UID: \"b2061553-dc6a-44aa-bfa1-02390bd63b8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e587514-z6mqr" Oct 29 23:23:40.116883 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:40.116854 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfqfs\" (UniqueName: \"kubernetes.io/projected/b2061553-dc6a-44aa-bfa1-02390bd63b8c-kube-api-access-kfqfs\") pod \"maas-api-revoke-sub-7e587514-z6mqr\" (UID: \"b2061553-dc6a-44aa-bfa1-02390bd63b8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e587514-z6mqr" Oct 29 23:23:40.213447 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:40.213420 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e587514-z6mqr" Oct 29 23:23:40.510321 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:40.510234 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:23:40.529966 ip-10-0-140-74 kubenswrapper[5057]: W1029 23:23:40.529936 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2061553_dc6a_44aa_bfa1_02390bd63b8c.slice/crio-7908bf3ac7538cccdf5b33208420232b7f5c6c6a9a0cb569333c12adc95b302c WatchSource:0}: Error finding container 7908bf3ac7538cccdf5b33208420232b7f5c6c6a9a0cb569333c12adc95b302c: Status 404 returned error can't find the container with id 7908bf3ac7538cccdf5b33208420232b7f5c6c6a9a0cb569333c12adc95b302c Oct 29 23:23:40.531150 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:40.531127 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e587514-z6mqr"] Oct 29 23:23:40.620057 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:40.620028 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-462fb2-tlzsg" Oct 29 23:23:40.620218 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:40.620057 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-462fb2-tlzsg" event={"ID":"d5e27ce5-3b2b-41d1-ac96-e82780ff1c4e","Type":"ContainerDied","Data":"e156174a0fece04cc5407b2c463050e216fd01bb00e1e4629b18f5769aee5a42"} Oct 29 23:23:40.620289 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:40.620226 5057 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e156174a0fece04cc5407b2c463050e216fd01bb00e1e4629b18f5769aee5a42" Oct 29 23:23:40.622186 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:40.622157 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e587514-z6mqr" event={"ID":"b2061553-dc6a-44aa-bfa1-02390bd63b8c","Type":"ContainerStarted","Data":"7908bf3ac7538cccdf5b33208420232b7f5c6c6a9a0cb569333c12adc95b302c"} Oct 29 23:23:41.517870 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:41.517816 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:23:41.633917 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:41.633876 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e587514-z6mqr" event={"ID":"b2061553-dc6a-44aa-bfa1-02390bd63b8c","Type":"ContainerStarted","Data":"ef290954fc8f72696e0cd88454e6ba0373fe7038a44d7a39bc145b3b42a90cb6"} Oct 29 23:23:41.688860 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:41.688796 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e587514-z6mqr" podStartSLOduration=2.688781801 podStartE2EDuration="2.688781801s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:23:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:23:41.674641663 +0000 UTC m=+3002.457855113" watchObservedRunningTime="2026-10-29 23:23:41.688781801 +0000 UTC m=+3002.471995226" Oct 29 23:23:42.184234 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:42.184134 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:23:42.644781 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:42.644748 5057 generic.go:356] "Generic (PLEG): container finished" podID="b2061553-dc6a-44aa-bfa1-02390bd63b8c" containerID="ef290954fc8f72696e0cd88454e6ba0373fe7038a44d7a39bc145b3b42a90cb6" exitCode=0 Oct 29 23:23:42.645231 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:42.644812 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e587514-z6mqr" event={"ID":"b2061553-dc6a-44aa-bfa1-02390bd63b8c","Type":"ContainerDied","Data":"ef290954fc8f72696e0cd88454e6ba0373fe7038a44d7a39bc145b3b42a90cb6"} Oct 29 23:23:43.722150 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:43.722106 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:23:43.921160 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:43.921135 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e587514-z6mqr" Oct 29 23:23:44.001069 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:44.001031 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2061553-dc6a-44aa-bfa1-02390bd63b8c-kube-api-access-kfqfs\" (UniqueName: \"kubernetes.io/projected/b2061553-dc6a-44aa-bfa1-02390bd63b8c-kube-api-access-kfqfs\") pod \"b2061553-dc6a-44aa-bfa1-02390bd63b8c\" (UID: \"b2061553-dc6a-44aa-bfa1-02390bd63b8c\") " Oct 29 23:23:44.001261 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:44.001170 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b2061553-dc6a-44aa-bfa1-02390bd63b8c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2061553-dc6a-44aa-bfa1-02390bd63b8c-maas-api-xks-ca\") pod \"b2061553-dc6a-44aa-bfa1-02390bd63b8c\" (UID: \"b2061553-dc6a-44aa-bfa1-02390bd63b8c\") " Oct 29 23:23:44.001416 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:44.001393 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b2061553-dc6a-44aa-bfa1-02390bd63b8c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2061553-dc6a-44aa-bfa1-02390bd63b8c-maas-api-openshift-ca\") pod \"b2061553-dc6a-44aa-bfa1-02390bd63b8c\" (UID: \"b2061553-dc6a-44aa-bfa1-02390bd63b8c\") " Oct 29 23:23:44.001870 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:44.001830 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2061553-dc6a-44aa-bfa1-02390bd63b8c-maas-api-openshift-ca" pod "b2061553-dc6a-44aa-bfa1-02390bd63b8c" (UID: "b2061553-dc6a-44aa-bfa1-02390bd63b8c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:23:44.002147 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:44.002126 5057 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2061553-dc6a-44aa-bfa1-02390bd63b8c-maas-api-openshift-ca\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:23:44.003723 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:44.003695 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2061553-dc6a-44aa-bfa1-02390bd63b8c-kube-api-access-kfqfs" pod "b2061553-dc6a-44aa-bfa1-02390bd63b8c" (UID: "b2061553-dc6a-44aa-bfa1-02390bd63b8c"). InnerVolumeSpecName "kube-api-access-kfqfs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:23:44.004059 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:44.004039 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2061553-dc6a-44aa-bfa1-02390bd63b8c-maas-api-xks-ca" pod "b2061553-dc6a-44aa-bfa1-02390bd63b8c" (UID: "b2061553-dc6a-44aa-bfa1-02390bd63b8c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:23:44.104208 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:44.104177 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kfqfs\" (UniqueName: \"kubernetes.io/projected/b2061553-dc6a-44aa-bfa1-02390bd63b8c-kube-api-access-kfqfs\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:23:44.104208 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:44.104204 5057 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2061553-dc6a-44aa-bfa1-02390bd63b8c-maas-api-xks-ca\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:23:44.667754 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:44.667721 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e587514-z6mqr" event={"ID":"b2061553-dc6a-44aa-bfa1-02390bd63b8c","Type":"ContainerDied","Data":"7908bf3ac7538cccdf5b33208420232b7f5c6c6a9a0cb569333c12adc95b302c"} Oct 29 23:23:44.667927 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:44.667771 5057 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7908bf3ac7538cccdf5b33208420232b7f5c6c6a9a0cb569333c12adc95b302c" Oct 29 23:23:44.667927 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:44.667774 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e587514-z6mqr" Oct 29 23:23:45.034170 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:45.034136 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e17e54f8-rqrcd"] Oct 29 23:23:45.043138 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:45.043112 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e17e54f8-rqrcd"] Oct 29 23:23:45.335316 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:45.335220 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40f5dbc7-03cf-4d4f-8257-5dd830f64b96" path="/var/lib/kubelet/pods/40f5dbc7-03cf-4d4f-8257-5dd830f64b96/volumes" Oct 29 23:23:45.389755 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:45.389715 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:23:46.867096 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:46.867063 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-462fb2-6999659b56-gh9vv"] Oct 29 23:23:46.957198 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:46.957160 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:23:47.072565 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:47.072527 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:23:48.717009 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:48.716970 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:23:49.032442 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:23:49.032363 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d2dnn" containerName="extractor" Oct 29 23:23:50.316213 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:50.316167 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:23:51.865911 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:51.865875 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:23:51.867716 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:51.867685 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-462fb2-6999659b56-gh9vv" podUID="d5c09a5d-ceed-4746-a4e7-31714b345610" containerName="payload-processing" containerID="cri-o://342ced5cabd04c31d69529dc652f207c19c8dec0fcc7db5ef94b46a48fc3084b" gracePeriod=25 Oct 29 23:23:53.422019 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:53.421980 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:23:54.020252 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:54.020211 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-462fb2-6999659b56-gh9vv" podUID="d5c09a5d-ceed-4746-a4e7-31714b345610" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:23:54.545733 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:54.545693 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:23:55.113693 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:55.113652 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb829d96-9l8hz"] Oct 29 23:23:55.115348 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:55.115309 5057 cpu_manager.go:414] "containerMap: removing container" podUID="b2061553-dc6a-44aa-bfa1-02390bd63b8c" containerName="revoke-keys" Oct 29 23:23:55.115511 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:55.115349 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2061553-dc6a-44aa-bfa1-02390bd63b8c" containerName="revoke-keys" Oct 29 23:23:55.115674 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:55.115658 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2061553-dc6a-44aa-bfa1-02390bd63b8c" containerName="revoke-keys" Oct 29 23:23:55.122584 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:55.122559 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb829d96-9l8hz" Oct 29 23:23:55.137639 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:55.137610 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb829d96-9l8hz"] Oct 29 23:23:55.144019 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:55.143985 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqg22\" (UniqueName: \"kubernetes.io/projected/7cff5a32-f779-4aa6-98c2-7b8ff21991c5-kube-api-access-fqg22\") pod \"maas-api-revoke-sub-fb829d96-9l8hz\" (UID: \"7cff5a32-f779-4aa6-98c2-7b8ff21991c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb829d96-9l8hz" Oct 29 23:23:55.144652 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:55.144190 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7cff5a32-f779-4aa6-98c2-7b8ff21991c5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fb829d96-9l8hz\" (UID: \"7cff5a32-f779-4aa6-98c2-7b8ff21991c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb829d96-9l8hz" Oct 29 23:23:55.144652 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:55.144368 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7cff5a32-f779-4aa6-98c2-7b8ff21991c5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fb829d96-9l8hz\" (UID: \"7cff5a32-f779-4aa6-98c2-7b8ff21991c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb829d96-9l8hz" Oct 29 23:23:55.152148 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:55.151977 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dmx57\"" Oct 29 23:23:55.152148 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:55.152082 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:23:55.215726 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:55.215677 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:23:55.276091 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:55.270855 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7cff5a32-f779-4aa6-98c2-7b8ff21991c5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fb829d96-9l8hz\" (UID: \"7cff5a32-f779-4aa6-98c2-7b8ff21991c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb829d96-9l8hz" Oct 29 23:23:55.276091 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:55.270996 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7cff5a32-f779-4aa6-98c2-7b8ff21991c5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fb829d96-9l8hz\" (UID: \"7cff5a32-f779-4aa6-98c2-7b8ff21991c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb829d96-9l8hz" Oct 29 23:23:55.276091 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:55.271427 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqg22\" (UniqueName: \"kubernetes.io/projected/7cff5a32-f779-4aa6-98c2-7b8ff21991c5-kube-api-access-fqg22\") pod \"maas-api-revoke-sub-fb829d96-9l8hz\" (UID: \"7cff5a32-f779-4aa6-98c2-7b8ff21991c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb829d96-9l8hz" Oct 29 23:23:55.306377 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:55.292567 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7cff5a32-f779-4aa6-98c2-7b8ff21991c5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fb829d96-9l8hz\" (UID: \"7cff5a32-f779-4aa6-98c2-7b8ff21991c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb829d96-9l8hz" Oct 29 23:23:55.310020 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:55.309979 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7cff5a32-f779-4aa6-98c2-7b8ff21991c5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fb829d96-9l8hz\" (UID: \"7cff5a32-f779-4aa6-98c2-7b8ff21991c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb829d96-9l8hz" Oct 29 23:23:55.330388 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:55.329858 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqg22\" (UniqueName: \"kubernetes.io/projected/7cff5a32-f779-4aa6-98c2-7b8ff21991c5-kube-api-access-fqg22\") pod \"maas-api-revoke-sub-fb829d96-9l8hz\" (UID: \"7cff5a32-f779-4aa6-98c2-7b8ff21991c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb829d96-9l8hz" Oct 29 23:23:55.446429 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:55.446316 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb829d96-9l8hz" Oct 29 23:23:55.765862 ip-10-0-140-74 kubenswrapper[5057]: W1029 23:23:55.765833 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cff5a32_f779_4aa6_98c2_7b8ff21991c5.slice/crio-833aa81ec7b37dc8f9930e3825d51a3adbaa8a7761576c043f12c123d1708ae2 WatchSource:0}: Error finding container 833aa81ec7b37dc8f9930e3825d51a3adbaa8a7761576c043f12c123d1708ae2: Status 404 returned error can't find the container with id 833aa81ec7b37dc8f9930e3825d51a3adbaa8a7761576c043f12c123d1708ae2 Oct 29 23:23:55.767340 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:55.767301 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb829d96-9l8hz"] Oct 29 23:23:55.828720 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:55.828688 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb829d96-9l8hz" event={"ID":"7cff5a32-f779-4aa6-98c2-7b8ff21991c5","Type":"ContainerStarted","Data":"833aa81ec7b37dc8f9930e3825d51a3adbaa8a7761576c043f12c123d1708ae2"} Oct 29 23:23:56.840588 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:56.840552 5057 generic.go:356] "Generic (PLEG): container finished" podID="7cff5a32-f779-4aa6-98c2-7b8ff21991c5" containerID="6aab7b4b33e9fe9a90e8953ddc3741dee968cfb8d23b118ddc04fdac74827168" exitCode=0 Oct 29 23:23:56.840977 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:56.840631 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb829d96-9l8hz" event={"ID":"7cff5a32-f779-4aa6-98c2-7b8ff21991c5","Type":"ContainerDied","Data":"6aab7b4b33e9fe9a90e8953ddc3741dee968cfb8d23b118ddc04fdac74827168"} Oct 29 23:23:56.858601 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:56.858573 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:23:58.074419 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:58.074389 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb829d96-9l8hz" Oct 29 23:23:58.120544 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:58.120465 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7cff5a32-f779-4aa6-98c2-7b8ff21991c5-kube-api-access-fqg22\" (UniqueName: \"kubernetes.io/projected/7cff5a32-f779-4aa6-98c2-7b8ff21991c5-kube-api-access-fqg22\") pod \"7cff5a32-f779-4aa6-98c2-7b8ff21991c5\" (UID: \"7cff5a32-f779-4aa6-98c2-7b8ff21991c5\") " Oct 29 23:23:58.120922 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:58.120899 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7cff5a32-f779-4aa6-98c2-7b8ff21991c5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7cff5a32-f779-4aa6-98c2-7b8ff21991c5-maas-api-openshift-ca\") pod \"7cff5a32-f779-4aa6-98c2-7b8ff21991c5\" (UID: \"7cff5a32-f779-4aa6-98c2-7b8ff21991c5\") " Oct 29 23:23:58.121199 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:58.121173 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7cff5a32-f779-4aa6-98c2-7b8ff21991c5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7cff5a32-f779-4aa6-98c2-7b8ff21991c5-maas-api-xks-ca\") pod \"7cff5a32-f779-4aa6-98c2-7b8ff21991c5\" (UID: \"7cff5a32-f779-4aa6-98c2-7b8ff21991c5\") " Oct 29 23:23:58.121638 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:58.121613 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7cff5a32-f779-4aa6-98c2-7b8ff21991c5-maas-api-openshift-ca" pod "7cff5a32-f779-4aa6-98c2-7b8ff21991c5" (UID: "7cff5a32-f779-4aa6-98c2-7b8ff21991c5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:23:58.122227 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:58.122197 5057 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7cff5a32-f779-4aa6-98c2-7b8ff21991c5-maas-api-openshift-ca\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:23:58.123608 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:58.123582 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7cff5a32-f779-4aa6-98c2-7b8ff21991c5-kube-api-access-fqg22" pod "7cff5a32-f779-4aa6-98c2-7b8ff21991c5" (UID: "7cff5a32-f779-4aa6-98c2-7b8ff21991c5"). InnerVolumeSpecName "kube-api-access-fqg22". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:23:58.123801 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:58.123782 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7cff5a32-f779-4aa6-98c2-7b8ff21991c5-maas-api-xks-ca" pod "7cff5a32-f779-4aa6-98c2-7b8ff21991c5" (UID: "7cff5a32-f779-4aa6-98c2-7b8ff21991c5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:23:58.223863 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:58.223822 5057 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7cff5a32-f779-4aa6-98c2-7b8ff21991c5-maas-api-xks-ca\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:23:58.223863 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:58.223857 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fqg22\" (UniqueName: \"kubernetes.io/projected/7cff5a32-f779-4aa6-98c2-7b8ff21991c5-kube-api-access-fqg22\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:23:58.423200 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:58.423066 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:23:58.867693 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:58.867664 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb829d96-9l8hz" Oct 29 23:23:58.867885 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:58.867698 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb829d96-9l8hz" event={"ID":"7cff5a32-f779-4aa6-98c2-7b8ff21991c5","Type":"ContainerDied","Data":"833aa81ec7b37dc8f9930e3825d51a3adbaa8a7761576c043f12c123d1708ae2"} Oct 29 23:23:58.867885 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:58.867750 5057 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="833aa81ec7b37dc8f9930e3825d51a3adbaa8a7761576c043f12c123d1708ae2" Oct 29 23:23:59.611639 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:23:59.611605 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:24:00.111089 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:00.111055 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9e7cc9-x27bf"] Oct 29 23:24:00.112764 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:00.112739 5057 cpu_manager.go:414] "containerMap: removing container" podUID="7cff5a32-f779-4aa6-98c2-7b8ff21991c5" containerName="revoke-keys" Oct 29 23:24:00.112867 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:00.112768 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7cff5a32-f779-4aa6-98c2-7b8ff21991c5" containerName="revoke-keys" Oct 29 23:24:00.113111 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:00.113093 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="7cff5a32-f779-4aa6-98c2-7b8ff21991c5" containerName="revoke-keys" Oct 29 23:24:00.118806 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:00.118782 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9e7cc9-x27bf" Oct 29 23:24:00.138253 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:00.138227 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9e7cc9-x27bf"] Oct 29 23:24:00.184428 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:00.183991 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:24:00.184428 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:00.184097 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dmx57\"" Oct 29 23:24:00.263135 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:00.261066 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ceebf00-f0b1-41b4-9ab2-8476d9038de1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-9e7cc9-x27bf\" (UID: \"7ceebf00-f0b1-41b4-9ab2-8476d9038de1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9e7cc9-x27bf" Oct 29 23:24:00.263135 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:00.261505 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8tt2t\" (UniqueName: \"kubernetes.io/projected/7ceebf00-f0b1-41b4-9ab2-8476d9038de1-kube-api-access-8tt2t\") pod \"maas-api-revoke-keys-e2e-inf-a-9e7cc9-x27bf\" (UID: \"7ceebf00-f0b1-41b4-9ab2-8476d9038de1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9e7cc9-x27bf" Oct 29 23:24:00.263135 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:00.261591 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ceebf00-f0b1-41b4-9ab2-8476d9038de1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-9e7cc9-x27bf\" (UID: \"7ceebf00-f0b1-41b4-9ab2-8476d9038de1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9e7cc9-x27bf" Oct 29 23:24:00.292802 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:00.292769 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:24:00.367715 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:00.367619 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8tt2t\" (UniqueName: \"kubernetes.io/projected/7ceebf00-f0b1-41b4-9ab2-8476d9038de1-kube-api-access-8tt2t\") pod \"maas-api-revoke-keys-e2e-inf-a-9e7cc9-x27bf\" (UID: \"7ceebf00-f0b1-41b4-9ab2-8476d9038de1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9e7cc9-x27bf" Oct 29 23:24:00.367898 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:00.367723 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ceebf00-f0b1-41b4-9ab2-8476d9038de1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-9e7cc9-x27bf\" (UID: \"7ceebf00-f0b1-41b4-9ab2-8476d9038de1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9e7cc9-x27bf" Oct 29 23:24:00.368165 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:00.367897 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ceebf00-f0b1-41b4-9ab2-8476d9038de1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-9e7cc9-x27bf\" (UID: \"7ceebf00-f0b1-41b4-9ab2-8476d9038de1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9e7cc9-x27bf" Oct 29 23:24:00.369223 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:00.369187 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ceebf00-f0b1-41b4-9ab2-8476d9038de1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-9e7cc9-x27bf\" (UID: \"7ceebf00-f0b1-41b4-9ab2-8476d9038de1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9e7cc9-x27bf" Oct 29 23:24:00.375384 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:00.375318 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ceebf00-f0b1-41b4-9ab2-8476d9038de1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-9e7cc9-x27bf\" (UID: \"7ceebf00-f0b1-41b4-9ab2-8476d9038de1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9e7cc9-x27bf" Oct 29 23:24:00.405516 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:00.405480 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8tt2t\" (UniqueName: \"kubernetes.io/projected/7ceebf00-f0b1-41b4-9ab2-8476d9038de1-kube-api-access-8tt2t\") pod \"maas-api-revoke-keys-e2e-inf-a-9e7cc9-x27bf\" (UID: \"7ceebf00-f0b1-41b4-9ab2-8476d9038de1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9e7cc9-x27bf" Oct 29 23:24:00.440678 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:00.440641 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9e7cc9-x27bf" Oct 29 23:24:00.756293 ip-10-0-140-74 kubenswrapper[5057]: W1029 23:24:00.756256 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ceebf00_f0b1_41b4_9ab2_8476d9038de1.slice/crio-db6674c781e61bdb287138b94ef4258746cf6a120472e5799677d8947eee6e57 WatchSource:0}: Error finding container db6674c781e61bdb287138b94ef4258746cf6a120472e5799677d8947eee6e57: Status 404 returned error can't find the container with id db6674c781e61bdb287138b94ef4258746cf6a120472e5799677d8947eee6e57 Oct 29 23:24:00.757065 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:00.757042 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9e7cc9-x27bf"] Oct 29 23:24:00.898745 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:00.898707 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9e7cc9-x27bf" event={"ID":"7ceebf00-f0b1-41b4-9ab2-8476d9038de1","Type":"ContainerStarted","Data":"db6674c781e61bdb287138b94ef4258746cf6a120472e5799677d8947eee6e57"} Oct 29 23:24:01.883249 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:01.883212 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:24:01.954280 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:01.954225 5057 generic.go:356] "Generic (PLEG): container finished" podID="7ceebf00-f0b1-41b4-9ab2-8476d9038de1" containerID="b0f1a55b3c652898148768d4a20760b3fadcc936a6919318572cc2d1af4536ac" exitCode=0 Oct 29 23:24:01.955241 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:01.954677 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9e7cc9-x27bf" event={"ID":"7ceebf00-f0b1-41b4-9ab2-8476d9038de1","Type":"ContainerDied","Data":"b0f1a55b3c652898148768d4a20760b3fadcc936a6919318572cc2d1af4536ac"} Oct 29 23:24:02.031878 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:02.031838 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-845d5270-5vjc9"] Oct 29 23:24:02.037286 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:02.037255 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-845d5270-5vjc9"] Oct 29 23:24:03.173320 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:03.173290 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9e7cc9-x27bf" Oct 29 23:24:03.320217 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:03.320125 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ceebf00-f0b1-41b4-9ab2-8476d9038de1-kube-api-access-8tt2t\" (UniqueName: \"kubernetes.io/projected/7ceebf00-f0b1-41b4-9ab2-8476d9038de1-kube-api-access-8tt2t\") pod \"7ceebf00-f0b1-41b4-9ab2-8476d9038de1\" (UID: \"7ceebf00-f0b1-41b4-9ab2-8476d9038de1\") " Oct 29 23:24:03.320679 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:03.320649 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ceebf00-f0b1-41b4-9ab2-8476d9038de1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ceebf00-f0b1-41b4-9ab2-8476d9038de1-maas-api-openshift-ca\") pod \"7ceebf00-f0b1-41b4-9ab2-8476d9038de1\" (UID: \"7ceebf00-f0b1-41b4-9ab2-8476d9038de1\") " Oct 29 23:24:03.320864 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:03.320833 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7ceebf00-f0b1-41b4-9ab2-8476d9038de1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ceebf00-f0b1-41b4-9ab2-8476d9038de1-maas-api-xks-ca\") pod \"7ceebf00-f0b1-41b4-9ab2-8476d9038de1\" (UID: \"7ceebf00-f0b1-41b4-9ab2-8476d9038de1\") " Oct 29 23:24:03.321105 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:03.321078 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ceebf00-f0b1-41b4-9ab2-8476d9038de1-maas-api-openshift-ca" pod "7ceebf00-f0b1-41b4-9ab2-8476d9038de1" (UID: "7ceebf00-f0b1-41b4-9ab2-8476d9038de1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:24:03.321586 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:03.321555 5057 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ceebf00-f0b1-41b4-9ab2-8476d9038de1-maas-api-openshift-ca\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:24:03.323154 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:03.323126 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ceebf00-f0b1-41b4-9ab2-8476d9038de1-kube-api-access-8tt2t" pod "7ceebf00-f0b1-41b4-9ab2-8476d9038de1" (UID: "7ceebf00-f0b1-41b4-9ab2-8476d9038de1"). InnerVolumeSpecName "kube-api-access-8tt2t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:24:03.323556 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:03.323531 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ceebf00-f0b1-41b4-9ab2-8476d9038de1-maas-api-xks-ca" pod "7ceebf00-f0b1-41b4-9ab2-8476d9038de1" (UID: "7ceebf00-f0b1-41b4-9ab2-8476d9038de1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:24:03.333073 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:03.333049 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8c139aa-aaf8-4d0f-87b6-17f20c7c258c" path="/var/lib/kubelet/pods/a8c139aa-aaf8-4d0f-87b6-17f20c7c258c/volumes" Oct 29 23:24:03.414019 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:03.413980 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:24:03.424216 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:03.424182 5057 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ceebf00-f0b1-41b4-9ab2-8476d9038de1-maas-api-xks-ca\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:24:03.424447 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:03.424282 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8tt2t\" (UniqueName: \"kubernetes.io/projected/7ceebf00-f0b1-41b4-9ab2-8476d9038de1-kube-api-access-8tt2t\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:24:03.974741 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:03.974710 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9e7cc9-x27bf" event={"ID":"7ceebf00-f0b1-41b4-9ab2-8476d9038de1","Type":"ContainerDied","Data":"db6674c781e61bdb287138b94ef4258746cf6a120472e5799677d8947eee6e57"} Oct 29 23:24:03.974741 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:03.974735 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9e7cc9-x27bf" Oct 29 23:24:03.974955 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:03.974758 5057 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="db6674c781e61bdb287138b94ef4258746cf6a120472e5799677d8947eee6e57" Oct 29 23:24:04.019769 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:04.019727 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-462fb2-6999659b56-gh9vv" podUID="d5c09a5d-ceed-4746-a4e7-31714b345610" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:24:05.013138 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:05.013099 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:24:06.622217 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:06.622180 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:24:08.089905 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:08.089870 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:24:09.019185 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:09.019147 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v2tfm"] Oct 29 23:24:09.758889 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:09.758852 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:24:10.666366 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:10.666147 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-9e7cc9-784694f548-nrc56"] Oct 29 23:24:10.804372 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:10.802964 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:24:10.915944 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:10.915894 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:24:12.165132 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:12.165092 5057 generic.go:356] "Generic (PLEG): container finished" podID="d5c09a5d-ceed-4746-a4e7-31714b345610" containerID="342ced5cabd04c31d69529dc652f207c19c8dec0fcc7db5ef94b46a48fc3084b" exitCode=0 Oct 29 23:24:12.165606 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:12.165173 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-462fb2-6999659b56-gh9vv" event={"ID":"d5c09a5d-ceed-4746-a4e7-31714b345610","Type":"ContainerDied","Data":"342ced5cabd04c31d69529dc652f207c19c8dec0fcc7db5ef94b46a48fc3084b"} Oct 29 23:24:12.165606 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:12.165272 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-462fb2-6999659b56-gh9vv" event={"ID":"d5c09a5d-ceed-4746-a4e7-31714b345610","Type":"ContainerDied","Data":"80e2db92fd507774dbba858b9e3d1d5cf00c2675dfdeb31779fa6d0074b98d75"} Oct 29 23:24:12.165606 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:12.165295 5057 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="80e2db92fd507774dbba858b9e3d1d5cf00c2675dfdeb31779fa6d0074b98d75" Oct 29 23:24:12.189209 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:12.189144 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-462fb2-6999659b56-gh9vv" Oct 29 23:24:12.197112 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:12.197072 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:24:12.267214 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:12.267169 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5c09a5d-ceed-4746-a4e7-31714b345610-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5c09a5d-ceed-4746-a4e7-31714b345610-plugins-config-volume\") pod \"d5c09a5d-ceed-4746-a4e7-31714b345610\" (UID: \"d5c09a5d-ceed-4746-a4e7-31714b345610\") " Oct 29 23:24:12.267872 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:12.267843 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5c09a5d-ceed-4746-a4e7-31714b345610-kube-api-access-g9zbd\" (UniqueName: \"kubernetes.io/projected/d5c09a5d-ceed-4746-a4e7-31714b345610-kube-api-access-g9zbd\") pod \"d5c09a5d-ceed-4746-a4e7-31714b345610\" (UID: \"d5c09a5d-ceed-4746-a4e7-31714b345610\") " Oct 29 23:24:12.297490 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:12.297452 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5c09a5d-ceed-4746-a4e7-31714b345610-plugins-config-volume" pod "d5c09a5d-ceed-4746-a4e7-31714b345610" (UID: "d5c09a5d-ceed-4746-a4e7-31714b345610"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:24:12.303862 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:12.303829 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5c09a5d-ceed-4746-a4e7-31714b345610-kube-api-access-g9zbd" pod "d5c09a5d-ceed-4746-a4e7-31714b345610" (UID: "d5c09a5d-ceed-4746-a4e7-31714b345610"). InnerVolumeSpecName "kube-api-access-g9zbd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:24:12.379519 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:12.379487 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g9zbd\" (UniqueName: \"kubernetes.io/projected/d5c09a5d-ceed-4746-a4e7-31714b345610-kube-api-access-g9zbd\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:24:12.379719 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:12.379617 5057 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5c09a5d-ceed-4746-a4e7-31714b345610-plugins-config-volume\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:24:13.175699 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:13.175666 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-462fb2-6999659b56-gh9vv" Oct 29 23:24:13.229934 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:13.229901 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-462fb2-6999659b56-gh9vv"] Oct 29 23:24:13.235550 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:13.235522 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-462fb2-6999659b56-gh9vv"] Oct 29 23:24:13.331719 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:13.331689 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5c09a5d-ceed-4746-a4e7-31714b345610" path="/var/lib/kubelet/pods/d5c09a5d-ceed-4746-a4e7-31714b345610/volumes" Oct 29 23:24:13.812723 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:13.812689 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:24:15.420846 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:15.420809 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:24:15.667024 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:15.666980 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-9e7cc9-784694f548-nrc56" podUID="238bb923-ad70-47a6-8e04-2023e2042006" containerName="payload-processing" containerID="cri-o://2c10d09dd3853f21965bee2efd8e9d4a0c2e38c24f0ffde0d1bc8e91564346f1" gracePeriod=25 Oct 29 23:24:17.019932 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:17.019857 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:24:18.553922 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:18.553878 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:24:19.030966 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:24:19.030930 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d2dnn" containerName="extractor" Oct 29 23:24:20.214273 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:20.214230 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-9e7cc9-784694f548-nrc56" podUID="238bb923-ad70-47a6-8e04-2023e2042006" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:24:20.289958 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:20.289923 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:24:21.620190 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:21.620147 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:24:22.541922 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:22.541886 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:24:22.682629 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:22.682594 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:24:22.764403 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:22.764367 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-462fb2-tlzsg"] Oct 29 23:24:22.775631 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:22.775560 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-462fb2-tlzsg"] Oct 29 23:24:22.939741 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:22.939621 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:24:23.268015 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:23.267980 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:24:23.333422 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:23.333387 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5e27ce5-3b2b-41d1-ac96-e82780ff1c4e" path="/var/lib/kubelet/pods/d5e27ce5-3b2b-41d1-ac96-e82780ff1c4e/volumes" Oct 29 23:24:24.590828 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:24.590789 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:24:25.983715 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:25.983682 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-b2b71e/test-model-b2b71e-kserve-6cf7d6cbd5-cmw67"] Oct 29 23:24:26.026972 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:26.026938 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:24:26.190100 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:26.190068 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:24:26.319683 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:26.319600 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1056cc-djplr"] Oct 29 23:24:26.321197 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:26.321167 5057 cpu_manager.go:414] "containerMap: removing container" podUID="7ceebf00-f0b1-41b4-9ab2-8476d9038de1" containerName="revoke-keys" Oct 29 23:24:26.321388 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:26.321372 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ceebf00-f0b1-41b4-9ab2-8476d9038de1" containerName="revoke-keys" Oct 29 23:24:26.321525 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:26.321513 5057 cpu_manager.go:414] "containerMap: removing container" podUID="d5c09a5d-ceed-4746-a4e7-31714b345610" containerName="payload-processing" Oct 29 23:24:26.321615 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:26.321604 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5c09a5d-ceed-4746-a4e7-31714b345610" containerName="payload-processing" Oct 29 23:24:26.321981 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:26.321962 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ceebf00-f0b1-41b4-9ab2-8476d9038de1" containerName="revoke-keys" Oct 29 23:24:26.322111 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:26.322100 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5c09a5d-ceed-4746-a4e7-31714b345610" containerName="payload-processing" Oct 29 23:24:26.336730 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:26.336273 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1056cc-djplr" Oct 29 23:24:26.343571 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:26.343533 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1056cc-djplr"] Oct 29 23:24:26.354865 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:26.354182 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:24:26.354865 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:26.354205 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dmx57\"" Oct 29 23:24:26.357803 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:26.357774 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zb4q2\" (UniqueName: \"kubernetes.io/projected/cef7c673-4967-458a-8e9d-267a288ff0c4-kube-api-access-zb4q2\") pod \"maas-api-revoke-keys-e2e-inf-a-1056cc-djplr\" (UID: \"cef7c673-4967-458a-8e9d-267a288ff0c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1056cc-djplr" Oct 29 23:24:26.357950 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:26.357843 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cef7c673-4967-458a-8e9d-267a288ff0c4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-1056cc-djplr\" (UID: \"cef7c673-4967-458a-8e9d-267a288ff0c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1056cc-djplr" Oct 29 23:24:26.357950 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:26.357892 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cef7c673-4967-458a-8e9d-267a288ff0c4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-1056cc-djplr\" (UID: \"cef7c673-4967-458a-8e9d-267a288ff0c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1056cc-djplr" Oct 29 23:24:26.460433 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:26.460392 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zb4q2\" (UniqueName: \"kubernetes.io/projected/cef7c673-4967-458a-8e9d-267a288ff0c4-kube-api-access-zb4q2\") pod \"maas-api-revoke-keys-e2e-inf-a-1056cc-djplr\" (UID: \"cef7c673-4967-458a-8e9d-267a288ff0c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1056cc-djplr" Oct 29 23:24:26.460642 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:26.460462 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cef7c673-4967-458a-8e9d-267a288ff0c4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-1056cc-djplr\" (UID: \"cef7c673-4967-458a-8e9d-267a288ff0c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1056cc-djplr" Oct 29 23:24:26.460642 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:26.460491 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cef7c673-4967-458a-8e9d-267a288ff0c4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-1056cc-djplr\" (UID: \"cef7c673-4967-458a-8e9d-267a288ff0c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1056cc-djplr" Oct 29 23:24:26.461204 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:26.461178 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cef7c673-4967-458a-8e9d-267a288ff0c4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-1056cc-djplr\" (UID: \"cef7c673-4967-458a-8e9d-267a288ff0c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1056cc-djplr" Oct 29 23:24:26.463106 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:26.463078 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cef7c673-4967-458a-8e9d-267a288ff0c4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-1056cc-djplr\" (UID: \"cef7c673-4967-458a-8e9d-267a288ff0c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1056cc-djplr" Oct 29 23:24:26.486589 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:26.486561 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zb4q2\" (UniqueName: \"kubernetes.io/projected/cef7c673-4967-458a-8e9d-267a288ff0c4-kube-api-access-zb4q2\") pod \"maas-api-revoke-keys-e2e-inf-a-1056cc-djplr\" (UID: \"cef7c673-4967-458a-8e9d-267a288ff0c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1056cc-djplr" Oct 29 23:24:26.686169 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:26.686077 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1056cc-djplr" Oct 29 23:24:27.016500 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:27.016467 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1056cc-djplr"] Oct 29 23:24:27.334804 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:27.334719 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1056cc-djplr" event={"ID":"cef7c673-4967-458a-8e9d-267a288ff0c4","Type":"ContainerStarted","Data":"aaccd9b688fa867f55a70170e3c9b0b8364aa967154f168d1cc53812d6ac6871"} Oct 29 23:24:28.107324 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:28.107283 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:24:28.341419 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:28.341377 5057 generic.go:356] "Generic (PLEG): container finished" podID="cef7c673-4967-458a-8e9d-267a288ff0c4" containerID="66bb09abfff0bfd4ff98b4b202dd35234eb6d192448c9b84d205e5bee42ba498" exitCode=0 Oct 29 23:24:28.341574 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:28.341438 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1056cc-djplr" event={"ID":"cef7c673-4967-458a-8e9d-267a288ff0c4","Type":"ContainerDied","Data":"66bb09abfff0bfd4ff98b4b202dd35234eb6d192448c9b84d205e5bee42ba498"} Oct 29 23:24:29.570088 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:29.570060 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1056cc-djplr" Oct 29 23:24:29.629855 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:29.629782 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cef7c673-4967-458a-8e9d-267a288ff0c4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cef7c673-4967-458a-8e9d-267a288ff0c4-maas-api-openshift-ca\") pod \"cef7c673-4967-458a-8e9d-267a288ff0c4\" (UID: \"cef7c673-4967-458a-8e9d-267a288ff0c4\") " Oct 29 23:24:29.630056 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:29.629906 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cef7c673-4967-458a-8e9d-267a288ff0c4-kube-api-access-zb4q2\" (UniqueName: \"kubernetes.io/projected/cef7c673-4967-458a-8e9d-267a288ff0c4-kube-api-access-zb4q2\") pod \"cef7c673-4967-458a-8e9d-267a288ff0c4\" (UID: \"cef7c673-4967-458a-8e9d-267a288ff0c4\") " Oct 29 23:24:29.630121 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:29.630079 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cef7c673-4967-458a-8e9d-267a288ff0c4-maas-api-openshift-ca" pod "cef7c673-4967-458a-8e9d-267a288ff0c4" (UID: "cef7c673-4967-458a-8e9d-267a288ff0c4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:24:29.630221 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:29.630200 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cef7c673-4967-458a-8e9d-267a288ff0c4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cef7c673-4967-458a-8e9d-267a288ff0c4-maas-api-xks-ca\") pod \"cef7c673-4967-458a-8e9d-267a288ff0c4\" (UID: \"cef7c673-4967-458a-8e9d-267a288ff0c4\") " Oct 29 23:24:29.630834 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:29.630812 5057 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cef7c673-4967-458a-8e9d-267a288ff0c4-maas-api-openshift-ca\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:24:29.632164 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:29.632137 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cef7c673-4967-458a-8e9d-267a288ff0c4-maas-api-xks-ca" pod "cef7c673-4967-458a-8e9d-267a288ff0c4" (UID: "cef7c673-4967-458a-8e9d-267a288ff0c4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:24:29.632282 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:29.632186 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cef7c673-4967-458a-8e9d-267a288ff0c4-kube-api-access-zb4q2" pod "cef7c673-4967-458a-8e9d-267a288ff0c4" (UID: "cef7c673-4967-458a-8e9d-267a288ff0c4"). InnerVolumeSpecName "kube-api-access-zb4q2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:24:29.732946 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:29.732872 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zb4q2\" (UniqueName: \"kubernetes.io/projected/cef7c673-4967-458a-8e9d-267a288ff0c4-kube-api-access-zb4q2\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:24:29.732946 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:29.732905 5057 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cef7c673-4967-458a-8e9d-267a288ff0c4-maas-api-xks-ca\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:24:29.749114 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:29.749085 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:24:29.923097 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:29.923012 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:24:30.034636 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:30.034598 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:24:30.222394 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:30.219819 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-9e7cc9-784694f548-nrc56" podUID="238bb923-ad70-47a6-8e04-2023e2042006" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:24:30.286069 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:30.285967 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:24:30.506407 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:30.500166 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1056cc-djplr" event={"ID":"cef7c673-4967-458a-8e9d-267a288ff0c4","Type":"ContainerDied","Data":"aaccd9b688fa867f55a70170e3c9b0b8364aa967154f168d1cc53812d6ac6871"} Oct 29 23:24:30.506407 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:30.500241 5057 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aaccd9b688fa867f55a70170e3c9b0b8364aa967154f168d1cc53812d6ac6871" Oct 29 23:24:30.506407 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:30.500322 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1056cc-djplr" Oct 29 23:24:30.610797 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:24:30.610765 5057 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcef7c673_4967_458a_8e9d_267a288ff0c4.slice/crio-aaccd9b688fa867f55a70170e3c9b0b8364aa967154f168d1cc53812d6ac6871\": RecentStats: unable to find data in memory cache]" Oct 29 23:24:30.862973 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:30.862892 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f4ce70-ql59q"] Oct 29 23:24:30.864033 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:30.864013 5057 cpu_manager.go:414] "containerMap: removing container" podUID="cef7c673-4967-458a-8e9d-267a288ff0c4" containerName="revoke-keys" Oct 29 23:24:30.864086 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:30.864034 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cef7c673-4967-458a-8e9d-267a288ff0c4" containerName="revoke-keys" Oct 29 23:24:30.864199 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:30.864189 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="cef7c673-4967-458a-8e9d-267a288ff0c4" containerName="revoke-keys" Oct 29 23:24:30.868742 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:30.868722 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f4ce70-ql59q" Oct 29 23:24:30.885677 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:30.885649 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:24:30.885814 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:30.885684 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dmx57\"" Oct 29 23:24:30.891793 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:30.891768 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f4ce70-ql59q"] Oct 29 23:24:30.980037 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:30.980002 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e8a5db2-c4f1-42de-ab4c-344a862ecec3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-f4ce70-ql59q\" (UID: \"3e8a5db2-c4f1-42de-ab4c-344a862ecec3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f4ce70-ql59q" Oct 29 23:24:30.980279 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:30.980256 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ksqwl\" (UniqueName: \"kubernetes.io/projected/3e8a5db2-c4f1-42de-ab4c-344a862ecec3-kube-api-access-ksqwl\") pod \"maas-api-revoke-keys-e2e-inf-b-f4ce70-ql59q\" (UID: \"3e8a5db2-c4f1-42de-ab4c-344a862ecec3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f4ce70-ql59q" Oct 29 23:24:30.980381 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:30.980304 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e8a5db2-c4f1-42de-ab4c-344a862ecec3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-f4ce70-ql59q\" (UID: \"3e8a5db2-c4f1-42de-ab4c-344a862ecec3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f4ce70-ql59q" Oct 29 23:24:31.081723 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:31.081690 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e8a5db2-c4f1-42de-ab4c-344a862ecec3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-f4ce70-ql59q\" (UID: \"3e8a5db2-c4f1-42de-ab4c-344a862ecec3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f4ce70-ql59q" Oct 29 23:24:31.081902 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:31.081859 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ksqwl\" (UniqueName: \"kubernetes.io/projected/3e8a5db2-c4f1-42de-ab4c-344a862ecec3-kube-api-access-ksqwl\") pod \"maas-api-revoke-keys-e2e-inf-b-f4ce70-ql59q\" (UID: \"3e8a5db2-c4f1-42de-ab4c-344a862ecec3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f4ce70-ql59q" Oct 29 23:24:31.081902 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:31.081889 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e8a5db2-c4f1-42de-ab4c-344a862ecec3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-f4ce70-ql59q\" (UID: \"3e8a5db2-c4f1-42de-ab4c-344a862ecec3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f4ce70-ql59q" Oct 29 23:24:31.082671 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:31.082643 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e8a5db2-c4f1-42de-ab4c-344a862ecec3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-f4ce70-ql59q\" (UID: \"3e8a5db2-c4f1-42de-ab4c-344a862ecec3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f4ce70-ql59q" Oct 29 23:24:31.084241 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:31.084215 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e8a5db2-c4f1-42de-ab4c-344a862ecec3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-f4ce70-ql59q\" (UID: \"3e8a5db2-c4f1-42de-ab4c-344a862ecec3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f4ce70-ql59q" Oct 29 23:24:31.105057 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:31.105027 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ksqwl\" (UniqueName: \"kubernetes.io/projected/3e8a5db2-c4f1-42de-ab4c-344a862ecec3-kube-api-access-ksqwl\") pod \"maas-api-revoke-keys-e2e-inf-b-f4ce70-ql59q\" (UID: \"3e8a5db2-c4f1-42de-ab4c-344a862ecec3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f4ce70-ql59q" Oct 29 23:24:31.185785 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:31.185705 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f4ce70-ql59q" Oct 29 23:24:31.505832 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:31.505802 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f4ce70-ql59q"] Oct 29 23:24:31.583935 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:31.583903 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:24:32.527553 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:32.527522 5057 generic.go:356] "Generic (PLEG): container finished" podID="3e8a5db2-c4f1-42de-ab4c-344a862ecec3" containerID="c78d29d3d8e8b0093bbb73cd61b996e54910b45024427dc93a98bbc0f44a71a8" exitCode=0 Oct 29 23:24:32.527975 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:32.527602 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f4ce70-ql59q" event={"ID":"3e8a5db2-c4f1-42de-ab4c-344a862ecec3","Type":"ContainerDied","Data":"c78d29d3d8e8b0093bbb73cd61b996e54910b45024427dc93a98bbc0f44a71a8"} Oct 29 23:24:32.527975 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:32.527662 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f4ce70-ql59q" event={"ID":"3e8a5db2-c4f1-42de-ab4c-344a862ecec3","Type":"ContainerStarted","Data":"902c49fc998bc0392bd909a09fcc1698eca21859354731d986fb7aa1be655458"} Oct 29 23:24:33.321263 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:33.321218 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:24:33.759297 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:33.759273 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f4ce70-ql59q" Oct 29 23:24:33.941649 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:33.941555 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e8a5db2-c4f1-42de-ab4c-344a862ecec3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e8a5db2-c4f1-42de-ab4c-344a862ecec3-maas-api-openshift-ca\") pod \"3e8a5db2-c4f1-42de-ab4c-344a862ecec3\" (UID: \"3e8a5db2-c4f1-42de-ab4c-344a862ecec3\") " Oct 29 23:24:33.941649 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:33.941645 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e8a5db2-c4f1-42de-ab4c-344a862ecec3-kube-api-access-ksqwl\" (UniqueName: \"kubernetes.io/projected/3e8a5db2-c4f1-42de-ab4c-344a862ecec3-kube-api-access-ksqwl\") pod \"3e8a5db2-c4f1-42de-ab4c-344a862ecec3\" (UID: \"3e8a5db2-c4f1-42de-ab4c-344a862ecec3\") " Oct 29 23:24:33.941908 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:33.941740 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3e8a5db2-c4f1-42de-ab4c-344a862ecec3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e8a5db2-c4f1-42de-ab4c-344a862ecec3-maas-api-xks-ca\") pod \"3e8a5db2-c4f1-42de-ab4c-344a862ecec3\" (UID: \"3e8a5db2-c4f1-42de-ab4c-344a862ecec3\") " Oct 29 23:24:33.942012 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:33.941987 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e8a5db2-c4f1-42de-ab4c-344a862ecec3-maas-api-openshift-ca" pod "3e8a5db2-c4f1-42de-ab4c-344a862ecec3" (UID: "3e8a5db2-c4f1-42de-ab4c-344a862ecec3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:24:33.942571 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:33.942552 5057 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e8a5db2-c4f1-42de-ab4c-344a862ecec3-maas-api-openshift-ca\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:24:33.943769 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:33.943735 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3e8a5db2-c4f1-42de-ab4c-344a862ecec3-maas-api-xks-ca" pod "3e8a5db2-c4f1-42de-ab4c-344a862ecec3" (UID: "3e8a5db2-c4f1-42de-ab4c-344a862ecec3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:24:33.943858 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:33.943775 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e8a5db2-c4f1-42de-ab4c-344a862ecec3-kube-api-access-ksqwl" pod "3e8a5db2-c4f1-42de-ab4c-344a862ecec3" (UID: "3e8a5db2-c4f1-42de-ab4c-344a862ecec3"). InnerVolumeSpecName "kube-api-access-ksqwl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:24:34.045191 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:34.045153 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ksqwl\" (UniqueName: \"kubernetes.io/projected/3e8a5db2-c4f1-42de-ab4c-344a862ecec3-kube-api-access-ksqwl\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:24:34.045191 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:34.045194 5057 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e8a5db2-c4f1-42de-ab4c-344a862ecec3-maas-api-xks-ca\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:24:34.547997 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:34.547965 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f4ce70-ql59q" Oct 29 23:24:34.548173 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:34.548006 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f4ce70-ql59q" event={"ID":"3e8a5db2-c4f1-42de-ab4c-344a862ecec3","Type":"ContainerDied","Data":"902c49fc998bc0392bd909a09fcc1698eca21859354731d986fb7aa1be655458"} Oct 29 23:24:34.548173 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:34.548060 5057 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="902c49fc998bc0392bd909a09fcc1698eca21859354731d986fb7aa1be655458" Oct 29 23:24:34.824916 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:34.824816 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:24:35.028621 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:35.028589 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ab5c06a-lkxql"] Oct 29 23:24:35.029665 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:35.029639 5057 cpu_manager.go:414] "containerMap: removing container" podUID="3e8a5db2-c4f1-42de-ab4c-344a862ecec3" containerName="revoke-keys" Oct 29 23:24:35.029665 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:35.029658 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e8a5db2-c4f1-42de-ab4c-344a862ecec3" containerName="revoke-keys" Oct 29 23:24:35.029907 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:35.029894 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e8a5db2-c4f1-42de-ab4c-344a862ecec3" containerName="revoke-keys" Oct 29 23:24:35.034427 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:35.034410 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ab5c06a-lkxql" Oct 29 23:24:35.049102 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:35.049077 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dmx57\"" Oct 29 23:24:35.049405 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:35.049384 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:24:35.058321 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:35.058295 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ab5c06a-lkxql"] Oct 29 23:24:35.096236 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:35.096156 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79428e20-0cad-4a0a-8529-d1677be60c68-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4ab5c06a-lkxql\" (UID: \"79428e20-0cad-4a0a-8529-d1677be60c68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ab5c06a-lkxql" Oct 29 23:24:35.096407 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:35.096310 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mx4t4\" (UniqueName: \"kubernetes.io/projected/79428e20-0cad-4a0a-8529-d1677be60c68-kube-api-access-mx4t4\") pod \"maas-api-revoke-sub-4ab5c06a-lkxql\" (UID: \"79428e20-0cad-4a0a-8529-d1677be60c68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ab5c06a-lkxql" Oct 29 23:24:35.096407 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:35.096356 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79428e20-0cad-4a0a-8529-d1677be60c68-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4ab5c06a-lkxql\" (UID: \"79428e20-0cad-4a0a-8529-d1677be60c68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ab5c06a-lkxql" Oct 29 23:24:35.198748 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:35.198702 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79428e20-0cad-4a0a-8529-d1677be60c68-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4ab5c06a-lkxql\" (UID: \"79428e20-0cad-4a0a-8529-d1677be60c68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ab5c06a-lkxql" Oct 29 23:24:35.198917 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:35.198885 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mx4t4\" (UniqueName: \"kubernetes.io/projected/79428e20-0cad-4a0a-8529-d1677be60c68-kube-api-access-mx4t4\") pod \"maas-api-revoke-sub-4ab5c06a-lkxql\" (UID: \"79428e20-0cad-4a0a-8529-d1677be60c68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ab5c06a-lkxql" Oct 29 23:24:35.199003 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:35.198938 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79428e20-0cad-4a0a-8529-d1677be60c68-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4ab5c06a-lkxql\" (UID: \"79428e20-0cad-4a0a-8529-d1677be60c68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ab5c06a-lkxql" Oct 29 23:24:35.199445 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:35.199417 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79428e20-0cad-4a0a-8529-d1677be60c68-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4ab5c06a-lkxql\" (UID: \"79428e20-0cad-4a0a-8529-d1677be60c68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ab5c06a-lkxql" Oct 29 23:24:35.201426 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:35.201400 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79428e20-0cad-4a0a-8529-d1677be60c68-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4ab5c06a-lkxql\" (UID: \"79428e20-0cad-4a0a-8529-d1677be60c68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ab5c06a-lkxql" Oct 29 23:24:35.221675 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:35.221649 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mx4t4\" (UniqueName: \"kubernetes.io/projected/79428e20-0cad-4a0a-8529-d1677be60c68-kube-api-access-mx4t4\") pod \"maas-api-revoke-sub-4ab5c06a-lkxql\" (UID: \"79428e20-0cad-4a0a-8529-d1677be60c68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ab5c06a-lkxql" Oct 29 23:24:35.366057 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:35.365975 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ab5c06a-lkxql" Oct 29 23:24:35.689267 ip-10-0-140-74 kubenswrapper[5057]: W1029 23:24:35.689227 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79428e20_0cad_4a0a_8529_d1677be60c68.slice/crio-bb197f4b865e98eb0f9349c3a5d14978d20b2318bf0287f26b4ac9143d77f373 WatchSource:0}: Error finding container bb197f4b865e98eb0f9349c3a5d14978d20b2318bf0287f26b4ac9143d77f373: Status 404 returned error can't find the container with id bb197f4b865e98eb0f9349c3a5d14978d20b2318bf0287f26b4ac9143d77f373 Oct 29 23:24:35.689787 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:35.689764 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ab5c06a-lkxql"] Oct 29 23:24:35.732893 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:35.732853 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:24:35.989590 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:35.989563 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-9e7cc9-784694f548-nrc56" Oct 29 23:24:36.012698 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:36.012667 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/238bb923-ad70-47a6-8e04-2023e2042006-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/238bb923-ad70-47a6-8e04-2023e2042006-plugins-config-volume\") pod \"238bb923-ad70-47a6-8e04-2023e2042006\" (UID: \"238bb923-ad70-47a6-8e04-2023e2042006\") " Oct 29 23:24:36.012888 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:36.012872 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/238bb923-ad70-47a6-8e04-2023e2042006-kube-api-access-csllq\" (UniqueName: \"kubernetes.io/projected/238bb923-ad70-47a6-8e04-2023e2042006-kube-api-access-csllq\") pod \"238bb923-ad70-47a6-8e04-2023e2042006\" (UID: \"238bb923-ad70-47a6-8e04-2023e2042006\") " Oct 29 23:24:36.013025 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:36.013005 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/238bb923-ad70-47a6-8e04-2023e2042006-plugins-config-volume" pod "238bb923-ad70-47a6-8e04-2023e2042006" (UID: "238bb923-ad70-47a6-8e04-2023e2042006"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:24:36.013757 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:36.013734 5057 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/238bb923-ad70-47a6-8e04-2023e2042006-plugins-config-volume\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:24:36.014909 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:36.014884 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/238bb923-ad70-47a6-8e04-2023e2042006-kube-api-access-csllq" pod "238bb923-ad70-47a6-8e04-2023e2042006" (UID: "238bb923-ad70-47a6-8e04-2023e2042006"). InnerVolumeSpecName "kube-api-access-csllq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:24:36.115552 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:36.115517 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-csllq\" (UniqueName: \"kubernetes.io/projected/238bb923-ad70-47a6-8e04-2023e2042006-kube-api-access-csllq\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:24:36.248350 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:36.248297 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:24:36.653407 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:36.649726 5057 generic.go:356] "Generic (PLEG): container finished" podID="238bb923-ad70-47a6-8e04-2023e2042006" containerID="2c10d09dd3853f21965bee2efd8e9d4a0c2e38c24f0ffde0d1bc8e91564346f1" exitCode=0 Oct 29 23:24:36.653407 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:36.649840 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-9e7cc9-784694f548-nrc56" event={"ID":"238bb923-ad70-47a6-8e04-2023e2042006","Type":"ContainerDied","Data":"2c10d09dd3853f21965bee2efd8e9d4a0c2e38c24f0ffde0d1bc8e91564346f1"} Oct 29 23:24:36.653407 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:36.649892 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-9e7cc9-784694f548-nrc56" event={"ID":"238bb923-ad70-47a6-8e04-2023e2042006","Type":"ContainerDied","Data":"4be2436a9384811d63659b2a73954c0a644c174d3d74449e48017095abfbe69e"} Oct 29 23:24:36.653407 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:36.649919 5057 scope.go:122] "RemoveContainer" containerID="2c10d09dd3853f21965bee2efd8e9d4a0c2e38c24f0ffde0d1bc8e91564346f1" Oct 29 23:24:36.653407 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:36.650068 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-9e7cc9-784694f548-nrc56" Oct 29 23:24:36.671925 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:36.668246 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ab5c06a-lkxql" event={"ID":"79428e20-0cad-4a0a-8529-d1677be60c68","Type":"ContainerStarted","Data":"5f642f6da1321340d8836d25584c21bac0d601e4535e3cfe4c49802b7593c203"} Oct 29 23:24:36.671925 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:36.668301 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ab5c06a-lkxql" event={"ID":"79428e20-0cad-4a0a-8529-d1677be60c68","Type":"ContainerStarted","Data":"bb197f4b865e98eb0f9349c3a5d14978d20b2318bf0287f26b4ac9143d77f373"} Oct 29 23:24:36.685253 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:36.685231 5057 scope.go:122] "RemoveContainer" containerID="2c10d09dd3853f21965bee2efd8e9d4a0c2e38c24f0ffde0d1bc8e91564346f1" Oct 29 23:24:36.685737 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:24:36.685716 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c10d09dd3853f21965bee2efd8e9d4a0c2e38c24f0ffde0d1bc8e91564346f1\": container with ID starting with 2c10d09dd3853f21965bee2efd8e9d4a0c2e38c24f0ffde0d1bc8e91564346f1 not found: ID does not exist" containerID="2c10d09dd3853f21965bee2efd8e9d4a0c2e38c24f0ffde0d1bc8e91564346f1" Oct 29 23:24:36.685831 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:36.685745 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c10d09dd3853f21965bee2efd8e9d4a0c2e38c24f0ffde0d1bc8e91564346f1"} err="failed to get container status \"2c10d09dd3853f21965bee2efd8e9d4a0c2e38c24f0ffde0d1bc8e91564346f1\": rpc error: code = NotFound desc = could not find container \"2c10d09dd3853f21965bee2efd8e9d4a0c2e38c24f0ffde0d1bc8e91564346f1\": container with ID starting with 2c10d09dd3853f21965bee2efd8e9d4a0c2e38c24f0ffde0d1bc8e91564346f1 not found: ID does not exist" Oct 29 23:24:36.709399 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:36.709372 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-9e7cc9-784694f548-nrc56"] Oct 29 23:24:36.715104 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:36.715078 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-9e7cc9-784694f548-nrc56"] Oct 29 23:24:36.750227 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:36.750181 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ab5c06a-lkxql" podStartSLOduration=1.750167728 podStartE2EDuration="1.750167728s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:24:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:24:36.745419392 +0000 UTC m=+3057.528632814" watchObservedRunningTime="2026-10-29 23:24:36.750167728 +0000 UTC m=+3057.533381153" Oct 29 23:24:37.020035 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:37.019997 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:24:37.130500 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:37.130464 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:24:37.243198 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:37.243158 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:24:37.334691 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:37.334616 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="238bb923-ad70-47a6-8e04-2023e2042006" path="/var/lib/kubelet/pods/238bb923-ad70-47a6-8e04-2023e2042006/volumes" Oct 29 23:24:37.385674 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:37.385636 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:24:37.412032 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:37.412006 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9e7cc9-x27bf"] Oct 29 23:24:37.417310 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:37.417282 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9e7cc9-x27bf"] Oct 29 23:24:37.493727 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:37.493689 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:24:37.716356 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:37.710367 5057 generic.go:356] "Generic (PLEG): container finished" podID="79428e20-0cad-4a0a-8529-d1677be60c68" containerID="5f642f6da1321340d8836d25584c21bac0d601e4535e3cfe4c49802b7593c203" exitCode=0 Oct 29 23:24:37.716356 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:37.710445 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ab5c06a-lkxql" event={"ID":"79428e20-0cad-4a0a-8529-d1677be60c68","Type":"ContainerDied","Data":"5f642f6da1321340d8836d25584c21bac0d601e4535e3cfe4c49802b7593c203"} Oct 29 23:24:37.903026 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:37.902991 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:24:38.918009 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:38.917978 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ab5c06a-lkxql" Oct 29 23:24:38.987618 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:38.987553 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79428e20-0cad-4a0a-8529-d1677be60c68-kube-api-access-mx4t4\" (UniqueName: \"kubernetes.io/projected/79428e20-0cad-4a0a-8529-d1677be60c68-kube-api-access-mx4t4\") pod \"79428e20-0cad-4a0a-8529-d1677be60c68\" (UID: \"79428e20-0cad-4a0a-8529-d1677be60c68\") " Oct 29 23:24:38.987816 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:38.987751 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/79428e20-0cad-4a0a-8529-d1677be60c68-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79428e20-0cad-4a0a-8529-d1677be60c68-maas-api-xks-ca\") pod \"79428e20-0cad-4a0a-8529-d1677be60c68\" (UID: \"79428e20-0cad-4a0a-8529-d1677be60c68\") " Oct 29 23:24:38.987816 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:38.987780 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79428e20-0cad-4a0a-8529-d1677be60c68-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79428e20-0cad-4a0a-8529-d1677be60c68-maas-api-openshift-ca\") pod \"79428e20-0cad-4a0a-8529-d1677be60c68\" (UID: \"79428e20-0cad-4a0a-8529-d1677be60c68\") " Oct 29 23:24:38.988264 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:38.988231 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79428e20-0cad-4a0a-8529-d1677be60c68-maas-api-openshift-ca" pod "79428e20-0cad-4a0a-8529-d1677be60c68" (UID: "79428e20-0cad-4a0a-8529-d1677be60c68"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:24:38.988568 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:38.988546 5057 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79428e20-0cad-4a0a-8529-d1677be60c68-maas-api-openshift-ca\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:24:38.990066 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:38.990042 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79428e20-0cad-4a0a-8529-d1677be60c68-kube-api-access-mx4t4" pod "79428e20-0cad-4a0a-8529-d1677be60c68" (UID: "79428e20-0cad-4a0a-8529-d1677be60c68"). InnerVolumeSpecName "kube-api-access-mx4t4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:24:38.990308 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:38.990285 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79428e20-0cad-4a0a-8529-d1677be60c68-maas-api-xks-ca" pod "79428e20-0cad-4a0a-8529-d1677be60c68" (UID: "79428e20-0cad-4a0a-8529-d1677be60c68"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:24:39.090221 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:39.090186 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mx4t4\" (UniqueName: \"kubernetes.io/projected/79428e20-0cad-4a0a-8529-d1677be60c68-kube-api-access-mx4t4\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:24:39.090221 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:39.090218 5057 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79428e20-0cad-4a0a-8529-d1677be60c68-maas-api-xks-ca\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:24:39.353758 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:39.353678 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ceebf00-f0b1-41b4-9ab2-8476d9038de1" path="/var/lib/kubelet/pods/7ceebf00-f0b1-41b4-9ab2-8476d9038de1/volumes" Oct 29 23:24:39.411164 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:39.411129 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:24:39.734805 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:39.734763 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ab5c06a-lkxql" event={"ID":"79428e20-0cad-4a0a-8529-d1677be60c68","Type":"ContainerDied","Data":"bb197f4b865e98eb0f9349c3a5d14978d20b2318bf0287f26b4ac9143d77f373"} Oct 29 23:24:39.734805 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:39.734786 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ab5c06a-lkxql" Oct 29 23:24:39.734805 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:39.734813 5057 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bb197f4b865e98eb0f9349c3a5d14978d20b2318bf0287f26b4ac9143d77f373" Oct 29 23:24:41.016378 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:41.016285 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-b2b71e/test-model-b2b71e-kserve-6cf7d6cbd5-cmw67" podUID="c711b857-12cd-4bad-bec7-29e25e5e810d" containerName="main" containerID="cri-o://c33c4413cf6ca3c33363fd6250b4d33b856a8cb109b23f5d76b10e32fe997722" gracePeriod=45 Oct 29 23:24:41.054898 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:41.054863 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-b2b71e/test-model-b2b71e-kserve-6cf7d6cbd5-cmw67" podUID="c711b857-12cd-4bad-bec7-29e25e5e810d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.218:8000/ready\": dial tcp 10.132.0.218:8000: connect: connection refused" Oct 29 23:24:41.101561 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:41.101520 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:24:41.404876 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:41.404844 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-b2b71e/test-model-b2b71e-kserve-6cf7d6cbd5-cmw67" Oct 29 23:24:41.431018 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:41.430988 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c711b857-12cd-4bad-bec7-29e25e5e810d-home\" (UniqueName: \"kubernetes.io/empty-dir/c711b857-12cd-4bad-bec7-29e25e5e810d-home\") pod \"c711b857-12cd-4bad-bec7-29e25e5e810d\" (UID: \"c711b857-12cd-4bad-bec7-29e25e5e810d\") " Oct 29 23:24:41.431237 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:41.431086 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c711b857-12cd-4bad-bec7-29e25e5e810d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c711b857-12cd-4bad-bec7-29e25e5e810d-dshm\") pod \"c711b857-12cd-4bad-bec7-29e25e5e810d\" (UID: \"c711b857-12cd-4bad-bec7-29e25e5e810d\") " Oct 29 23:24:41.431237 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:41.431114 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c711b857-12cd-4bad-bec7-29e25e5e810d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c711b857-12cd-4bad-bec7-29e25e5e810d-model-cache\") pod \"c711b857-12cd-4bad-bec7-29e25e5e810d\" (UID: \"c711b857-12cd-4bad-bec7-29e25e5e810d\") " Oct 29 23:24:41.431237 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:41.431178 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c711b857-12cd-4bad-bec7-29e25e5e810d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c711b857-12cd-4bad-bec7-29e25e5e810d-tmp-dir\") pod \"c711b857-12cd-4bad-bec7-29e25e5e810d\" (UID: \"c711b857-12cd-4bad-bec7-29e25e5e810d\") " Oct 29 23:24:41.431237 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:41.431218 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c711b857-12cd-4bad-bec7-29e25e5e810d-kube-api-access-rc8gw\" (UniqueName: \"kubernetes.io/projected/c711b857-12cd-4bad-bec7-29e25e5e810d-kube-api-access-rc8gw\") pod \"c711b857-12cd-4bad-bec7-29e25e5e810d\" (UID: \"c711b857-12cd-4bad-bec7-29e25e5e810d\") " Oct 29 23:24:41.431519 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:41.431376 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c711b857-12cd-4bad-bec7-29e25e5e810d-home" pod "c711b857-12cd-4bad-bec7-29e25e5e810d" (UID: "c711b857-12cd-4bad-bec7-29e25e5e810d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:24:41.431519 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:41.431420 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c711b857-12cd-4bad-bec7-29e25e5e810d-tls-certs\" (UniqueName: \"kubernetes.io/secret/c711b857-12cd-4bad-bec7-29e25e5e810d-tls-certs\") pod \"c711b857-12cd-4bad-bec7-29e25e5e810d\" (UID: \"c711b857-12cd-4bad-bec7-29e25e5e810d\") " Oct 29 23:24:41.431634 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:41.431561 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c711b857-12cd-4bad-bec7-29e25e5e810d-model-cache" pod "c711b857-12cd-4bad-bec7-29e25e5e810d" (UID: "c711b857-12cd-4bad-bec7-29e25e5e810d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:24:41.432274 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:41.432250 5057 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c711b857-12cd-4bad-bec7-29e25e5e810d-home\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:24:41.432458 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:41.432278 5057 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c711b857-12cd-4bad-bec7-29e25e5e810d-model-cache\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:24:41.434028 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:41.434005 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c711b857-12cd-4bad-bec7-29e25e5e810d-tmp-dir" pod "c711b857-12cd-4bad-bec7-29e25e5e810d" (UID: "c711b857-12cd-4bad-bec7-29e25e5e810d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:24:41.434819 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:41.434794 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c711b857-12cd-4bad-bec7-29e25e5e810d-dshm" pod "c711b857-12cd-4bad-bec7-29e25e5e810d" (UID: "c711b857-12cd-4bad-bec7-29e25e5e810d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:24:41.435525 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:41.435505 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c711b857-12cd-4bad-bec7-29e25e5e810d-tls-certs" pod "c711b857-12cd-4bad-bec7-29e25e5e810d" (UID: "c711b857-12cd-4bad-bec7-29e25e5e810d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:24:41.437552 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:41.437525 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c711b857-12cd-4bad-bec7-29e25e5e810d-kube-api-access-rc8gw" pod "c711b857-12cd-4bad-bec7-29e25e5e810d" (UID: "c711b857-12cd-4bad-bec7-29e25e5e810d"). InnerVolumeSpecName "kube-api-access-rc8gw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:24:41.486287 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:41.486248 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:24:41.533879 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:41.533843 5057 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c711b857-12cd-4bad-bec7-29e25e5e810d-dshm\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:24:41.533879 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:41.533874 5057 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c711b857-12cd-4bad-bec7-29e25e5e810d-tmp-dir\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:24:41.533879 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:41.533885 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rc8gw\" (UniqueName: \"kubernetes.io/projected/c711b857-12cd-4bad-bec7-29e25e5e810d-kube-api-access-rc8gw\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:24:41.534125 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:41.533895 5057 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c711b857-12cd-4bad-bec7-29e25e5e810d-tls-certs\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:24:41.602366 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:41.599116 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:24:41.757540 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:41.757496 5057 generic.go:356] "Generic (PLEG): container finished" podID="c711b857-12cd-4bad-bec7-29e25e5e810d" containerID="c33c4413cf6ca3c33363fd6250b4d33b856a8cb109b23f5d76b10e32fe997722" exitCode=0 Oct 29 23:24:41.757739 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:41.757541 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-b2b71e/test-model-b2b71e-kserve-6cf7d6cbd5-cmw67" event={"ID":"c711b857-12cd-4bad-bec7-29e25e5e810d","Type":"ContainerDied","Data":"c33c4413cf6ca3c33363fd6250b4d33b856a8cb109b23f5d76b10e32fe997722"} Oct 29 23:24:41.757739 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:41.757562 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-b2b71e/test-model-b2b71e-kserve-6cf7d6cbd5-cmw67" Oct 29 23:24:41.757739 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:41.757589 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-b2b71e/test-model-b2b71e-kserve-6cf7d6cbd5-cmw67" event={"ID":"c711b857-12cd-4bad-bec7-29e25e5e810d","Type":"ContainerDied","Data":"7a90ef8b77a3db367339d5e5215783cf7a14cfcd5e1b3b7a028b6f83722eac4d"} Oct 29 23:24:41.757739 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:41.757614 5057 scope.go:122] "RemoveContainer" containerID="c33c4413cf6ca3c33363fd6250b4d33b856a8cb109b23f5d76b10e32fe997722" Oct 29 23:24:41.777669 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:41.777649 5057 scope.go:122] "RemoveContainer" containerID="c33c4413cf6ca3c33363fd6250b4d33b856a8cb109b23f5d76b10e32fe997722" Oct 29 23:24:41.777937 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:24:41.777916 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c33c4413cf6ca3c33363fd6250b4d33b856a8cb109b23f5d76b10e32fe997722\": container with ID starting with c33c4413cf6ca3c33363fd6250b4d33b856a8cb109b23f5d76b10e32fe997722 not found: ID does not exist" containerID="c33c4413cf6ca3c33363fd6250b4d33b856a8cb109b23f5d76b10e32fe997722" Oct 29 23:24:41.777992 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:41.777946 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c33c4413cf6ca3c33363fd6250b4d33b856a8cb109b23f5d76b10e32fe997722"} err="failed to get container status \"c33c4413cf6ca3c33363fd6250b4d33b856a8cb109b23f5d76b10e32fe997722\": rpc error: code = NotFound desc = could not find container \"c33c4413cf6ca3c33363fd6250b4d33b856a8cb109b23f5d76b10e32fe997722\": container with ID starting with c33c4413cf6ca3c33363fd6250b4d33b856a8cb109b23f5d76b10e32fe997722 not found: ID does not exist" Oct 29 23:24:41.812268 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:41.812241 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-b2b71e/test-model-b2b71e-kserve-6cf7d6cbd5-cmw67"] Oct 29 23:24:41.823302 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:41.823269 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-b2b71e/test-model-b2b71e-kserve-6cf7d6cbd5-cmw67"] Oct 29 23:24:42.829521 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:42.829484 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:24:43.336023 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:43.335986 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c711b857-12cd-4bad-bec7-29e25e5e810d" path="/var/lib/kubelet/pods/c711b857-12cd-4bad-bec7-29e25e5e810d/volumes" Oct 29 23:24:43.567752 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:43.567716 5057 scope.go:122] "RemoveContainer" containerID="f5d5cdc1327016cfc7d43a99858ee2558845fc5ab2094d7b066fdb45db2d5836" Oct 29 23:24:43.579594 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:43.579570 5057 scope.go:122] "RemoveContainer" containerID="342ced5cabd04c31d69529dc652f207c19c8dec0fcc7db5ef94b46a48fc3084b" Oct 29 23:24:44.353003 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:44.352968 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:24:44.484371 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:44.473481 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:24:44.699707 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:44.699675 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:24:45.274700 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:45.274662 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a9ae14-c5qxl"] Oct 29 23:24:45.275722 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:45.275700 5057 cpu_manager.go:414] "containerMap: removing container" podUID="238bb923-ad70-47a6-8e04-2023e2042006" containerName="payload-processing" Oct 29 23:24:45.275722 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:45.275719 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="238bb923-ad70-47a6-8e04-2023e2042006" containerName="payload-processing" Oct 29 23:24:45.275929 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:45.275739 5057 cpu_manager.go:414] "containerMap: removing container" podUID="c711b857-12cd-4bad-bec7-29e25e5e810d" containerName="main" Oct 29 23:24:45.275929 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:45.275746 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c711b857-12cd-4bad-bec7-29e25e5e810d" containerName="main" Oct 29 23:24:45.275929 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:45.275773 5057 cpu_manager.go:414] "containerMap: removing container" podUID="79428e20-0cad-4a0a-8529-d1677be60c68" containerName="revoke-keys" Oct 29 23:24:45.275929 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:45.275779 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79428e20-0cad-4a0a-8529-d1677be60c68" containerName="revoke-keys" Oct 29 23:24:45.276133 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:45.275950 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="238bb923-ad70-47a6-8e04-2023e2042006" containerName="payload-processing" Oct 29 23:24:45.276133 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:45.275959 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="c711b857-12cd-4bad-bec7-29e25e5e810d" containerName="main" Oct 29 23:24:45.276133 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:45.275972 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="79428e20-0cad-4a0a-8529-d1677be60c68" containerName="revoke-keys" Oct 29 23:24:45.282160 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:45.282136 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a9ae14-c5qxl" Oct 29 23:24:45.301004 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:45.300976 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a9ae14-c5qxl"] Oct 29 23:24:45.340342 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:45.340205 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99d91e9c-a637-4834-b840-bb0bc29bc66d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-a9ae14-c5qxl\" (UID: \"99d91e9c-a637-4834-b840-bb0bc29bc66d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a9ae14-c5qxl" Oct 29 23:24:45.340511 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:45.340348 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l746t\" (UniqueName: \"kubernetes.io/projected/99d91e9c-a637-4834-b840-bb0bc29bc66d-kube-api-access-l746t\") pod \"maas-api-revoke-keys-e2e-inf-b-a9ae14-c5qxl\" (UID: \"99d91e9c-a637-4834-b840-bb0bc29bc66d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a9ae14-c5qxl" Oct 29 23:24:45.340566 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:45.340550 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99d91e9c-a637-4834-b840-bb0bc29bc66d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-a9ae14-c5qxl\" (UID: \"99d91e9c-a637-4834-b840-bb0bc29bc66d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a9ae14-c5qxl" Oct 29 23:24:45.357690 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:45.357637 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:24:45.358181 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:45.357737 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dmx57\"" Oct 29 23:24:45.441732 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:45.441700 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99d91e9c-a637-4834-b840-bb0bc29bc66d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-a9ae14-c5qxl\" (UID: \"99d91e9c-a637-4834-b840-bb0bc29bc66d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a9ae14-c5qxl" Oct 29 23:24:45.442146 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:45.442110 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99d91e9c-a637-4834-b840-bb0bc29bc66d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-a9ae14-c5qxl\" (UID: \"99d91e9c-a637-4834-b840-bb0bc29bc66d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a9ae14-c5qxl" Oct 29 23:24:45.442290 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:45.442271 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l746t\" (UniqueName: \"kubernetes.io/projected/99d91e9c-a637-4834-b840-bb0bc29bc66d-kube-api-access-l746t\") pod \"maas-api-revoke-keys-e2e-inf-b-a9ae14-c5qxl\" (UID: \"99d91e9c-a637-4834-b840-bb0bc29bc66d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a9ae14-c5qxl" Oct 29 23:24:45.442724 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:45.442702 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99d91e9c-a637-4834-b840-bb0bc29bc66d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-a9ae14-c5qxl\" (UID: \"99d91e9c-a637-4834-b840-bb0bc29bc66d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a9ae14-c5qxl" Oct 29 23:24:45.444207 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:45.444190 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99d91e9c-a637-4834-b840-bb0bc29bc66d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-a9ae14-c5qxl\" (UID: \"99d91e9c-a637-4834-b840-bb0bc29bc66d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a9ae14-c5qxl" Oct 29 23:24:45.465500 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:45.465475 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l746t\" (UniqueName: \"kubernetes.io/projected/99d91e9c-a637-4834-b840-bb0bc29bc66d-kube-api-access-l746t\") pod \"maas-api-revoke-keys-e2e-inf-b-a9ae14-c5qxl\" (UID: \"99d91e9c-a637-4834-b840-bb0bc29bc66d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a9ae14-c5qxl" Oct 29 23:24:45.600250 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:45.600167 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a9ae14-c5qxl" Oct 29 23:24:45.911818 ip-10-0-140-74 kubenswrapper[5057]: W1029 23:24:45.911785 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99d91e9c_a637_4834_b840_bb0bc29bc66d.slice/crio-95c91dadb17ac96d787bf993af0ed6dee918a7e31341529e58cad090dec440a5 WatchSource:0}: Error finding container 95c91dadb17ac96d787bf993af0ed6dee918a7e31341529e58cad090dec440a5: Status 404 returned error can't find the container with id 95c91dadb17ac96d787bf993af0ed6dee918a7e31341529e58cad090dec440a5 Oct 29 23:24:45.912630 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:45.912600 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a9ae14-c5qxl"] Oct 29 23:24:46.309826 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:46.309759 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:24:46.858959 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:46.858921 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a9ae14-c5qxl" event={"ID":"99d91e9c-a637-4834-b840-bb0bc29bc66d","Type":"ContainerStarted","Data":"e07f3c463f24e027d9e9e362be1ea0eba9b4d43bc226953a3bd4fa2ee94537b4"} Oct 29 23:24:46.859418 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:46.858985 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a9ae14-c5qxl" event={"ID":"99d91e9c-a637-4834-b840-bb0bc29bc66d","Type":"ContainerStarted","Data":"95c91dadb17ac96d787bf993af0ed6dee918a7e31341529e58cad090dec440a5"} Oct 29 23:24:46.903240 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:46.903185 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a9ae14-c5qxl" podStartSLOduration=1.903169585 podStartE2EDuration="1.903169585s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:24:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:24:46.898161889 +0000 UTC m=+3067.681375315" watchObservedRunningTime="2026-10-29 23:24:46.903169585 +0000 UTC m=+3067.686383011" Oct 29 23:24:47.872803 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:47.872761 5057 generic.go:356] "Generic (PLEG): container finished" podID="99d91e9c-a637-4834-b840-bb0bc29bc66d" containerID="e07f3c463f24e027d9e9e362be1ea0eba9b4d43bc226953a3bd4fa2ee94537b4" exitCode=0 Oct 29 23:24:47.873302 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:47.872859 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a9ae14-c5qxl" event={"ID":"99d91e9c-a637-4834-b840-bb0bc29bc66d","Type":"ContainerDied","Data":"e07f3c463f24e027d9e9e362be1ea0eba9b4d43bc226953a3bd4fa2ee94537b4"} Oct 29 23:24:48.015128 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:48.015082 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:24:49.036183 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:24:49.036148 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d2dnn" containerName="extractor" Oct 29 23:24:49.087665 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:49.087626 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a9ae14-c5qxl" Oct 29 23:24:49.234642 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:49.234614 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/99d91e9c-a637-4834-b840-bb0bc29bc66d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99d91e9c-a637-4834-b840-bb0bc29bc66d-maas-api-openshift-ca\") pod \"99d91e9c-a637-4834-b840-bb0bc29bc66d\" (UID: \"99d91e9c-a637-4834-b840-bb0bc29bc66d\") " Oct 29 23:24:49.234821 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:49.234790 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99d91e9c-a637-4834-b840-bb0bc29bc66d-kube-api-access-l746t\" (UniqueName: \"kubernetes.io/projected/99d91e9c-a637-4834-b840-bb0bc29bc66d-kube-api-access-l746t\") pod \"99d91e9c-a637-4834-b840-bb0bc29bc66d\" (UID: \"99d91e9c-a637-4834-b840-bb0bc29bc66d\") " Oct 29 23:24:49.234873 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:49.234863 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/99d91e9c-a637-4834-b840-bb0bc29bc66d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99d91e9c-a637-4834-b840-bb0bc29bc66d-maas-api-xks-ca\") pod \"99d91e9c-a637-4834-b840-bb0bc29bc66d\" (UID: \"99d91e9c-a637-4834-b840-bb0bc29bc66d\") " Oct 29 23:24:49.234977 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:49.234954 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99d91e9c-a637-4834-b840-bb0bc29bc66d-maas-api-openshift-ca" pod "99d91e9c-a637-4834-b840-bb0bc29bc66d" (UID: "99d91e9c-a637-4834-b840-bb0bc29bc66d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:24:49.235371 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:49.235350 5057 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99d91e9c-a637-4834-b840-bb0bc29bc66d-maas-api-openshift-ca\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:24:49.236977 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:49.236955 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99d91e9c-a637-4834-b840-bb0bc29bc66d-maas-api-xks-ca" pod "99d91e9c-a637-4834-b840-bb0bc29bc66d" (UID: "99d91e9c-a637-4834-b840-bb0bc29bc66d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:24:49.237048 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:49.237010 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99d91e9c-a637-4834-b840-bb0bc29bc66d-kube-api-access-l746t" pod "99d91e9c-a637-4834-b840-bb0bc29bc66d" (UID: "99d91e9c-a637-4834-b840-bb0bc29bc66d"). InnerVolumeSpecName "kube-api-access-l746t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:24:49.336768 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:49.336726 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l746t\" (UniqueName: \"kubernetes.io/projected/99d91e9c-a637-4834-b840-bb0bc29bc66d-kube-api-access-l746t\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:24:49.336768 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:49.336763 5057 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99d91e9c-a637-4834-b840-bb0bc29bc66d-maas-api-xks-ca\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:24:49.525506 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:49.525398 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:24:49.643669 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:49.643627 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:24:49.895936 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:49.895864 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a9ae14-c5qxl" Oct 29 23:24:49.895936 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:49.895890 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a9ae14-c5qxl" event={"ID":"99d91e9c-a637-4834-b840-bb0bc29bc66d","Type":"ContainerDied","Data":"95c91dadb17ac96d787bf993af0ed6dee918a7e31341529e58cad090dec440a5"} Oct 29 23:24:49.896121 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:49.895937 5057 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="95c91dadb17ac96d787bf993af0ed6dee918a7e31341529e58cad090dec440a5" Oct 29 23:24:51.203887 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:51.203848 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:24:52.814738 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:52.814691 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:24:54.447596 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:54.447558 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:24:55.507977 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:55.507943 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-a9ae14-75774d5dcb-ddk62"] Oct 29 23:24:55.559183 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:55.559155 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-a9ae14-685b89c579-htlrh"] Oct 29 23:24:55.686644 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:55.686606 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:24:55.804093 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:55.803383 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:24:56.235867 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:56.235830 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:24:56.830997 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:56.830959 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:24:57.359431 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:57.356914 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-93b5809f-cxw6q"] Oct 29 23:24:57.359608 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:57.359583 5057 cpu_manager.go:414] "containerMap: removing container" podUID="99d91e9c-a637-4834-b840-bb0bc29bc66d" containerName="revoke-keys" Oct 29 23:24:57.359608 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:57.359605 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99d91e9c-a637-4834-b840-bb0bc29bc66d" containerName="revoke-keys" Oct 29 23:24:57.360401 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:57.360367 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="99d91e9c-a637-4834-b840-bb0bc29bc66d" containerName="revoke-keys" Oct 29 23:24:57.374586 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:57.374561 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-93b5809f-cxw6q"] Oct 29 23:24:57.374859 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:57.374806 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93b5809f-cxw6q" Oct 29 23:24:57.392063 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:57.392037 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:24:57.403217 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:57.403197 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dmx57\"" Oct 29 23:24:57.532263 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:57.532225 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kcvhl\" (UniqueName: \"kubernetes.io/projected/480effeb-d77e-407d-95c7-d242ab181afe-kube-api-access-kcvhl\") pod \"maas-api-revoke-sub-93b5809f-cxw6q\" (UID: \"480effeb-d77e-407d-95c7-d242ab181afe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93b5809f-cxw6q" Oct 29 23:24:57.532469 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:57.532361 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/480effeb-d77e-407d-95c7-d242ab181afe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-93b5809f-cxw6q\" (UID: \"480effeb-d77e-407d-95c7-d242ab181afe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93b5809f-cxw6q" Oct 29 23:24:57.532610 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:57.532578 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/480effeb-d77e-407d-95c7-d242ab181afe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-93b5809f-cxw6q\" (UID: \"480effeb-d77e-407d-95c7-d242ab181afe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93b5809f-cxw6q" Oct 29 23:24:57.634363 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:57.634231 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kcvhl\" (UniqueName: \"kubernetes.io/projected/480effeb-d77e-407d-95c7-d242ab181afe-kube-api-access-kcvhl\") pod \"maas-api-revoke-sub-93b5809f-cxw6q\" (UID: \"480effeb-d77e-407d-95c7-d242ab181afe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93b5809f-cxw6q" Oct 29 23:24:57.634554 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:57.634367 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/480effeb-d77e-407d-95c7-d242ab181afe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-93b5809f-cxw6q\" (UID: \"480effeb-d77e-407d-95c7-d242ab181afe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93b5809f-cxw6q" Oct 29 23:24:57.634621 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:57.634604 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/480effeb-d77e-407d-95c7-d242ab181afe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-93b5809f-cxw6q\" (UID: \"480effeb-d77e-407d-95c7-d242ab181afe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93b5809f-cxw6q" Oct 29 23:24:57.635212 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:57.635186 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/480effeb-d77e-407d-95c7-d242ab181afe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-93b5809f-cxw6q\" (UID: \"480effeb-d77e-407d-95c7-d242ab181afe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93b5809f-cxw6q" Oct 29 23:24:57.636857 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:57.636832 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/480effeb-d77e-407d-95c7-d242ab181afe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-93b5809f-cxw6q\" (UID: \"480effeb-d77e-407d-95c7-d242ab181afe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93b5809f-cxw6q" Oct 29 23:24:57.657246 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:57.657222 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kcvhl\" (UniqueName: \"kubernetes.io/projected/480effeb-d77e-407d-95c7-d242ab181afe-kube-api-access-kcvhl\") pod \"maas-api-revoke-sub-93b5809f-cxw6q\" (UID: \"480effeb-d77e-407d-95c7-d242ab181afe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93b5809f-cxw6q" Oct 29 23:24:57.702742 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:57.702715 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93b5809f-cxw6q" Oct 29 23:24:57.887459 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:57.887364 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:24:58.031650 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:58.031612 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-93b5809f-cxw6q"] Oct 29 23:24:58.032521 ip-10-0-140-74 kubenswrapper[5057]: W1029 23:24:58.032485 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod480effeb_d77e_407d_95c7_d242ab181afe.slice/crio-1149ea2c1e0258dae970e9ce552f855547f3b9b7573155a2db1b790b206ede6f WatchSource:0}: Error finding container 1149ea2c1e0258dae970e9ce552f855547f3b9b7573155a2db1b790b206ede6f: Status 404 returned error can't find the container with id 1149ea2c1e0258dae970e9ce552f855547f3b9b7573155a2db1b790b206ede6f Oct 29 23:24:59.037687 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:59.037657 5057 generic.go:356] "Generic (PLEG): container finished" podID="480effeb-d77e-407d-95c7-d242ab181afe" containerID="81dec6d33a1cdcb1f0d9abf7e190eb24d1da254eec4ae5de05a3eeb2625f826b" exitCode=0 Oct 29 23:24:59.038087 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:59.037741 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93b5809f-cxw6q" event={"ID":"480effeb-d77e-407d-95c7-d242ab181afe","Type":"ContainerDied","Data":"81dec6d33a1cdcb1f0d9abf7e190eb24d1da254eec4ae5de05a3eeb2625f826b"} Oct 29 23:24:59.038087 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:59.037806 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93b5809f-cxw6q" event={"ID":"480effeb-d77e-407d-95c7-d242ab181afe","Type":"ContainerStarted","Data":"1149ea2c1e0258dae970e9ce552f855547f3b9b7573155a2db1b790b206ede6f"} Oct 29 23:24:59.501747 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:24:59.501699 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:25:00.249253 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:00.249221 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93b5809f-cxw6q" Oct 29 23:25:00.380894 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:00.380813 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/480effeb-d77e-407d-95c7-d242ab181afe-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/480effeb-d77e-407d-95c7-d242ab181afe-maas-api-openshift-ca\") pod \"480effeb-d77e-407d-95c7-d242ab181afe\" (UID: \"480effeb-d77e-407d-95c7-d242ab181afe\") " Oct 29 23:25:00.381082 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:00.381062 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/480effeb-d77e-407d-95c7-d242ab181afe-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/480effeb-d77e-407d-95c7-d242ab181afe-maas-api-xks-ca\") pod \"480effeb-d77e-407d-95c7-d242ab181afe\" (UID: \"480effeb-d77e-407d-95c7-d242ab181afe\") " Oct 29 23:25:00.381172 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:00.381156 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/480effeb-d77e-407d-95c7-d242ab181afe-kube-api-access-kcvhl\" (UniqueName: \"kubernetes.io/projected/480effeb-d77e-407d-95c7-d242ab181afe-kube-api-access-kcvhl\") pod \"480effeb-d77e-407d-95c7-d242ab181afe\" (UID: \"480effeb-d77e-407d-95c7-d242ab181afe\") " Oct 29 23:25:00.381257 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:00.381220 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/480effeb-d77e-407d-95c7-d242ab181afe-maas-api-openshift-ca" pod "480effeb-d77e-407d-95c7-d242ab181afe" (UID: "480effeb-d77e-407d-95c7-d242ab181afe"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:25:00.381901 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:00.381875 5057 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/480effeb-d77e-407d-95c7-d242ab181afe-maas-api-openshift-ca\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:25:00.383230 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:00.383207 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/480effeb-d77e-407d-95c7-d242ab181afe-maas-api-xks-ca" pod "480effeb-d77e-407d-95c7-d242ab181afe" (UID: "480effeb-d77e-407d-95c7-d242ab181afe"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:25:00.383706 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:00.383680 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/480effeb-d77e-407d-95c7-d242ab181afe-kube-api-access-kcvhl" pod "480effeb-d77e-407d-95c7-d242ab181afe" (UID: "480effeb-d77e-407d-95c7-d242ab181afe"). InnerVolumeSpecName "kube-api-access-kcvhl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:25:00.483702 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:00.483673 5057 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/480effeb-d77e-407d-95c7-d242ab181afe-maas-api-xks-ca\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:25:00.483702 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:00.483704 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kcvhl\" (UniqueName: \"kubernetes.io/projected/480effeb-d77e-407d-95c7-d242ab181afe-kube-api-access-kcvhl\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:25:00.508283 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:00.508248 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-a9ae14-75774d5dcb-ddk62" podUID="3ef2b071-92d9-44ae-9068-cf14644d05d5" containerName="payload-processing" containerID="cri-o://29269ad39ce8c4695f85ab24c1d9a7f879d3439f101a7e0e5c5672acae85162d" gracePeriod=25 Oct 29 23:25:00.560240 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:00.560209 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a9ae14-685b89c579-htlrh" podUID="c68bc52c-5a5a-49ca-bd1a-4dc908b4aee0" containerName="payload-pre-processing" containerID="cri-o://3aec26c4039a003051f43c5542694ce94abc5decb7a965a2aac1c00f7f53a339" gracePeriod=25 Oct 29 23:25:00.794057 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:00.794021 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-a9ae14-75774d5dcb-ddk62" Oct 29 23:25:00.891415 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:00.891382 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ef2b071-92d9-44ae-9068-cf14644d05d5-kube-api-access-s58fz\" (UniqueName: \"kubernetes.io/projected/3ef2b071-92d9-44ae-9068-cf14644d05d5-kube-api-access-s58fz\") pod \"3ef2b071-92d9-44ae-9068-cf14644d05d5\" (UID: \"3ef2b071-92d9-44ae-9068-cf14644d05d5\") " Oct 29 23:25:00.891719 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:00.891698 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ef2b071-92d9-44ae-9068-cf14644d05d5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ef2b071-92d9-44ae-9068-cf14644d05d5-plugins-config-volume\") pod \"3ef2b071-92d9-44ae-9068-cf14644d05d5\" (UID: \"3ef2b071-92d9-44ae-9068-cf14644d05d5\") " Oct 29 23:25:00.892687 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:00.892658 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ef2b071-92d9-44ae-9068-cf14644d05d5-plugins-config-volume" pod "3ef2b071-92d9-44ae-9068-cf14644d05d5" (UID: "3ef2b071-92d9-44ae-9068-cf14644d05d5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:25:00.894509 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:00.894474 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ef2b071-92d9-44ae-9068-cf14644d05d5-kube-api-access-s58fz" pod "3ef2b071-92d9-44ae-9068-cf14644d05d5" (UID: "3ef2b071-92d9-44ae-9068-cf14644d05d5"). InnerVolumeSpecName "kube-api-access-s58fz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:25:00.993860 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:00.993826 5057 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ef2b071-92d9-44ae-9068-cf14644d05d5-plugins-config-volume\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:25:00.993860 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:00.993861 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s58fz\" (UniqueName: \"kubernetes.io/projected/3ef2b071-92d9-44ae-9068-cf14644d05d5-kube-api-access-s58fz\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:25:01.058032 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:01.058003 5057 generic.go:356] "Generic (PLEG): container finished" podID="3ef2b071-92d9-44ae-9068-cf14644d05d5" containerID="29269ad39ce8c4695f85ab24c1d9a7f879d3439f101a7e0e5c5672acae85162d" exitCode=0 Oct 29 23:25:01.058211 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:01.058049 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-a9ae14-75774d5dcb-ddk62" Oct 29 23:25:01.058211 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:01.058088 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-a9ae14-75774d5dcb-ddk62" event={"ID":"3ef2b071-92d9-44ae-9068-cf14644d05d5","Type":"ContainerDied","Data":"29269ad39ce8c4695f85ab24c1d9a7f879d3439f101a7e0e5c5672acae85162d"} Oct 29 23:25:01.058211 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:01.058146 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-a9ae14-75774d5dcb-ddk62" event={"ID":"3ef2b071-92d9-44ae-9068-cf14644d05d5","Type":"ContainerDied","Data":"519dad2a8ebc5a94746918b2e5ab6dee1a22e4f830f9b921af986c1b1f1dcd1d"} Oct 29 23:25:01.058211 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:01.058176 5057 scope.go:122] "RemoveContainer" containerID="29269ad39ce8c4695f85ab24c1d9a7f879d3439f101a7e0e5c5672acae85162d" Oct 29 23:25:01.059911 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:01.059872 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93b5809f-cxw6q" Oct 29 23:25:01.059911 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:01.059888 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93b5809f-cxw6q" event={"ID":"480effeb-d77e-407d-95c7-d242ab181afe","Type":"ContainerDied","Data":"1149ea2c1e0258dae970e9ce552f855547f3b9b7573155a2db1b790b206ede6f"} Oct 29 23:25:01.060041 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:01.059928 5057 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1149ea2c1e0258dae970e9ce552f855547f3b9b7573155a2db1b790b206ede6f" Oct 29 23:25:01.071729 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:01.071709 5057 scope.go:122] "RemoveContainer" containerID="29269ad39ce8c4695f85ab24c1d9a7f879d3439f101a7e0e5c5672acae85162d" Oct 29 23:25:01.072094 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:25:01.072023 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29269ad39ce8c4695f85ab24c1d9a7f879d3439f101a7e0e5c5672acae85162d\": container with ID starting with 29269ad39ce8c4695f85ab24c1d9a7f879d3439f101a7e0e5c5672acae85162d not found: ID does not exist" containerID="29269ad39ce8c4695f85ab24c1d9a7f879d3439f101a7e0e5c5672acae85162d" Oct 29 23:25:01.072094 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:01.072062 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29269ad39ce8c4695f85ab24c1d9a7f879d3439f101a7e0e5c5672acae85162d"} err="failed to get container status \"29269ad39ce8c4695f85ab24c1d9a7f879d3439f101a7e0e5c5672acae85162d\": rpc error: code = NotFound desc = could not find container \"29269ad39ce8c4695f85ab24c1d9a7f879d3439f101a7e0e5c5672acae85162d\": container with ID starting with 29269ad39ce8c4695f85ab24c1d9a7f879d3439f101a7e0e5c5672acae85162d not found: ID does not exist" Oct 29 23:25:01.121534 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:01.121495 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-a9ae14-75774d5dcb-ddk62"] Oct 29 23:25:01.129701 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:01.129669 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:25:01.133139 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:01.133104 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-a9ae14-75774d5dcb-ddk62"] Oct 29 23:25:01.333433 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:01.333353 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ef2b071-92d9-44ae-9068-cf14644d05d5" path="/var/lib/kubelet/pods/3ef2b071-92d9-44ae-9068-cf14644d05d5/volumes" Oct 29 23:25:02.578510 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:02.578473 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:25:03.373119 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:03.373085 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:25:03.426093 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:03.426048 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a9ae14-685b89c579-htlrh" podUID="c68bc52c-5a5a-49ca-bd1a-4dc908b4aee0" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:25:03.510898 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:03.510862 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:25:03.559284 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:03.559251 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1056cc-djplr"] Oct 29 23:25:03.568577 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:03.568551 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1056cc-djplr"] Oct 29 23:25:03.663947 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:03.663866 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:25:04.083144 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:04.083101 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:25:04.890012 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:04.889961 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:25:05.003558 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:05.003507 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:25:05.334465 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:05.334433 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cef7c673-4967-458a-8e9d-267a288ff0c4" path="/var/lib/kubelet/pods/cef7c673-4967-458a-8e9d-267a288ff0c4/volumes" Oct 29 23:25:05.513193 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:05.513157 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:25:07.399381 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:07.399237 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:25:09.135723 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:09.135687 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:25:09.400023 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:09.399938 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:25:09.469489 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:09.469070 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-1a8107/test-model-1a8107-kserve-9bc6dcdc-z65tc"] Oct 29 23:25:11.043743 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:11.043709 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:25:11.190802 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:11.190758 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:25:11.306036 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:11.305913 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:25:11.538429 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:11.538393 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-1056cc-openshift-default-7b4b9768c9-vsbgm"] Oct 29 23:25:11.539106 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:11.539072 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-1056cc-openshift-default-7b4b9768c9-vsbgm" podUID="21034c72-9803-4050-b5b5-df014d3668e4" containerName="istio-proxy" containerID="cri-o://8483f8d3504552b3c392e1b0198deb2044ee95f156ee1c04f18c4f050688ae69" gracePeriod=30 Oct 29 23:25:11.564644 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:11.564569 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:25:12.154302 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:12.154267 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b2b71e-vbzwv"] Oct 29 23:25:12.155631 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:12.155608 5057 cpu_manager.go:414] "containerMap: removing container" podUID="480effeb-d77e-407d-95c7-d242ab181afe" containerName="revoke-keys" Oct 29 23:25:12.155631 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:12.155630 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="480effeb-d77e-407d-95c7-d242ab181afe" containerName="revoke-keys" Oct 29 23:25:12.155798 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:12.155667 5057 cpu_manager.go:414] "containerMap: removing container" podUID="3ef2b071-92d9-44ae-9068-cf14644d05d5" containerName="payload-processing" Oct 29 23:25:12.155798 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:12.155674 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ef2b071-92d9-44ae-9068-cf14644d05d5" containerName="payload-processing" Oct 29 23:25:12.155901 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:12.155848 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="480effeb-d77e-407d-95c7-d242ab181afe" containerName="revoke-keys" Oct 29 23:25:12.155901 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:12.155861 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ef2b071-92d9-44ae-9068-cf14644d05d5" containerName="payload-processing" Oct 29 23:25:12.160478 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:12.160459 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b2b71e-vbzwv" Oct 29 23:25:12.178145 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:12.178117 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35e18ec0-85f5-497b-8f44-907262416627-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-b2b71e-vbzwv\" (UID: \"35e18ec0-85f5-497b-8f44-907262416627\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b2b71e-vbzwv" Oct 29 23:25:12.178385 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:12.178347 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35e18ec0-85f5-497b-8f44-907262416627-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-b2b71e-vbzwv\" (UID: \"35e18ec0-85f5-497b-8f44-907262416627\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b2b71e-vbzwv" Oct 29 23:25:12.178552 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:12.178531 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfx6j\" (UniqueName: \"kubernetes.io/projected/35e18ec0-85f5-497b-8f44-907262416627-kube-api-access-jfx6j\") pod \"maas-api-revoke-keys-e2e-inf-b-b2b71e-vbzwv\" (UID: \"35e18ec0-85f5-497b-8f44-907262416627\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b2b71e-vbzwv" Oct 29 23:25:12.178639 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:12.178622 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b2b71e-vbzwv"] Oct 29 23:25:12.204177 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:12.204143 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dmx57\"" Oct 29 23:25:12.204354 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:12.204194 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:25:12.280472 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:12.280436 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35e18ec0-85f5-497b-8f44-907262416627-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-b2b71e-vbzwv\" (UID: \"35e18ec0-85f5-497b-8f44-907262416627\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b2b71e-vbzwv" Oct 29 23:25:12.280666 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:12.280487 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35e18ec0-85f5-497b-8f44-907262416627-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-b2b71e-vbzwv\" (UID: \"35e18ec0-85f5-497b-8f44-907262416627\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b2b71e-vbzwv" Oct 29 23:25:12.280666 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:12.280585 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfx6j\" (UniqueName: \"kubernetes.io/projected/35e18ec0-85f5-497b-8f44-907262416627-kube-api-access-jfx6j\") pod \"maas-api-revoke-keys-e2e-inf-b-b2b71e-vbzwv\" (UID: \"35e18ec0-85f5-497b-8f44-907262416627\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b2b71e-vbzwv" Oct 29 23:25:12.281243 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:12.281219 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35e18ec0-85f5-497b-8f44-907262416627-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-b2b71e-vbzwv\" (UID: \"35e18ec0-85f5-497b-8f44-907262416627\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b2b71e-vbzwv" Oct 29 23:25:12.283172 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:12.283143 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35e18ec0-85f5-497b-8f44-907262416627-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-b2b71e-vbzwv\" (UID: \"35e18ec0-85f5-497b-8f44-907262416627\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b2b71e-vbzwv" Oct 29 23:25:12.311010 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:12.310975 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfx6j\" (UniqueName: \"kubernetes.io/projected/35e18ec0-85f5-497b-8f44-907262416627-kube-api-access-jfx6j\") pod \"maas-api-revoke-keys-e2e-inf-b-b2b71e-vbzwv\" (UID: \"35e18ec0-85f5-497b-8f44-907262416627\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b2b71e-vbzwv" Oct 29 23:25:12.478469 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:12.478434 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b2b71e-vbzwv" Oct 29 23:25:12.796477 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:12.796448 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b2b71e-vbzwv"] Oct 29 23:25:12.797372 ip-10-0-140-74 kubenswrapper[5057]: W1029 23:25:12.797299 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35e18ec0_85f5_497b_8f44_907262416627.slice/crio-82a377b1f14c50b93e6f22008b01d83de30ec981833ceac180538745675c00e0 WatchSource:0}: Error finding container 82a377b1f14c50b93e6f22008b01d83de30ec981833ceac180538745675c00e0: Status 404 returned error can't find the container with id 82a377b1f14c50b93e6f22008b01d83de30ec981833ceac180538745675c00e0 Oct 29 23:25:12.898917 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:12.898884 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:25:13.229557 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:13.229524 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b2b71e-vbzwv" event={"ID":"35e18ec0-85f5-497b-8f44-907262416627","Type":"ContainerStarted","Data":"e6f242d686dbf299d854b5a83769ff7f9e7396435f5cad56517d4b2a7ccf7307"} Oct 29 23:25:13.230038 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:13.229571 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b2b71e-vbzwv" event={"ID":"35e18ec0-85f5-497b-8f44-907262416627","Type":"ContainerStarted","Data":"82a377b1f14c50b93e6f22008b01d83de30ec981833ceac180538745675c00e0"} Oct 29 23:25:13.274022 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:13.273974 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b2b71e-vbzwv" podStartSLOduration=1.273960969 podStartE2EDuration="1.273960969s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:25:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:25:13.268916701 +0000 UTC m=+3094.052130128" watchObservedRunningTime="2026-10-29 23:25:13.273960969 +0000 UTC m=+3094.057174395" Oct 29 23:25:13.425634 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:13.425552 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a9ae14-685b89c579-htlrh" podUID="c68bc52c-5a5a-49ca-bd1a-4dc908b4aee0" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:25:14.240608 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:14.240576 5057 generic.go:356] "Generic (PLEG): container finished" podID="35e18ec0-85f5-497b-8f44-907262416627" containerID="e6f242d686dbf299d854b5a83769ff7f9e7396435f5cad56517d4b2a7ccf7307" exitCode=0 Oct 29 23:25:14.241018 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:14.240667 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b2b71e-vbzwv" event={"ID":"35e18ec0-85f5-497b-8f44-907262416627","Type":"ContainerDied","Data":"e6f242d686dbf299d854b5a83769ff7f9e7396435f5cad56517d4b2a7ccf7307"} Oct 29 23:25:14.487301 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:14.487264 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:25:15.541161 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:15.540739 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b2b71e-vbzwv" Oct 29 23:25:15.652727 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:15.652688 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/35e18ec0-85f5-497b-8f44-907262416627-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35e18ec0-85f5-497b-8f44-907262416627-maas-api-xks-ca\") pod \"35e18ec0-85f5-497b-8f44-907262416627\" (UID: \"35e18ec0-85f5-497b-8f44-907262416627\") " Oct 29 23:25:15.652928 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:15.652761 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35e18ec0-85f5-497b-8f44-907262416627-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35e18ec0-85f5-497b-8f44-907262416627-maas-api-openshift-ca\") pod \"35e18ec0-85f5-497b-8f44-907262416627\" (UID: \"35e18ec0-85f5-497b-8f44-907262416627\") " Oct 29 23:25:15.653053 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:15.653032 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35e18ec0-85f5-497b-8f44-907262416627-kube-api-access-jfx6j\" (UniqueName: \"kubernetes.io/projected/35e18ec0-85f5-497b-8f44-907262416627-kube-api-access-jfx6j\") pod \"35e18ec0-85f5-497b-8f44-907262416627\" (UID: \"35e18ec0-85f5-497b-8f44-907262416627\") " Oct 29 23:25:15.674354 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:15.667399 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35e18ec0-85f5-497b-8f44-907262416627-maas-api-openshift-ca" pod "35e18ec0-85f5-497b-8f44-907262416627" (UID: "35e18ec0-85f5-497b-8f44-907262416627"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:25:15.677363 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:15.675771 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/35e18ec0-85f5-497b-8f44-907262416627-maas-api-xks-ca" pod "35e18ec0-85f5-497b-8f44-907262416627" (UID: "35e18ec0-85f5-497b-8f44-907262416627"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:25:15.691033 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:15.680774 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35e18ec0-85f5-497b-8f44-907262416627-kube-api-access-jfx6j" pod "35e18ec0-85f5-497b-8f44-907262416627" (UID: "35e18ec0-85f5-497b-8f44-907262416627"). InnerVolumeSpecName "kube-api-access-jfx6j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:25:15.709121 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:15.709023 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:25:15.764448 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:15.757586 5057 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35e18ec0-85f5-497b-8f44-907262416627-maas-api-xks-ca\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:25:15.764448 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:15.757625 5057 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35e18ec0-85f5-497b-8f44-907262416627-maas-api-openshift-ca\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:25:15.764448 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:15.757642 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jfx6j\" (UniqueName: \"kubernetes.io/projected/35e18ec0-85f5-497b-8f44-907262416627-kube-api-access-jfx6j\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:25:16.263273 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:16.263239 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b2b71e-vbzwv" Oct 29 23:25:16.263494 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:16.263264 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b2b71e-vbzwv" event={"ID":"35e18ec0-85f5-497b-8f44-907262416627","Type":"ContainerDied","Data":"82a377b1f14c50b93e6f22008b01d83de30ec981833ceac180538745675c00e0"} Oct 29 23:25:16.263494 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:16.263367 5057 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="82a377b1f14c50b93e6f22008b01d83de30ec981833ceac180538745675c00e0" Oct 29 23:25:16.664713 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:16.664679 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:25:17.145243 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:17.145205 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1056cc-openshift-default-7b4b9768c9-vsbgm" Oct 29 23:25:17.171306 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:17.171264 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21034c72-9803-4050-b5b5-df014d3668e4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21034c72-9803-4050-b5b5-df014d3668e4-istio-data\") pod \"21034c72-9803-4050-b5b5-df014d3668e4\" (UID: \"21034c72-9803-4050-b5b5-df014d3668e4\") " Oct 29 23:25:17.171582 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:17.171366 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21034c72-9803-4050-b5b5-df014d3668e4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21034c72-9803-4050-b5b5-df014d3668e4-workload-socket\") pod \"21034c72-9803-4050-b5b5-df014d3668e4\" (UID: \"21034c72-9803-4050-b5b5-df014d3668e4\") " Oct 29 23:25:17.171582 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:17.171438 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21034c72-9803-4050-b5b5-df014d3668e4-kube-api-access-vhx4l\" (UniqueName: \"kubernetes.io/projected/21034c72-9803-4050-b5b5-df014d3668e4-kube-api-access-vhx4l\") pod \"21034c72-9803-4050-b5b5-df014d3668e4\" (UID: \"21034c72-9803-4050-b5b5-df014d3668e4\") " Oct 29 23:25:17.171582 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:17.171516 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/21034c72-9803-4050-b5b5-df014d3668e4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21034c72-9803-4050-b5b5-df014d3668e4-istio-podinfo\") pod \"21034c72-9803-4050-b5b5-df014d3668e4\" (UID: \"21034c72-9803-4050-b5b5-df014d3668e4\") " Oct 29 23:25:17.171582 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:17.171559 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21034c72-9803-4050-b5b5-df014d3668e4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21034c72-9803-4050-b5b5-df014d3668e4-workload-certs\") pod \"21034c72-9803-4050-b5b5-df014d3668e4\" (UID: \"21034c72-9803-4050-b5b5-df014d3668e4\") " Oct 29 23:25:17.171830 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:17.171635 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21034c72-9803-4050-b5b5-df014d3668e4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21034c72-9803-4050-b5b5-df014d3668e4-istio-envoy\") pod \"21034c72-9803-4050-b5b5-df014d3668e4\" (UID: \"21034c72-9803-4050-b5b5-df014d3668e4\") " Oct 29 23:25:17.171830 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:17.171710 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21034c72-9803-4050-b5b5-df014d3668e4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21034c72-9803-4050-b5b5-df014d3668e4-credential-socket\") pod \"21034c72-9803-4050-b5b5-df014d3668e4\" (UID: \"21034c72-9803-4050-b5b5-df014d3668e4\") " Oct 29 23:25:17.171830 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:17.171784 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21034c72-9803-4050-b5b5-df014d3668e4-istio-token\" (UniqueName: \"kubernetes.io/projected/21034c72-9803-4050-b5b5-df014d3668e4-istio-token\") pod \"21034c72-9803-4050-b5b5-df014d3668e4\" (UID: \"21034c72-9803-4050-b5b5-df014d3668e4\") " Oct 29 23:25:17.171993 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:17.171893 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21034c72-9803-4050-b5b5-df014d3668e4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21034c72-9803-4050-b5b5-df014d3668e4-istiod-ca-cert\") pod \"21034c72-9803-4050-b5b5-df014d3668e4\" (UID: \"21034c72-9803-4050-b5b5-df014d3668e4\") " Oct 29 23:25:17.173652 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:17.173120 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21034c72-9803-4050-b5b5-df014d3668e4-istiod-ca-cert" pod "21034c72-9803-4050-b5b5-df014d3668e4" (UID: "21034c72-9803-4050-b5b5-df014d3668e4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:25:17.175124 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:17.172752 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21034c72-9803-4050-b5b5-df014d3668e4-credential-socket" pod "21034c72-9803-4050-b5b5-df014d3668e4" (UID: "21034c72-9803-4050-b5b5-df014d3668e4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:25:17.175484 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:17.175299 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21034c72-9803-4050-b5b5-df014d3668e4-workload-certs" pod "21034c72-9803-4050-b5b5-df014d3668e4" (UID: "21034c72-9803-4050-b5b5-df014d3668e4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:25:17.175786 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:17.175607 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21034c72-9803-4050-b5b5-df014d3668e4-istio-data" pod "21034c72-9803-4050-b5b5-df014d3668e4" (UID: "21034c72-9803-4050-b5b5-df014d3668e4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:25:17.176232 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:17.175911 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21034c72-9803-4050-b5b5-df014d3668e4-workload-socket" pod "21034c72-9803-4050-b5b5-df014d3668e4" (UID: "21034c72-9803-4050-b5b5-df014d3668e4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:25:17.204590 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:17.204542 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21034c72-9803-4050-b5b5-df014d3668e4-istio-envoy" pod "21034c72-9803-4050-b5b5-df014d3668e4" (UID: "21034c72-9803-4050-b5b5-df014d3668e4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:25:17.220849 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:17.220655 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21034c72-9803-4050-b5b5-df014d3668e4-istio-token" pod "21034c72-9803-4050-b5b5-df014d3668e4" (UID: "21034c72-9803-4050-b5b5-df014d3668e4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:25:17.244607 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:17.237497 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/21034c72-9803-4050-b5b5-df014d3668e4-istio-podinfo" pod "21034c72-9803-4050-b5b5-df014d3668e4" (UID: "21034c72-9803-4050-b5b5-df014d3668e4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:25:17.281429 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:17.263200 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21034c72-9803-4050-b5b5-df014d3668e4-kube-api-access-vhx4l" pod "21034c72-9803-4050-b5b5-df014d3668e4" (UID: "21034c72-9803-4050-b5b5-df014d3668e4"). InnerVolumeSpecName "kube-api-access-vhx4l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:25:17.307518 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:17.307481 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vhx4l\" (UniqueName: \"kubernetes.io/projected/21034c72-9803-4050-b5b5-df014d3668e4-kube-api-access-vhx4l\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:25:17.307751 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:17.307730 5057 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21034c72-9803-4050-b5b5-df014d3668e4-istio-podinfo\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:25:17.307857 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:17.307845 5057 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21034c72-9803-4050-b5b5-df014d3668e4-workload-certs\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:25:17.307943 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:17.307932 5057 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21034c72-9803-4050-b5b5-df014d3668e4-istio-envoy\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:25:17.308037 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:17.308026 5057 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21034c72-9803-4050-b5b5-df014d3668e4-credential-socket\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:25:17.308152 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:17.308139 5057 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21034c72-9803-4050-b5b5-df014d3668e4-istio-token\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:25:17.308244 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:17.308233 5057 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21034c72-9803-4050-b5b5-df014d3668e4-istiod-ca-cert\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:25:17.308386 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:17.308357 5057 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21034c72-9803-4050-b5b5-df014d3668e4-istio-data\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:25:17.308534 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:17.308520 5057 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21034c72-9803-4050-b5b5-df014d3668e4-workload-socket\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:25:17.349933 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:17.349893 5057 generic.go:356] "Generic (PLEG): container finished" podID="21034c72-9803-4050-b5b5-df014d3668e4" containerID="8483f8d3504552b3c392e1b0198deb2044ee95f156ee1c04f18c4f050688ae69" exitCode=0 Oct 29 23:25:17.350265 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:17.350243 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1056cc-openshift-default-7b4b9768c9-vsbgm" Oct 29 23:25:17.385689 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:17.385649 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1056cc-openshift-default-7b4b9768c9-vsbgm" event={"ID":"21034c72-9803-4050-b5b5-df014d3668e4","Type":"ContainerDied","Data":"8483f8d3504552b3c392e1b0198deb2044ee95f156ee1c04f18c4f050688ae69"} Oct 29 23:25:17.385816 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:17.385695 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1056cc-openshift-default-7b4b9768c9-vsbgm" event={"ID":"21034c72-9803-4050-b5b5-df014d3668e4","Type":"ContainerDied","Data":"32746f43afcb0588619144ef6bd702aac8f9665448a3281ab7c5eab24023eb84"} Oct 29 23:25:17.385816 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:17.385717 5057 scope.go:122] "RemoveContainer" containerID="8483f8d3504552b3c392e1b0198deb2044ee95f156ee1c04f18c4f050688ae69" Oct 29 23:25:17.458523 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:17.457954 5057 scope.go:122] "RemoveContainer" containerID="8483f8d3504552b3c392e1b0198deb2044ee95f156ee1c04f18c4f050688ae69" Oct 29 23:25:17.458523 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:25:17.458367 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8483f8d3504552b3c392e1b0198deb2044ee95f156ee1c04f18c4f050688ae69\": container with ID starting with 8483f8d3504552b3c392e1b0198deb2044ee95f156ee1c04f18c4f050688ae69 not found: ID does not exist" containerID="8483f8d3504552b3c392e1b0198deb2044ee95f156ee1c04f18c4f050688ae69" Oct 29 23:25:17.458523 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:17.458407 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8483f8d3504552b3c392e1b0198deb2044ee95f156ee1c04f18c4f050688ae69"} err="failed to get container status \"8483f8d3504552b3c392e1b0198deb2044ee95f156ee1c04f18c4f050688ae69\": rpc error: code = NotFound desc = could not find container \"8483f8d3504552b3c392e1b0198deb2044ee95f156ee1c04f18c4f050688ae69\": container with ID starting with 8483f8d3504552b3c392e1b0198deb2044ee95f156ee1c04f18c4f050688ae69 not found: ID does not exist" Oct 29 23:25:17.462947 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:17.462921 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-1056cc-openshift-default-7b4b9768c9-vsbgm"] Oct 29 23:25:17.471457 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:17.471432 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-1056cc-openshift-default-7b4b9768c9-vsbgm"] Oct 29 23:25:18.523542 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:18.523513 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:25:18.629293 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:18.629259 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:25:19.031761 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:25:19.031724 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d2dnn" containerName="extractor" Oct 29 23:25:19.334425 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:19.334327 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21034c72-9803-4050-b5b5-df014d3668e4" path="/var/lib/kubelet/pods/21034c72-9803-4050-b5b5-df014d3668e4/volumes" Oct 29 23:25:19.958208 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:19.958171 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-faa4fc-5f7c674df4-mgk4c"] Oct 29 23:25:20.100095 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:20.100050 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:25:20.241875 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:20.241787 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:25:20.994254 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:20.994228 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a9ae14-685b89c579-htlrh" Oct 29 23:25:21.098088 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:21.098052 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c68bc52c-5a5a-49ca-bd1a-4dc908b4aee0-kube-api-access-75b54\" (UniqueName: \"kubernetes.io/projected/c68bc52c-5a5a-49ca-bd1a-4dc908b4aee0-kube-api-access-75b54\") pod \"c68bc52c-5a5a-49ca-bd1a-4dc908b4aee0\" (UID: \"c68bc52c-5a5a-49ca-bd1a-4dc908b4aee0\") " Oct 29 23:25:21.098378 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:21.098352 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c68bc52c-5a5a-49ca-bd1a-4dc908b4aee0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c68bc52c-5a5a-49ca-bd1a-4dc908b4aee0-plugins-config-volume\") pod \"c68bc52c-5a5a-49ca-bd1a-4dc908b4aee0\" (UID: \"c68bc52c-5a5a-49ca-bd1a-4dc908b4aee0\") " Oct 29 23:25:21.098945 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:21.098914 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c68bc52c-5a5a-49ca-bd1a-4dc908b4aee0-plugins-config-volume" pod "c68bc52c-5a5a-49ca-bd1a-4dc908b4aee0" (UID: "c68bc52c-5a5a-49ca-bd1a-4dc908b4aee0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:25:21.099247 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:21.099225 5057 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c68bc52c-5a5a-49ca-bd1a-4dc908b4aee0-plugins-config-volume\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:25:21.101963 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:21.101931 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c68bc52c-5a5a-49ca-bd1a-4dc908b4aee0-kube-api-access-75b54" pod "c68bc52c-5a5a-49ca-bd1a-4dc908b4aee0" (UID: "c68bc52c-5a5a-49ca-bd1a-4dc908b4aee0"). InnerVolumeSpecName "kube-api-access-75b54". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:25:21.200718 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:21.200688 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-75b54\" (UniqueName: \"kubernetes.io/projected/c68bc52c-5a5a-49ca-bd1a-4dc908b4aee0-kube-api-access-75b54\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:25:21.399271 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:21.399180 5057 generic.go:356] "Generic (PLEG): container finished" podID="c68bc52c-5a5a-49ca-bd1a-4dc908b4aee0" containerID="3aec26c4039a003051f43c5542694ce94abc5decb7a965a2aac1c00f7f53a339" exitCode=0 Oct 29 23:25:21.399271 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:21.399226 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a9ae14-685b89c579-htlrh" Oct 29 23:25:21.399271 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:21.399238 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a9ae14-685b89c579-htlrh" event={"ID":"c68bc52c-5a5a-49ca-bd1a-4dc908b4aee0","Type":"ContainerDied","Data":"3aec26c4039a003051f43c5542694ce94abc5decb7a965a2aac1c00f7f53a339"} Oct 29 23:25:21.399543 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:21.399292 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a9ae14-685b89c579-htlrh" event={"ID":"c68bc52c-5a5a-49ca-bd1a-4dc908b4aee0","Type":"ContainerDied","Data":"3c108f2faddce9bfb5e1fa2cc40a9a0c16337408d1cb70173c88eee067757858"} Oct 29 23:25:21.399543 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:21.399326 5057 scope.go:122] "RemoveContainer" containerID="3aec26c4039a003051f43c5542694ce94abc5decb7a965a2aac1c00f7f53a339" Oct 29 23:25:21.413770 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:21.413750 5057 scope.go:122] "RemoveContainer" containerID="3aec26c4039a003051f43c5542694ce94abc5decb7a965a2aac1c00f7f53a339" Oct 29 23:25:21.414113 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:25:21.414085 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3aec26c4039a003051f43c5542694ce94abc5decb7a965a2aac1c00f7f53a339\": container with ID starting with 3aec26c4039a003051f43c5542694ce94abc5decb7a965a2aac1c00f7f53a339 not found: ID does not exist" containerID="3aec26c4039a003051f43c5542694ce94abc5decb7a965a2aac1c00f7f53a339" Oct 29 23:25:21.414190 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:21.414123 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3aec26c4039a003051f43c5542694ce94abc5decb7a965a2aac1c00f7f53a339"} err="failed to get container status \"3aec26c4039a003051f43c5542694ce94abc5decb7a965a2aac1c00f7f53a339\": rpc error: code = NotFound desc = could not find container \"3aec26c4039a003051f43c5542694ce94abc5decb7a965a2aac1c00f7f53a339\": container with ID starting with 3aec26c4039a003051f43c5542694ce94abc5decb7a965a2aac1c00f7f53a339 not found: ID does not exist" Oct 29 23:25:21.454246 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:21.454214 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-a9ae14-685b89c579-htlrh"] Oct 29 23:25:21.466045 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:21.466019 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-a9ae14-685b89c579-htlrh"] Oct 29 23:25:21.547884 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:21.547851 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-qbx4b"] Oct 29 23:25:21.548073 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:21.548052 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-qbx4b" podUID="e7dbc6f5-3755-4686-b1af-4fb84d22db3a" containerName="istio-proxy" containerID="cri-o://84195371efdc4d7172b68a9dc56f842bf74b50f4a8573093a1b0ea4dd9aad2b1" gracePeriod=30 Oct 29 23:25:21.839872 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:21.839842 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:25:21.967523 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:21.952927 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:25:22.715234 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:22.715185 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:25:22.835967 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:22.835928 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:25:23.397144 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:23.397099 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c68bc52c-5a5a-49ca-bd1a-4dc908b4aee0" path="/var/lib/kubelet/pods/c68bc52c-5a5a-49ca-bd1a-4dc908b4aee0/volumes" Oct 29 23:25:23.865655 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:23.865608 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:25:24.540743 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:24.540682 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-1a8107/test-model-1a8107-kserve-9bc6dcdc-z65tc" podUID="23b3cc65-71ec-4963-9045-175ab7a0b0d5" containerName="main" containerID="cri-o://cbd1651bf8b60d7555a78a4211056fbd3acf51576bac4f5a1a7d63b08cc92389" gracePeriod=45 Oct 29 23:25:24.856155 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:24.856126 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-1a8107/test-model-1a8107-kserve-9bc6dcdc-z65tc" Oct 29 23:25:24.958912 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:24.958856 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-faa4fc-5f7c674df4-mgk4c" podUID="32151649-e3d5-434d-8385-d318db8ade35" containerName="payload-processing" containerID="cri-o://471095009c90e5135edc3e4bfbc83d9fb155a16bed636ca1b37f53a5e922abf1" gracePeriod=25 Oct 29 23:25:24.976033 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:24.976003 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23b3cc65-71ec-4963-9045-175ab7a0b0d5-kube-api-access-nfp64\" (UniqueName: \"kubernetes.io/projected/23b3cc65-71ec-4963-9045-175ab7a0b0d5-kube-api-access-nfp64\") pod \"23b3cc65-71ec-4963-9045-175ab7a0b0d5\" (UID: \"23b3cc65-71ec-4963-9045-175ab7a0b0d5\") " Oct 29 23:25:24.976211 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:24.976188 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23b3cc65-71ec-4963-9045-175ab7a0b0d5-home\" (UniqueName: \"kubernetes.io/empty-dir/23b3cc65-71ec-4963-9045-175ab7a0b0d5-home\") pod \"23b3cc65-71ec-4963-9045-175ab7a0b0d5\" (UID: \"23b3cc65-71ec-4963-9045-175ab7a0b0d5\") " Oct 29 23:25:24.976294 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:24.976273 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/23b3cc65-71ec-4963-9045-175ab7a0b0d5-tls-certs\" (UniqueName: \"kubernetes.io/secret/23b3cc65-71ec-4963-9045-175ab7a0b0d5-tls-certs\") pod \"23b3cc65-71ec-4963-9045-175ab7a0b0d5\" (UID: \"23b3cc65-71ec-4963-9045-175ab7a0b0d5\") " Oct 29 23:25:24.976714 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:24.976436 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23b3cc65-71ec-4963-9045-175ab7a0b0d5-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/23b3cc65-71ec-4963-9045-175ab7a0b0d5-tmp-dir\") pod \"23b3cc65-71ec-4963-9045-175ab7a0b0d5\" (UID: \"23b3cc65-71ec-4963-9045-175ab7a0b0d5\") " Oct 29 23:25:24.976833 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:24.976805 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23b3cc65-71ec-4963-9045-175ab7a0b0d5-dshm\" (UniqueName: \"kubernetes.io/empty-dir/23b3cc65-71ec-4963-9045-175ab7a0b0d5-dshm\") pod \"23b3cc65-71ec-4963-9045-175ab7a0b0d5\" (UID: \"23b3cc65-71ec-4963-9045-175ab7a0b0d5\") " Oct 29 23:25:24.977000 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:24.976977 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23b3cc65-71ec-4963-9045-175ab7a0b0d5-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/23b3cc65-71ec-4963-9045-175ab7a0b0d5-model-cache\") pod \"23b3cc65-71ec-4963-9045-175ab7a0b0d5\" (UID: \"23b3cc65-71ec-4963-9045-175ab7a0b0d5\") " Oct 29 23:25:24.978158 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:24.976555 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23b3cc65-71ec-4963-9045-175ab7a0b0d5-home" pod "23b3cc65-71ec-4963-9045-175ab7a0b0d5" (UID: "23b3cc65-71ec-4963-9045-175ab7a0b0d5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:25:24.978253 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:24.976666 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23b3cc65-71ec-4963-9045-175ab7a0b0d5-tmp-dir" pod "23b3cc65-71ec-4963-9045-175ab7a0b0d5" (UID: "23b3cc65-71ec-4963-9045-175ab7a0b0d5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:25:24.978253 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:24.978132 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23b3cc65-71ec-4963-9045-175ab7a0b0d5-model-cache" pod "23b3cc65-71ec-4963-9045-175ab7a0b0d5" (UID: "23b3cc65-71ec-4963-9045-175ab7a0b0d5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:25:24.979243 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:24.979216 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23b3cc65-71ec-4963-9045-175ab7a0b0d5-kube-api-access-nfp64" pod "23b3cc65-71ec-4963-9045-175ab7a0b0d5" (UID: "23b3cc65-71ec-4963-9045-175ab7a0b0d5"). InnerVolumeSpecName "kube-api-access-nfp64". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:25:24.981171 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:24.981141 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23b3cc65-71ec-4963-9045-175ab7a0b0d5-dshm" pod "23b3cc65-71ec-4963-9045-175ab7a0b0d5" (UID: "23b3cc65-71ec-4963-9045-175ab7a0b0d5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:25:24.982482 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:24.982449 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23b3cc65-71ec-4963-9045-175ab7a0b0d5-tls-certs" pod "23b3cc65-71ec-4963-9045-175ab7a0b0d5" (UID: "23b3cc65-71ec-4963-9045-175ab7a0b0d5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:25:25.079743 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:25.079648 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nfp64\" (UniqueName: \"kubernetes.io/projected/23b3cc65-71ec-4963-9045-175ab7a0b0d5-kube-api-access-nfp64\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:25:25.079743 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:25.079677 5057 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/23b3cc65-71ec-4963-9045-175ab7a0b0d5-home\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:25:25.079743 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:25.079694 5057 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/23b3cc65-71ec-4963-9045-175ab7a0b0d5-tls-certs\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:25:25.079743 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:25.079707 5057 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/23b3cc65-71ec-4963-9045-175ab7a0b0d5-tmp-dir\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:25:25.079743 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:25.079719 5057 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/23b3cc65-71ec-4963-9045-175ab7a0b0d5-dshm\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:25:25.079743 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:25.079732 5057 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/23b3cc65-71ec-4963-9045-175ab7a0b0d5-model-cache\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:25:25.211425 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:25.211376 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:25:25.301553 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:25.301524 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-faa4fc-5f7c674df4-mgk4c" Oct 29 23:25:25.411352 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:25.396012 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32151649-e3d5-434d-8385-d318db8ade35-kube-api-access-7jstp\" (UniqueName: \"kubernetes.io/projected/32151649-e3d5-434d-8385-d318db8ade35-kube-api-access-7jstp\") pod \"32151649-e3d5-434d-8385-d318db8ade35\" (UID: \"32151649-e3d5-434d-8385-d318db8ade35\") " Oct 29 23:25:25.411352 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:25.396389 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/32151649-e3d5-434d-8385-d318db8ade35-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32151649-e3d5-434d-8385-d318db8ade35-plugins-config-volume\") pod \"32151649-e3d5-434d-8385-d318db8ade35\" (UID: \"32151649-e3d5-434d-8385-d318db8ade35\") " Oct 29 23:25:25.466359 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:25.445785 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32151649-e3d5-434d-8385-d318db8ade35-plugins-config-volume" pod "32151649-e3d5-434d-8385-d318db8ade35" (UID: "32151649-e3d5-434d-8385-d318db8ade35"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:25:25.468522 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:25.468477 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32151649-e3d5-434d-8385-d318db8ade35-kube-api-access-7jstp" pod "32151649-e3d5-434d-8385-d318db8ade35" (UID: "32151649-e3d5-434d-8385-d318db8ade35"). InnerVolumeSpecName "kube-api-access-7jstp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:25:25.569532 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:25.568775 5057 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32151649-e3d5-434d-8385-d318db8ade35-plugins-config-volume\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:25:25.569532 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:25.568803 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7jstp\" (UniqueName: \"kubernetes.io/projected/32151649-e3d5-434d-8385-d318db8ade35-kube-api-access-7jstp\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:25:25.596235 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:25.595214 5057 generic.go:356] "Generic (PLEG): container finished" podID="32151649-e3d5-434d-8385-d318db8ade35" containerID="471095009c90e5135edc3e4bfbc83d9fb155a16bed636ca1b37f53a5e922abf1" exitCode=0 Oct 29 23:25:25.596235 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:25.595509 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-faa4fc-5f7c674df4-mgk4c" event={"ID":"32151649-e3d5-434d-8385-d318db8ade35","Type":"ContainerDied","Data":"471095009c90e5135edc3e4bfbc83d9fb155a16bed636ca1b37f53a5e922abf1"} Oct 29 23:25:25.596235 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:25.595569 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-faa4fc-5f7c674df4-mgk4c" event={"ID":"32151649-e3d5-434d-8385-d318db8ade35","Type":"ContainerDied","Data":"09fa602e23669d995e8521c514bcc6e5c51f23d962b9ead4db7e46ce742441e7"} Oct 29 23:25:25.596235 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:25.595599 5057 scope.go:122] "RemoveContainer" containerID="471095009c90e5135edc3e4bfbc83d9fb155a16bed636ca1b37f53a5e922abf1" Oct 29 23:25:25.596235 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:25.595765 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-faa4fc-5f7c674df4-mgk4c" Oct 29 23:25:25.613673 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:25.606689 5057 generic.go:356] "Generic (PLEG): container finished" podID="23b3cc65-71ec-4963-9045-175ab7a0b0d5" containerID="cbd1651bf8b60d7555a78a4211056fbd3acf51576bac4f5a1a7d63b08cc92389" exitCode=0 Oct 29 23:25:25.613673 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:25.606795 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-1a8107/test-model-1a8107-kserve-9bc6dcdc-z65tc" event={"ID":"23b3cc65-71ec-4963-9045-175ab7a0b0d5","Type":"ContainerDied","Data":"cbd1651bf8b60d7555a78a4211056fbd3acf51576bac4f5a1a7d63b08cc92389"} Oct 29 23:25:25.613673 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:25.606830 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-1a8107/test-model-1a8107-kserve-9bc6dcdc-z65tc" event={"ID":"23b3cc65-71ec-4963-9045-175ab7a0b0d5","Type":"ContainerDied","Data":"ef1c9511b95015ea2226cd37bdb378d0c98976527721eb5262e9ff7a8ef36ae3"} Oct 29 23:25:25.613673 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:25.606894 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-1a8107/test-model-1a8107-kserve-9bc6dcdc-z65tc" Oct 29 23:25:25.638289 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:25.638258 5057 scope.go:122] "RemoveContainer" containerID="471095009c90e5135edc3e4bfbc83d9fb155a16bed636ca1b37f53a5e922abf1" Oct 29 23:25:25.655817 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:25:25.655757 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"471095009c90e5135edc3e4bfbc83d9fb155a16bed636ca1b37f53a5e922abf1\": container with ID starting with 471095009c90e5135edc3e4bfbc83d9fb155a16bed636ca1b37f53a5e922abf1 not found: ID does not exist" containerID="471095009c90e5135edc3e4bfbc83d9fb155a16bed636ca1b37f53a5e922abf1" Oct 29 23:25:25.655982 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:25.655828 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"471095009c90e5135edc3e4bfbc83d9fb155a16bed636ca1b37f53a5e922abf1"} err="failed to get container status \"471095009c90e5135edc3e4bfbc83d9fb155a16bed636ca1b37f53a5e922abf1\": rpc error: code = NotFound desc = could not find container \"471095009c90e5135edc3e4bfbc83d9fb155a16bed636ca1b37f53a5e922abf1\": container with ID starting with 471095009c90e5135edc3e4bfbc83d9fb155a16bed636ca1b37f53a5e922abf1 not found: ID does not exist" Oct 29 23:25:25.655982 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:25.655875 5057 scope.go:122] "RemoveContainer" containerID="cbd1651bf8b60d7555a78a4211056fbd3acf51576bac4f5a1a7d63b08cc92389" Oct 29 23:25:25.723626 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:25.723476 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-1a8107/test-model-1a8107-kserve-9bc6dcdc-z65tc"] Oct 29 23:25:25.762090 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:25.758462 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-1a8107/test-model-1a8107-kserve-9bc6dcdc-z65tc"] Oct 29 23:25:25.765619 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:25.765219 5057 scope.go:122] "RemoveContainer" containerID="cbd1651bf8b60d7555a78a4211056fbd3acf51576bac4f5a1a7d63b08cc92389" Oct 29 23:25:25.765744 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:25:25.765620 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cbd1651bf8b60d7555a78a4211056fbd3acf51576bac4f5a1a7d63b08cc92389\": container with ID starting with cbd1651bf8b60d7555a78a4211056fbd3acf51576bac4f5a1a7d63b08cc92389 not found: ID does not exist" containerID="cbd1651bf8b60d7555a78a4211056fbd3acf51576bac4f5a1a7d63b08cc92389" Oct 29 23:25:25.765744 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:25.765651 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cbd1651bf8b60d7555a78a4211056fbd3acf51576bac4f5a1a7d63b08cc92389"} err="failed to get container status \"cbd1651bf8b60d7555a78a4211056fbd3acf51576bac4f5a1a7d63b08cc92389\": rpc error: code = NotFound desc = could not find container \"cbd1651bf8b60d7555a78a4211056fbd3acf51576bac4f5a1a7d63b08cc92389\": container with ID starting with cbd1651bf8b60d7555a78a4211056fbd3acf51576bac4f5a1a7d63b08cc92389 not found: ID does not exist" Oct 29 23:25:25.767802 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:25.767778 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-faa4fc-5f7c674df4-mgk4c"] Oct 29 23:25:25.773657 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:25.773630 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-faa4fc-5f7c674df4-mgk4c"] Oct 29 23:25:25.938031 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:25.937945 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:25:26.048023 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:26.047988 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:25:26.100417 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:26.100376 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f4ce70-ql59q"] Oct 29 23:25:26.109133 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:26.109102 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f4ce70-ql59q"] Oct 29 23:25:26.152659 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:26.152626 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:25:26.586909 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:26.586878 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:25:26.713355 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:26.711001 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wcqcf"] Oct 29 23:25:26.713355 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:26.711294 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wcqcf" podUID="91fd5f87-ec89-46b8-8d57-e07cfec8a5b7" containerName="discovery" containerID="cri-o://88623bf82f730409d092c1c267bcccb1362d460ad50721336562d0e9d6152b8a" gracePeriod=30 Oct 29 23:25:26.919423 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:26.918699 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-qbx4b" Oct 29 23:25:26.931809 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:26.930471 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7dbc6f5-3755-4686-b1af-4fb84d22db3a-kube-api-access-4jbn5\" (UniqueName: \"kubernetes.io/projected/e7dbc6f5-3755-4686-b1af-4fb84d22db3a-kube-api-access-4jbn5\") pod \"e7dbc6f5-3755-4686-b1af-4fb84d22db3a\" (UID: \"e7dbc6f5-3755-4686-b1af-4fb84d22db3a\") " Oct 29 23:25:26.931809 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:26.930627 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7dbc6f5-3755-4686-b1af-4fb84d22db3a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7dbc6f5-3755-4686-b1af-4fb84d22db3a-credential-socket\") pod \"e7dbc6f5-3755-4686-b1af-4fb84d22db3a\" (UID: \"e7dbc6f5-3755-4686-b1af-4fb84d22db3a\") " Oct 29 23:25:26.931809 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:26.930658 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7dbc6f5-3755-4686-b1af-4fb84d22db3a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7dbc6f5-3755-4686-b1af-4fb84d22db3a-workload-certs\") pod \"e7dbc6f5-3755-4686-b1af-4fb84d22db3a\" (UID: \"e7dbc6f5-3755-4686-b1af-4fb84d22db3a\") " Oct 29 23:25:26.931809 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:26.930708 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7dbc6f5-3755-4686-b1af-4fb84d22db3a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7dbc6f5-3755-4686-b1af-4fb84d22db3a-workload-socket\") pod \"e7dbc6f5-3755-4686-b1af-4fb84d22db3a\" (UID: \"e7dbc6f5-3755-4686-b1af-4fb84d22db3a\") " Oct 29 23:25:26.931809 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:26.930769 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7dbc6f5-3755-4686-b1af-4fb84d22db3a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7dbc6f5-3755-4686-b1af-4fb84d22db3a-istio-data\") pod \"e7dbc6f5-3755-4686-b1af-4fb84d22db3a\" (UID: \"e7dbc6f5-3755-4686-b1af-4fb84d22db3a\") " Oct 29 23:25:26.931809 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:26.930802 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7dbc6f5-3755-4686-b1af-4fb84d22db3a-istio-token\" (UniqueName: \"kubernetes.io/projected/e7dbc6f5-3755-4686-b1af-4fb84d22db3a-istio-token\") pod \"e7dbc6f5-3755-4686-b1af-4fb84d22db3a\" (UID: \"e7dbc6f5-3755-4686-b1af-4fb84d22db3a\") " Oct 29 23:25:26.931809 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:26.930891 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7dbc6f5-3755-4686-b1af-4fb84d22db3a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7dbc6f5-3755-4686-b1af-4fb84d22db3a-istio-envoy\") pod \"e7dbc6f5-3755-4686-b1af-4fb84d22db3a\" (UID: \"e7dbc6f5-3755-4686-b1af-4fb84d22db3a\") " Oct 29 23:25:26.931809 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:26.930935 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e7dbc6f5-3755-4686-b1af-4fb84d22db3a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7dbc6f5-3755-4686-b1af-4fb84d22db3a-istio-podinfo\") pod \"e7dbc6f5-3755-4686-b1af-4fb84d22db3a\" (UID: \"e7dbc6f5-3755-4686-b1af-4fb84d22db3a\") " Oct 29 23:25:26.931809 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:26.931050 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e7dbc6f5-3755-4686-b1af-4fb84d22db3a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7dbc6f5-3755-4686-b1af-4fb84d22db3a-istiod-ca-cert\") pod \"e7dbc6f5-3755-4686-b1af-4fb84d22db3a\" (UID: \"e7dbc6f5-3755-4686-b1af-4fb84d22db3a\") " Oct 29 23:25:26.931809 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:26.931405 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7dbc6f5-3755-4686-b1af-4fb84d22db3a-workload-socket" pod "e7dbc6f5-3755-4686-b1af-4fb84d22db3a" (UID: "e7dbc6f5-3755-4686-b1af-4fb84d22db3a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:25:26.931809 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:26.931602 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7dbc6f5-3755-4686-b1af-4fb84d22db3a-credential-socket" pod "e7dbc6f5-3755-4686-b1af-4fb84d22db3a" (UID: "e7dbc6f5-3755-4686-b1af-4fb84d22db3a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:25:26.932568 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:26.931817 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7dbc6f5-3755-4686-b1af-4fb84d22db3a-istio-data" pod "e7dbc6f5-3755-4686-b1af-4fb84d22db3a" (UID: "e7dbc6f5-3755-4686-b1af-4fb84d22db3a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:25:26.932568 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:26.931830 5057 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7dbc6f5-3755-4686-b1af-4fb84d22db3a-credential-socket\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:25:26.932568 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:26.931846 5057 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7dbc6f5-3755-4686-b1af-4fb84d22db3a-workload-socket\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:25:26.932568 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:26.932037 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7dbc6f5-3755-4686-b1af-4fb84d22db3a-workload-certs" pod "e7dbc6f5-3755-4686-b1af-4fb84d22db3a" (UID: "e7dbc6f5-3755-4686-b1af-4fb84d22db3a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:25:26.932568 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:26.932061 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7dbc6f5-3755-4686-b1af-4fb84d22db3a-istiod-ca-cert" pod "e7dbc6f5-3755-4686-b1af-4fb84d22db3a" (UID: "e7dbc6f5-3755-4686-b1af-4fb84d22db3a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:25:26.933766 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:26.933695 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7dbc6f5-3755-4686-b1af-4fb84d22db3a-kube-api-access-4jbn5" pod "e7dbc6f5-3755-4686-b1af-4fb84d22db3a" (UID: "e7dbc6f5-3755-4686-b1af-4fb84d22db3a"). InnerVolumeSpecName "kube-api-access-4jbn5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:25:26.933766 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:26.933735 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7dbc6f5-3755-4686-b1af-4fb84d22db3a-istio-token" pod "e7dbc6f5-3755-4686-b1af-4fb84d22db3a" (UID: "e7dbc6f5-3755-4686-b1af-4fb84d22db3a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:25:26.934005 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:26.933831 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e7dbc6f5-3755-4686-b1af-4fb84d22db3a-istio-podinfo" pod "e7dbc6f5-3755-4686-b1af-4fb84d22db3a" (UID: "e7dbc6f5-3755-4686-b1af-4fb84d22db3a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:25:26.934668 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:26.934646 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7dbc6f5-3755-4686-b1af-4fb84d22db3a-istio-envoy" pod "e7dbc6f5-3755-4686-b1af-4fb84d22db3a" (UID: "e7dbc6f5-3755-4686-b1af-4fb84d22db3a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:25:27.032979 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:27.032948 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4jbn5\" (UniqueName: \"kubernetes.io/projected/e7dbc6f5-3755-4686-b1af-4fb84d22db3a-kube-api-access-4jbn5\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:25:27.032979 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:27.032978 5057 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7dbc6f5-3755-4686-b1af-4fb84d22db3a-workload-certs\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:25:27.032979 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:27.032988 5057 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7dbc6f5-3755-4686-b1af-4fb84d22db3a-istio-data\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:25:27.033181 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:27.032997 5057 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7dbc6f5-3755-4686-b1af-4fb84d22db3a-istio-token\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:25:27.033181 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:27.033005 5057 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7dbc6f5-3755-4686-b1af-4fb84d22db3a-istio-envoy\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:25:27.033181 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:27.033013 5057 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7dbc6f5-3755-4686-b1af-4fb84d22db3a-istio-podinfo\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:25:27.033181 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:27.033020 5057 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7dbc6f5-3755-4686-b1af-4fb84d22db3a-istiod-ca-cert\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:25:27.048263 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:27.048234 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wcqcf" Oct 29 23:25:27.134069 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:27.133986 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/91fd5f87-ec89-46b8-8d57-e07cfec8a5b7-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/91fd5f87-ec89-46b8-8d57-e07cfec8a5b7-istio-csr-dns-cert\") pod \"91fd5f87-ec89-46b8-8d57-e07cfec8a5b7\" (UID: \"91fd5f87-ec89-46b8-8d57-e07cfec8a5b7\") " Oct 29 23:25:27.134226 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:27.134117 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/91fd5f87-ec89-46b8-8d57-e07cfec8a5b7-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/91fd5f87-ec89-46b8-8d57-e07cfec8a5b7-istio-kubeconfig\") pod \"91fd5f87-ec89-46b8-8d57-e07cfec8a5b7\" (UID: \"91fd5f87-ec89-46b8-8d57-e07cfec8a5b7\") " Oct 29 23:25:27.134313 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:27.134272 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91fd5f87-ec89-46b8-8d57-e07cfec8a5b7-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/91fd5f87-ec89-46b8-8d57-e07cfec8a5b7-istio-csr-ca-configmap\") pod \"91fd5f87-ec89-46b8-8d57-e07cfec8a5b7\" (UID: \"91fd5f87-ec89-46b8-8d57-e07cfec8a5b7\") " Oct 29 23:25:27.134407 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:27.134386 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91fd5f87-ec89-46b8-8d57-e07cfec8a5b7-kube-api-access-7zf5s\" (UniqueName: \"kubernetes.io/projected/91fd5f87-ec89-46b8-8d57-e07cfec8a5b7-kube-api-access-7zf5s\") pod \"91fd5f87-ec89-46b8-8d57-e07cfec8a5b7\" (UID: \"91fd5f87-ec89-46b8-8d57-e07cfec8a5b7\") " Oct 29 23:25:27.134681 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:27.134652 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91fd5f87-ec89-46b8-8d57-e07cfec8a5b7-istio-csr-ca-configmap" pod "91fd5f87-ec89-46b8-8d57-e07cfec8a5b7" (UID: "91fd5f87-ec89-46b8-8d57-e07cfec8a5b7"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:25:27.134791 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:27.134771 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91fd5f87-ec89-46b8-8d57-e07cfec8a5b7-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/91fd5f87-ec89-46b8-8d57-e07cfec8a5b7-local-certs\") pod \"91fd5f87-ec89-46b8-8d57-e07cfec8a5b7\" (UID: \"91fd5f87-ec89-46b8-8d57-e07cfec8a5b7\") " Oct 29 23:25:27.135213 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:27.135190 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91fd5f87-ec89-46b8-8d57-e07cfec8a5b7-istio-token\" (UniqueName: \"kubernetes.io/projected/91fd5f87-ec89-46b8-8d57-e07cfec8a5b7-istio-token\") pod \"91fd5f87-ec89-46b8-8d57-e07cfec8a5b7\" (UID: \"91fd5f87-ec89-46b8-8d57-e07cfec8a5b7\") " Oct 29 23:25:27.135781 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:27.135755 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/91fd5f87-ec89-46b8-8d57-e07cfec8a5b7-cacerts\" (UniqueName: \"kubernetes.io/secret/91fd5f87-ec89-46b8-8d57-e07cfec8a5b7-cacerts\") pod \"91fd5f87-ec89-46b8-8d57-e07cfec8a5b7\" (UID: \"91fd5f87-ec89-46b8-8d57-e07cfec8a5b7\") " Oct 29 23:25:27.136839 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:27.136807 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91fd5f87-ec89-46b8-8d57-e07cfec8a5b7-istio-kubeconfig" pod "91fd5f87-ec89-46b8-8d57-e07cfec8a5b7" (UID: "91fd5f87-ec89-46b8-8d57-e07cfec8a5b7"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:25:27.137029 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:27.137005 5057 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/91fd5f87-ec89-46b8-8d57-e07cfec8a5b7-istio-csr-ca-configmap\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:25:27.137101 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:27.137034 5057 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/91fd5f87-ec89-46b8-8d57-e07cfec8a5b7-istio-kubeconfig\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:25:27.138514 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:27.138488 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91fd5f87-ec89-46b8-8d57-e07cfec8a5b7-cacerts" pod "91fd5f87-ec89-46b8-8d57-e07cfec8a5b7" (UID: "91fd5f87-ec89-46b8-8d57-e07cfec8a5b7"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:25:27.138597 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:27.138543 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91fd5f87-ec89-46b8-8d57-e07cfec8a5b7-istio-csr-dns-cert" pod "91fd5f87-ec89-46b8-8d57-e07cfec8a5b7" (UID: "91fd5f87-ec89-46b8-8d57-e07cfec8a5b7"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:25:27.138597 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:27.138577 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91fd5f87-ec89-46b8-8d57-e07cfec8a5b7-local-certs" pod "91fd5f87-ec89-46b8-8d57-e07cfec8a5b7" (UID: "91fd5f87-ec89-46b8-8d57-e07cfec8a5b7"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:25:27.138689 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:27.138586 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91fd5f87-ec89-46b8-8d57-e07cfec8a5b7-kube-api-access-7zf5s" pod "91fd5f87-ec89-46b8-8d57-e07cfec8a5b7" (UID: "91fd5f87-ec89-46b8-8d57-e07cfec8a5b7"). InnerVolumeSpecName "kube-api-access-7zf5s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:25:27.139024 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:27.139003 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91fd5f87-ec89-46b8-8d57-e07cfec8a5b7-istio-token" pod "91fd5f87-ec89-46b8-8d57-e07cfec8a5b7" (UID: "91fd5f87-ec89-46b8-8d57-e07cfec8a5b7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:25:27.238193 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:27.238157 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7zf5s\" (UniqueName: \"kubernetes.io/projected/91fd5f87-ec89-46b8-8d57-e07cfec8a5b7-kube-api-access-7zf5s\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:25:27.238193 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:27.238190 5057 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/91fd5f87-ec89-46b8-8d57-e07cfec8a5b7-local-certs\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:25:27.238193 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:27.238200 5057 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91fd5f87-ec89-46b8-8d57-e07cfec8a5b7-istio-token\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:25:27.238438 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:27.238209 5057 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/91fd5f87-ec89-46b8-8d57-e07cfec8a5b7-cacerts\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:25:27.238438 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:27.238217 5057 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/91fd5f87-ec89-46b8-8d57-e07cfec8a5b7-istio-csr-dns-cert\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:25:27.333407 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:27.333371 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23b3cc65-71ec-4963-9045-175ab7a0b0d5" path="/var/lib/kubelet/pods/23b3cc65-71ec-4963-9045-175ab7a0b0d5/volumes" Oct 29 23:25:27.333861 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:27.333847 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32151649-e3d5-434d-8385-d318db8ade35" path="/var/lib/kubelet/pods/32151649-e3d5-434d-8385-d318db8ade35/volumes" Oct 29 23:25:27.334268 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:27.334255 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e8a5db2-c4f1-42de-ab4c-344a862ecec3" path="/var/lib/kubelet/pods/3e8a5db2-c4f1-42de-ab4c-344a862ecec3/volumes" Oct 29 23:25:27.636657 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:27.636620 5057 generic.go:356] "Generic (PLEG): container finished" podID="e7dbc6f5-3755-4686-b1af-4fb84d22db3a" containerID="84195371efdc4d7172b68a9dc56f842bf74b50f4a8573093a1b0ea4dd9aad2b1" exitCode=0 Oct 29 23:25:27.636801 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:27.636668 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-qbx4b" Oct 29 23:25:27.636801 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:27.636703 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-qbx4b" event={"ID":"e7dbc6f5-3755-4686-b1af-4fb84d22db3a","Type":"ContainerDied","Data":"84195371efdc4d7172b68a9dc56f842bf74b50f4a8573093a1b0ea4dd9aad2b1"} Oct 29 23:25:27.636801 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:27.636762 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-qbx4b" event={"ID":"e7dbc6f5-3755-4686-b1af-4fb84d22db3a","Type":"ContainerDied","Data":"3496d3a5a685fd5d2ebfd5601b8df785d50cf47504f304b362105d3b4a5cb85b"} Oct 29 23:25:27.636801 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:27.636794 5057 scope.go:122] "RemoveContainer" containerID="84195371efdc4d7172b68a9dc56f842bf74b50f4a8573093a1b0ea4dd9aad2b1" Oct 29 23:25:27.638521 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:27.638489 5057 generic.go:356] "Generic (PLEG): container finished" podID="91fd5f87-ec89-46b8-8d57-e07cfec8a5b7" containerID="88623bf82f730409d092c1c267bcccb1362d460ad50721336562d0e9d6152b8a" exitCode=0 Oct 29 23:25:27.638647 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:27.638554 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wcqcf" Oct 29 23:25:27.638647 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:27.638571 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wcqcf" event={"ID":"91fd5f87-ec89-46b8-8d57-e07cfec8a5b7","Type":"ContainerDied","Data":"88623bf82f730409d092c1c267bcccb1362d460ad50721336562d0e9d6152b8a"} Oct 29 23:25:27.638647 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:27.638619 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wcqcf" event={"ID":"91fd5f87-ec89-46b8-8d57-e07cfec8a5b7","Type":"ContainerDied","Data":"ef6891e6d359d56e36b70a629a9ac243ca3835693505e8a2bf08bf62615e89fb"} Oct 29 23:25:27.649297 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:27.649279 5057 scope.go:122] "RemoveContainer" containerID="84195371efdc4d7172b68a9dc56f842bf74b50f4a8573093a1b0ea4dd9aad2b1" Oct 29 23:25:27.649598 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:25:27.649574 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84195371efdc4d7172b68a9dc56f842bf74b50f4a8573093a1b0ea4dd9aad2b1\": container with ID starting with 84195371efdc4d7172b68a9dc56f842bf74b50f4a8573093a1b0ea4dd9aad2b1 not found: ID does not exist" containerID="84195371efdc4d7172b68a9dc56f842bf74b50f4a8573093a1b0ea4dd9aad2b1" Oct 29 23:25:27.649692 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:27.649609 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84195371efdc4d7172b68a9dc56f842bf74b50f4a8573093a1b0ea4dd9aad2b1"} err="failed to get container status \"84195371efdc4d7172b68a9dc56f842bf74b50f4a8573093a1b0ea4dd9aad2b1\": rpc error: code = NotFound desc = could not find container \"84195371efdc4d7172b68a9dc56f842bf74b50f4a8573093a1b0ea4dd9aad2b1\": container with ID starting with 84195371efdc4d7172b68a9dc56f842bf74b50f4a8573093a1b0ea4dd9aad2b1 not found: ID does not exist" Oct 29 23:25:27.649692 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:27.649632 5057 scope.go:122] "RemoveContainer" containerID="88623bf82f730409d092c1c267bcccb1362d460ad50721336562d0e9d6152b8a" Oct 29 23:25:27.665519 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:27.665497 5057 scope.go:122] "RemoveContainer" containerID="88623bf82f730409d092c1c267bcccb1362d460ad50721336562d0e9d6152b8a" Oct 29 23:25:27.665934 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:25:27.665909 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88623bf82f730409d092c1c267bcccb1362d460ad50721336562d0e9d6152b8a\": container with ID starting with 88623bf82f730409d092c1c267bcccb1362d460ad50721336562d0e9d6152b8a not found: ID does not exist" containerID="88623bf82f730409d092c1c267bcccb1362d460ad50721336562d0e9d6152b8a" Oct 29 23:25:27.666015 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:27.665946 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88623bf82f730409d092c1c267bcccb1362d460ad50721336562d0e9d6152b8a"} err="failed to get container status \"88623bf82f730409d092c1c267bcccb1362d460ad50721336562d0e9d6152b8a\": rpc error: code = NotFound desc = could not find container \"88623bf82f730409d092c1c267bcccb1362d460ad50721336562d0e9d6152b8a\": container with ID starting with 88623bf82f730409d092c1c267bcccb1362d460ad50721336562d0e9d6152b8a not found: ID does not exist" Oct 29 23:25:27.691955 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:27.691929 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-qbx4b"] Oct 29 23:25:27.706578 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:27.706548 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-qbx4b"] Oct 29 23:25:27.733987 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:27.733960 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wcqcf"] Oct 29 23:25:27.740049 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:27.740019 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wcqcf"] Oct 29 23:25:27.940183 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:27.940104 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:25:28.269526 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:28.269495 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-dv8l2"] Oct 29 23:25:28.269989 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:28.269712 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-dv8l2" podUID="a0d8f2a6-a6b5-44dd-99fb-38aef41899e4" containerName="istio-proxy" containerID="cri-o://edb8f1cc17a92e3ed0cb999632507fc77704b3a8071b599d7aae9050d179ddf2" gracePeriod=30 Oct 29 23:25:28.456355 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:28.456307 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:25:29.347420 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:29.347385 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91fd5f87-ec89-46b8-8d57-e07cfec8a5b7" path="/var/lib/kubelet/pods/91fd5f87-ec89-46b8-8d57-e07cfec8a5b7/volumes" Oct 29 23:25:29.348077 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:29.348056 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e7dbc6f5-3755-4686-b1af-4fb84d22db3a" path="/var/lib/kubelet/pods/e7dbc6f5-3755-4686-b1af-4fb84d22db3a/volumes" Oct 29 23:25:29.788894 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:29.788856 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:25:31.046757 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:31.046720 5057 patch_prober.go:29] interesting pod/e2e-worker-w5-b2921b-openshift-default-5547c88fff-dv8l2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 23:25:31.047268 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:31.046786 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-dv8l2" podUID="a0d8f2a6-a6b5-44dd-99fb-38aef41899e4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 23:25:31.401306 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:31.401228 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:25:32.957600 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:32.957559 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:25:33.065997 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:33.065962 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:25:33.372879 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:33.372841 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:25:33.704346 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:33.704301 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-dv8l2" Oct 29 23:25:33.708994 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:33.708968 5057 generic.go:356] "Generic (PLEG): container finished" podID="a0d8f2a6-a6b5-44dd-99fb-38aef41899e4" containerID="edb8f1cc17a92e3ed0cb999632507fc77704b3a8071b599d7aae9050d179ddf2" exitCode=0 Oct 29 23:25:33.709143 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:33.709015 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-dv8l2" Oct 29 23:25:33.709143 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:33.709054 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-dv8l2" event={"ID":"a0d8f2a6-a6b5-44dd-99fb-38aef41899e4","Type":"ContainerDied","Data":"edb8f1cc17a92e3ed0cb999632507fc77704b3a8071b599d7aae9050d179ddf2"} Oct 29 23:25:33.709143 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:33.709119 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-dv8l2" event={"ID":"a0d8f2a6-a6b5-44dd-99fb-38aef41899e4","Type":"ContainerDied","Data":"7b7bdbf94b72dc8c9a557b42fa35cbc6d775153ad9dabe0b410d27f80094ab61"} Oct 29 23:25:33.709261 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:33.709150 5057 scope.go:122] "RemoveContainer" containerID="edb8f1cc17a92e3ed0cb999632507fc77704b3a8071b599d7aae9050d179ddf2" Oct 29 23:25:33.721394 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:33.721373 5057 scope.go:122] "RemoveContainer" containerID="edb8f1cc17a92e3ed0cb999632507fc77704b3a8071b599d7aae9050d179ddf2" Oct 29 23:25:33.721727 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:25:33.721702 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"edb8f1cc17a92e3ed0cb999632507fc77704b3a8071b599d7aae9050d179ddf2\": container with ID starting with edb8f1cc17a92e3ed0cb999632507fc77704b3a8071b599d7aae9050d179ddf2 not found: ID does not exist" containerID="edb8f1cc17a92e3ed0cb999632507fc77704b3a8071b599d7aae9050d179ddf2" Oct 29 23:25:33.721796 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:33.721739 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"edb8f1cc17a92e3ed0cb999632507fc77704b3a8071b599d7aae9050d179ddf2"} err="failed to get container status \"edb8f1cc17a92e3ed0cb999632507fc77704b3a8071b599d7aae9050d179ddf2\": rpc error: code = NotFound desc = could not find container \"edb8f1cc17a92e3ed0cb999632507fc77704b3a8071b599d7aae9050d179ddf2\": container with ID starting with edb8f1cc17a92e3ed0cb999632507fc77704b3a8071b599d7aae9050d179ddf2 not found: ID does not exist" Oct 29 23:25:33.762464 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:33.762435 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0d8f2a6-a6b5-44dd-99fb-38aef41899e4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0d8f2a6-a6b5-44dd-99fb-38aef41899e4-workload-certs\") pod \"a0d8f2a6-a6b5-44dd-99fb-38aef41899e4\" (UID: \"a0d8f2a6-a6b5-44dd-99fb-38aef41899e4\") " Oct 29 23:25:33.762639 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:33.762530 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0d8f2a6-a6b5-44dd-99fb-38aef41899e4-kube-api-access-pxjrb\" (UniqueName: \"kubernetes.io/projected/a0d8f2a6-a6b5-44dd-99fb-38aef41899e4-kube-api-access-pxjrb\") pod \"a0d8f2a6-a6b5-44dd-99fb-38aef41899e4\" (UID: \"a0d8f2a6-a6b5-44dd-99fb-38aef41899e4\") " Oct 29 23:25:33.762639 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:33.762589 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0d8f2a6-a6b5-44dd-99fb-38aef41899e4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0d8f2a6-a6b5-44dd-99fb-38aef41899e4-istio-envoy\") pod \"a0d8f2a6-a6b5-44dd-99fb-38aef41899e4\" (UID: \"a0d8f2a6-a6b5-44dd-99fb-38aef41899e4\") " Oct 29 23:25:33.762772 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:33.762687 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0d8f2a6-a6b5-44dd-99fb-38aef41899e4-workload-certs" pod "a0d8f2a6-a6b5-44dd-99fb-38aef41899e4" (UID: "a0d8f2a6-a6b5-44dd-99fb-38aef41899e4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:25:33.762772 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:33.762714 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a0d8f2a6-a6b5-44dd-99fb-38aef41899e4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0d8f2a6-a6b5-44dd-99fb-38aef41899e4-istio-podinfo\") pod \"a0d8f2a6-a6b5-44dd-99fb-38aef41899e4\" (UID: \"a0d8f2a6-a6b5-44dd-99fb-38aef41899e4\") " Oct 29 23:25:33.762890 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:33.762773 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0d8f2a6-a6b5-44dd-99fb-38aef41899e4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0d8f2a6-a6b5-44dd-99fb-38aef41899e4-credential-socket\") pod \"a0d8f2a6-a6b5-44dd-99fb-38aef41899e4\" (UID: \"a0d8f2a6-a6b5-44dd-99fb-38aef41899e4\") " Oct 29 23:25:33.763032 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:33.762950 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0d8f2a6-a6b5-44dd-99fb-38aef41899e4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0d8f2a6-a6b5-44dd-99fb-38aef41899e4-workload-socket\") pod \"a0d8f2a6-a6b5-44dd-99fb-38aef41899e4\" (UID: \"a0d8f2a6-a6b5-44dd-99fb-38aef41899e4\") " Oct 29 23:25:33.763032 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:33.762994 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0d8f2a6-a6b5-44dd-99fb-38aef41899e4-credential-socket" pod "a0d8f2a6-a6b5-44dd-99fb-38aef41899e4" (UID: "a0d8f2a6-a6b5-44dd-99fb-38aef41899e4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:25:33.765916 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:33.763084 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0d8f2a6-a6b5-44dd-99fb-38aef41899e4-istio-token\" (UniqueName: \"kubernetes.io/projected/a0d8f2a6-a6b5-44dd-99fb-38aef41899e4-istio-token\") pod \"a0d8f2a6-a6b5-44dd-99fb-38aef41899e4\" (UID: \"a0d8f2a6-a6b5-44dd-99fb-38aef41899e4\") " Oct 29 23:25:33.765916 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:33.763180 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0d8f2a6-a6b5-44dd-99fb-38aef41899e4-workload-socket" pod "a0d8f2a6-a6b5-44dd-99fb-38aef41899e4" (UID: "a0d8f2a6-a6b5-44dd-99fb-38aef41899e4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:25:33.765916 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:33.763179 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0d8f2a6-a6b5-44dd-99fb-38aef41899e4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0d8f2a6-a6b5-44dd-99fb-38aef41899e4-istio-data\") pod \"a0d8f2a6-a6b5-44dd-99fb-38aef41899e4\" (UID: \"a0d8f2a6-a6b5-44dd-99fb-38aef41899e4\") " Oct 29 23:25:33.765916 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:33.763371 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a0d8f2a6-a6b5-44dd-99fb-38aef41899e4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0d8f2a6-a6b5-44dd-99fb-38aef41899e4-istiod-ca-cert\") pod \"a0d8f2a6-a6b5-44dd-99fb-38aef41899e4\" (UID: \"a0d8f2a6-a6b5-44dd-99fb-38aef41899e4\") " Oct 29 23:25:33.765916 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:33.764396 5057 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0d8f2a6-a6b5-44dd-99fb-38aef41899e4-credential-socket\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:25:33.765916 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:33.764434 5057 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0d8f2a6-a6b5-44dd-99fb-38aef41899e4-workload-socket\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:25:33.765916 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:33.764451 5057 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0d8f2a6-a6b5-44dd-99fb-38aef41899e4-workload-certs\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:25:33.765916 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:33.764613 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0d8f2a6-a6b5-44dd-99fb-38aef41899e4-istio-data" pod "a0d8f2a6-a6b5-44dd-99fb-38aef41899e4" (UID: "a0d8f2a6-a6b5-44dd-99fb-38aef41899e4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:25:33.765916 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:33.764574 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0d8f2a6-a6b5-44dd-99fb-38aef41899e4-istiod-ca-cert" pod "a0d8f2a6-a6b5-44dd-99fb-38aef41899e4" (UID: "a0d8f2a6-a6b5-44dd-99fb-38aef41899e4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:25:33.765916 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:33.765543 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a0d8f2a6-a6b5-44dd-99fb-38aef41899e4-istio-podinfo" pod "a0d8f2a6-a6b5-44dd-99fb-38aef41899e4" (UID: "a0d8f2a6-a6b5-44dd-99fb-38aef41899e4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:25:33.765916 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:33.765879 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0d8f2a6-a6b5-44dd-99fb-38aef41899e4-kube-api-access-pxjrb" pod "a0d8f2a6-a6b5-44dd-99fb-38aef41899e4" (UID: "a0d8f2a6-a6b5-44dd-99fb-38aef41899e4"). InnerVolumeSpecName "kube-api-access-pxjrb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:25:33.767059 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:33.767034 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0d8f2a6-a6b5-44dd-99fb-38aef41899e4-istio-token" pod "a0d8f2a6-a6b5-44dd-99fb-38aef41899e4" (UID: "a0d8f2a6-a6b5-44dd-99fb-38aef41899e4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:25:33.769784 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:33.767828 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0d8f2a6-a6b5-44dd-99fb-38aef41899e4-istio-envoy" pod "a0d8f2a6-a6b5-44dd-99fb-38aef41899e4" (UID: "a0d8f2a6-a6b5-44dd-99fb-38aef41899e4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:25:33.865635 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:33.865606 5057 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0d8f2a6-a6b5-44dd-99fb-38aef41899e4-istio-token\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:25:33.865635 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:33.865635 5057 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0d8f2a6-a6b5-44dd-99fb-38aef41899e4-istio-data\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:25:33.865864 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:33.865650 5057 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0d8f2a6-a6b5-44dd-99fb-38aef41899e4-istiod-ca-cert\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:25:33.865864 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:33.865660 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pxjrb\" (UniqueName: \"kubernetes.io/projected/a0d8f2a6-a6b5-44dd-99fb-38aef41899e4-kube-api-access-pxjrb\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:25:33.865864 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:33.865669 5057 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0d8f2a6-a6b5-44dd-99fb-38aef41899e4-istio-envoy\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:25:33.865864 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:33.865678 5057 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0d8f2a6-a6b5-44dd-99fb-38aef41899e4-istio-podinfo\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:25:34.062905 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:34.062875 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-dv8l2"] Oct 29 23:25:34.068000 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:34.067969 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-dv8l2"] Oct 29 23:25:34.435308 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:34.435225 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:25:34.556034 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:34.555995 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:25:34.661682 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:34.661648 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-62lhb"] Oct 29 23:25:34.724505 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:34.724465 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-62lhb" podUID="21f95b5c-4c31-4836-a8ba-41dc2949edea" containerName="istio-proxy" containerID="cri-o://b644e48c01c6ec554b2034fed0bc7a97e610c06b5895fe0e34337dc8d360c311" gracePeriod=30 Oct 29 23:25:34.816234 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:34.816198 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:25:35.334453 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:35.334422 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0d8f2a6-a6b5-44dd-99fb-38aef41899e4" path="/var/lib/kubelet/pods/a0d8f2a6-a6b5-44dd-99fb-38aef41899e4/volumes" Oct 29 23:25:35.859188 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:35.859149 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:25:36.509531 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:36.509492 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:25:37.261714 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:37.261677 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:25:37.403275 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:37.403242 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:25:37.448853 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:37.448820 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a9ae14-c5qxl"] Oct 29 23:25:37.459935 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:37.459904 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a9ae14-c5qxl"] Oct 29 23:25:37.569472 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:37.569390 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:25:37.957388 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:37.957357 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:25:38.661940 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:38.661901 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-pk92s"] Oct 29 23:25:38.662417 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:38.662114 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-pk92s" podUID="9033e6ee-4fd2-41a2-8a7d-132b6ca20128" containerName="istio-proxy" containerID="cri-o://9c6dd6a2b4c881e2526567b87b68d980e665674e84932f76ecd320f922734f62" gracePeriod=30 Oct 29 23:25:39.334408 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:39.334373 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="99d91e9c-a637-4834-b840-bb0bc29bc66d" path="/var/lib/kubelet/pods/99d91e9c-a637-4834-b840-bb0bc29bc66d/volumes" Oct 29 23:25:39.633608 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:39.633524 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:25:40.034077 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:40.034050 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-62lhb" Oct 29 23:25:40.171966 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:40.171879 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21f95b5c-4c31-4836-a8ba-41dc2949edea-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21f95b5c-4c31-4836-a8ba-41dc2949edea-workload-socket\") pod \"21f95b5c-4c31-4836-a8ba-41dc2949edea\" (UID: \"21f95b5c-4c31-4836-a8ba-41dc2949edea\") " Oct 29 23:25:40.171966 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:40.171958 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21f95b5c-4c31-4836-a8ba-41dc2949edea-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21f95b5c-4c31-4836-a8ba-41dc2949edea-workload-certs\") pod \"21f95b5c-4c31-4836-a8ba-41dc2949edea\" (UID: \"21f95b5c-4c31-4836-a8ba-41dc2949edea\") " Oct 29 23:25:40.172210 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:40.172026 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21f95b5c-4c31-4836-a8ba-41dc2949edea-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21f95b5c-4c31-4836-a8ba-41dc2949edea-istio-envoy\") pod \"21f95b5c-4c31-4836-a8ba-41dc2949edea\" (UID: \"21f95b5c-4c31-4836-a8ba-41dc2949edea\") " Oct 29 23:25:40.172210 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:40.172118 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21f95b5c-4c31-4836-a8ba-41dc2949edea-workload-socket" pod "21f95b5c-4c31-4836-a8ba-41dc2949edea" (UID: "21f95b5c-4c31-4836-a8ba-41dc2949edea"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:25:40.172210 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:40.172126 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/21f95b5c-4c31-4836-a8ba-41dc2949edea-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21f95b5c-4c31-4836-a8ba-41dc2949edea-istio-podinfo\") pod \"21f95b5c-4c31-4836-a8ba-41dc2949edea\" (UID: \"21f95b5c-4c31-4836-a8ba-41dc2949edea\") " Oct 29 23:25:40.172210 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:40.172186 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21f95b5c-4c31-4836-a8ba-41dc2949edea-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21f95b5c-4c31-4836-a8ba-41dc2949edea-credential-socket\") pod \"21f95b5c-4c31-4836-a8ba-41dc2949edea\" (UID: \"21f95b5c-4c31-4836-a8ba-41dc2949edea\") " Oct 29 23:25:40.172480 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:40.172253 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21f95b5c-4c31-4836-a8ba-41dc2949edea-kube-api-access-wrkfn\" (UniqueName: \"kubernetes.io/projected/21f95b5c-4c31-4836-a8ba-41dc2949edea-kube-api-access-wrkfn\") pod \"21f95b5c-4c31-4836-a8ba-41dc2949edea\" (UID: \"21f95b5c-4c31-4836-a8ba-41dc2949edea\") " Oct 29 23:25:40.172480 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:40.172264 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21f95b5c-4c31-4836-a8ba-41dc2949edea-workload-certs" pod "21f95b5c-4c31-4836-a8ba-41dc2949edea" (UID: "21f95b5c-4c31-4836-a8ba-41dc2949edea"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:25:40.172480 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:40.172286 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21f95b5c-4c31-4836-a8ba-41dc2949edea-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21f95b5c-4c31-4836-a8ba-41dc2949edea-istio-data\") pod \"21f95b5c-4c31-4836-a8ba-41dc2949edea\" (UID: \"21f95b5c-4c31-4836-a8ba-41dc2949edea\") " Oct 29 23:25:40.172480 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:40.172426 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21f95b5c-4c31-4836-a8ba-41dc2949edea-istio-token\" (UniqueName: \"kubernetes.io/projected/21f95b5c-4c31-4836-a8ba-41dc2949edea-istio-token\") pod \"21f95b5c-4c31-4836-a8ba-41dc2949edea\" (UID: \"21f95b5c-4c31-4836-a8ba-41dc2949edea\") " Oct 29 23:25:40.172710 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:40.172485 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21f95b5c-4c31-4836-a8ba-41dc2949edea-credential-socket" pod "21f95b5c-4c31-4836-a8ba-41dc2949edea" (UID: "21f95b5c-4c31-4836-a8ba-41dc2949edea"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:25:40.172710 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:40.172560 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21f95b5c-4c31-4836-a8ba-41dc2949edea-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21f95b5c-4c31-4836-a8ba-41dc2949edea-istiod-ca-cert\") pod \"21f95b5c-4c31-4836-a8ba-41dc2949edea\" (UID: \"21f95b5c-4c31-4836-a8ba-41dc2949edea\") " Oct 29 23:25:40.172710 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:40.172632 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21f95b5c-4c31-4836-a8ba-41dc2949edea-istio-data" pod "21f95b5c-4c31-4836-a8ba-41dc2949edea" (UID: "21f95b5c-4c31-4836-a8ba-41dc2949edea"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:25:40.173030 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:40.173006 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21f95b5c-4c31-4836-a8ba-41dc2949edea-istiod-ca-cert" pod "21f95b5c-4c31-4836-a8ba-41dc2949edea" (UID: "21f95b5c-4c31-4836-a8ba-41dc2949edea"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:25:40.173379 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:40.173355 5057 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21f95b5c-4c31-4836-a8ba-41dc2949edea-istiod-ca-cert\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:25:40.173523 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:40.173384 5057 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21f95b5c-4c31-4836-a8ba-41dc2949edea-workload-socket\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:25:40.173523 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:40.173399 5057 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21f95b5c-4c31-4836-a8ba-41dc2949edea-workload-certs\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:25:40.173523 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:40.173413 5057 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21f95b5c-4c31-4836-a8ba-41dc2949edea-credential-socket\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:25:40.173523 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:40.173427 5057 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21f95b5c-4c31-4836-a8ba-41dc2949edea-istio-data\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:25:40.174835 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:40.174809 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21f95b5c-4c31-4836-a8ba-41dc2949edea-istio-token" pod "21f95b5c-4c31-4836-a8ba-41dc2949edea" (UID: "21f95b5c-4c31-4836-a8ba-41dc2949edea"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:25:40.174937 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:40.174892 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21f95b5c-4c31-4836-a8ba-41dc2949edea-kube-api-access-wrkfn" pod "21f95b5c-4c31-4836-a8ba-41dc2949edea" (UID: "21f95b5c-4c31-4836-a8ba-41dc2949edea"). InnerVolumeSpecName "kube-api-access-wrkfn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:25:40.175009 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:40.174961 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/21f95b5c-4c31-4836-a8ba-41dc2949edea-istio-podinfo" pod "21f95b5c-4c31-4836-a8ba-41dc2949edea" (UID: "21f95b5c-4c31-4836-a8ba-41dc2949edea"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:25:40.175713 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:40.175690 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21f95b5c-4c31-4836-a8ba-41dc2949edea-istio-envoy" pod "21f95b5c-4c31-4836-a8ba-41dc2949edea" (UID: "21f95b5c-4c31-4836-a8ba-41dc2949edea"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:25:40.274554 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:40.274525 5057 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21f95b5c-4c31-4836-a8ba-41dc2949edea-istio-token\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:25:40.274554 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:40.274553 5057 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21f95b5c-4c31-4836-a8ba-41dc2949edea-istio-envoy\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:25:40.274765 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:40.274565 5057 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21f95b5c-4c31-4836-a8ba-41dc2949edea-istio-podinfo\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:25:40.274765 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:40.274577 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wrkfn\" (UniqueName: \"kubernetes.io/projected/21f95b5c-4c31-4836-a8ba-41dc2949edea-kube-api-access-wrkfn\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:25:40.795477 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:40.795441 5057 generic.go:356] "Generic (PLEG): container finished" podID="21f95b5c-4c31-4836-a8ba-41dc2949edea" containerID="b644e48c01c6ec554b2034fed0bc7a97e610c06b5895fe0e34337dc8d360c311" exitCode=0 Oct 29 23:25:40.795688 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:40.795500 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-62lhb" Oct 29 23:25:40.795688 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:40.795524 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-62lhb" event={"ID":"21f95b5c-4c31-4836-a8ba-41dc2949edea","Type":"ContainerDied","Data":"b644e48c01c6ec554b2034fed0bc7a97e610c06b5895fe0e34337dc8d360c311"} Oct 29 23:25:40.795688 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:40.795594 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-62lhb" event={"ID":"21f95b5c-4c31-4836-a8ba-41dc2949edea","Type":"ContainerDied","Data":"9455feefaf15fe63ece1884b8357e4854c72960e3fad9f5f1537c00896736afb"} Oct 29 23:25:40.795688 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:40.795617 5057 scope.go:122] "RemoveContainer" containerID="b644e48c01c6ec554b2034fed0bc7a97e610c06b5895fe0e34337dc8d360c311" Oct 29 23:25:40.808322 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:40.808302 5057 scope.go:122] "RemoveContainer" containerID="b644e48c01c6ec554b2034fed0bc7a97e610c06b5895fe0e34337dc8d360c311" Oct 29 23:25:40.808664 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:25:40.808645 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b644e48c01c6ec554b2034fed0bc7a97e610c06b5895fe0e34337dc8d360c311\": container with ID starting with b644e48c01c6ec554b2034fed0bc7a97e610c06b5895fe0e34337dc8d360c311 not found: ID does not exist" containerID="b644e48c01c6ec554b2034fed0bc7a97e610c06b5895fe0e34337dc8d360c311" Oct 29 23:25:40.808728 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:40.808672 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b644e48c01c6ec554b2034fed0bc7a97e610c06b5895fe0e34337dc8d360c311"} err="failed to get container status \"b644e48c01c6ec554b2034fed0bc7a97e610c06b5895fe0e34337dc8d360c311\": rpc error: code = NotFound desc = could not find container \"b644e48c01c6ec554b2034fed0bc7a97e610c06b5895fe0e34337dc8d360c311\": container with ID starting with b644e48c01c6ec554b2034fed0bc7a97e610c06b5895fe0e34337dc8d360c311 not found: ID does not exist" Oct 29 23:25:40.862516 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:40.862485 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-62lhb"] Oct 29 23:25:40.874155 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:40.874124 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-62lhb"] Oct 29 23:25:40.905764 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:40.905738 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3b55d0dc-sbbh9"] Oct 29 23:25:40.906711 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:40.906694 5057 cpu_manager.go:414] "containerMap: removing container" podUID="c68bc52c-5a5a-49ca-bd1a-4dc908b4aee0" containerName="payload-pre-processing" Oct 29 23:25:40.906774 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:40.906712 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c68bc52c-5a5a-49ca-bd1a-4dc908b4aee0" containerName="payload-pre-processing" Oct 29 23:25:40.906774 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:40.906736 5057 cpu_manager.go:414] "containerMap: removing container" podUID="a0d8f2a6-a6b5-44dd-99fb-38aef41899e4" containerName="istio-proxy" Oct 29 23:25:40.906774 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:40.906741 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0d8f2a6-a6b5-44dd-99fb-38aef41899e4" containerName="istio-proxy" Oct 29 23:25:40.906774 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:40.906751 5057 cpu_manager.go:414] "containerMap: removing container" podUID="91fd5f87-ec89-46b8-8d57-e07cfec8a5b7" containerName="discovery" Oct 29 23:25:40.906774 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:40.906757 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91fd5f87-ec89-46b8-8d57-e07cfec8a5b7" containerName="discovery" Oct 29 23:25:40.906774 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:40.906769 5057 cpu_manager.go:414] "containerMap: removing container" podUID="21034c72-9803-4050-b5b5-df014d3668e4" containerName="istio-proxy" Oct 29 23:25:40.906774 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:40.906774 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21034c72-9803-4050-b5b5-df014d3668e4" containerName="istio-proxy" Oct 29 23:25:40.906993 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:40.906785 5057 cpu_manager.go:414] "containerMap: removing container" podUID="e7dbc6f5-3755-4686-b1af-4fb84d22db3a" containerName="istio-proxy" Oct 29 23:25:40.906993 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:40.906790 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7dbc6f5-3755-4686-b1af-4fb84d22db3a" containerName="istio-proxy" Oct 29 23:25:40.906993 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:40.906802 5057 cpu_manager.go:414] "containerMap: removing container" podUID="21f95b5c-4c31-4836-a8ba-41dc2949edea" containerName="istio-proxy" Oct 29 23:25:40.906993 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:40.906807 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21f95b5c-4c31-4836-a8ba-41dc2949edea" containerName="istio-proxy" Oct 29 23:25:40.906993 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:40.906815 5057 cpu_manager.go:414] "containerMap: removing container" podUID="32151649-e3d5-434d-8385-d318db8ade35" containerName="payload-processing" Oct 29 23:25:40.906993 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:40.906819 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32151649-e3d5-434d-8385-d318db8ade35" containerName="payload-processing" Oct 29 23:25:40.906993 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:40.906844 5057 cpu_manager.go:414] "containerMap: removing container" podUID="23b3cc65-71ec-4963-9045-175ab7a0b0d5" containerName="main" Oct 29 23:25:40.906993 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:40.906848 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23b3cc65-71ec-4963-9045-175ab7a0b0d5" containerName="main" Oct 29 23:25:40.906993 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:40.906856 5057 cpu_manager.go:414] "containerMap: removing container" podUID="35e18ec0-85f5-497b-8f44-907262416627" containerName="revoke-keys" Oct 29 23:25:40.906993 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:40.906860 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35e18ec0-85f5-497b-8f44-907262416627" containerName="revoke-keys" Oct 29 23:25:40.907313 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:40.907010 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="23b3cc65-71ec-4963-9045-175ab7a0b0d5" containerName="main" Oct 29 23:25:40.907313 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:40.907019 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="35e18ec0-85f5-497b-8f44-907262416627" containerName="revoke-keys" Oct 29 23:25:40.907313 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:40.907029 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="c68bc52c-5a5a-49ca-bd1a-4dc908b4aee0" containerName="payload-pre-processing" Oct 29 23:25:40.907313 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:40.907041 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0d8f2a6-a6b5-44dd-99fb-38aef41899e4" containerName="istio-proxy" Oct 29 23:25:40.907313 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:40.907050 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="91fd5f87-ec89-46b8-8d57-e07cfec8a5b7" containerName="discovery" Oct 29 23:25:40.907313 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:40.907057 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="21034c72-9803-4050-b5b5-df014d3668e4" containerName="istio-proxy" Oct 29 23:25:40.907313 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:40.907065 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7dbc6f5-3755-4686-b1af-4fb84d22db3a" containerName="istio-proxy" Oct 29 23:25:40.907313 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:40.907072 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="21f95b5c-4c31-4836-a8ba-41dc2949edea" containerName="istio-proxy" Oct 29 23:25:40.907313 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:40.907080 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="32151649-e3d5-434d-8385-d318db8ade35" containerName="payload-processing" Oct 29 23:25:40.912470 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:40.912452 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b55d0dc-sbbh9" Oct 29 23:25:40.927555 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:40.927531 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:25:40.927683 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:40.927564 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dmx57\"" Oct 29 23:25:40.929963 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:40.929386 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3b55d0dc-sbbh9"] Oct 29 23:25:40.947272 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:40.947247 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:25:40.988063 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:40.988032 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/288b2965-dc7b-4096-81dd-5369e2f9f3ce-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3b55d0dc-sbbh9\" (UID: \"288b2965-dc7b-4096-81dd-5369e2f9f3ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b55d0dc-sbbh9" Oct 29 23:25:40.988290 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:40.988268 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/288b2965-dc7b-4096-81dd-5369e2f9f3ce-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3b55d0dc-sbbh9\" (UID: \"288b2965-dc7b-4096-81dd-5369e2f9f3ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b55d0dc-sbbh9" Oct 29 23:25:40.988527 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:40.988489 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67xrh\" (UniqueName: \"kubernetes.io/projected/288b2965-dc7b-4096-81dd-5369e2f9f3ce-kube-api-access-67xrh\") pod \"maas-api-revoke-sub-3b55d0dc-sbbh9\" (UID: \"288b2965-dc7b-4096-81dd-5369e2f9f3ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b55d0dc-sbbh9" Oct 29 23:25:41.062895 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:41.062810 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:25:41.089451 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:41.089420 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67xrh\" (UniqueName: \"kubernetes.io/projected/288b2965-dc7b-4096-81dd-5369e2f9f3ce-kube-api-access-67xrh\") pod \"maas-api-revoke-sub-3b55d0dc-sbbh9\" (UID: \"288b2965-dc7b-4096-81dd-5369e2f9f3ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b55d0dc-sbbh9" Oct 29 23:25:41.089618 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:41.089483 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/288b2965-dc7b-4096-81dd-5369e2f9f3ce-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3b55d0dc-sbbh9\" (UID: \"288b2965-dc7b-4096-81dd-5369e2f9f3ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b55d0dc-sbbh9" Oct 29 23:25:41.089729 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:41.089708 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/288b2965-dc7b-4096-81dd-5369e2f9f3ce-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3b55d0dc-sbbh9\" (UID: \"288b2965-dc7b-4096-81dd-5369e2f9f3ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b55d0dc-sbbh9" Oct 29 23:25:41.092376 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:41.090552 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/288b2965-dc7b-4096-81dd-5369e2f9f3ce-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3b55d0dc-sbbh9\" (UID: \"288b2965-dc7b-4096-81dd-5369e2f9f3ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b55d0dc-sbbh9" Oct 29 23:25:41.092376 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:41.092116 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/288b2965-dc7b-4096-81dd-5369e2f9f3ce-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3b55d0dc-sbbh9\" (UID: \"288b2965-dc7b-4096-81dd-5369e2f9f3ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b55d0dc-sbbh9" Oct 29 23:25:41.113034 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:41.113007 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-67xrh\" (UniqueName: \"kubernetes.io/projected/288b2965-dc7b-4096-81dd-5369e2f9f3ce-kube-api-access-67xrh\") pod \"maas-api-revoke-sub-3b55d0dc-sbbh9\" (UID: \"288b2965-dc7b-4096-81dd-5369e2f9f3ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b55d0dc-sbbh9" Oct 29 23:25:41.233158 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:41.233129 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b55d0dc-sbbh9" Oct 29 23:25:41.297101 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:41.297068 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:25:41.367019 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:41.358526 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21f95b5c-4c31-4836-a8ba-41dc2949edea" path="/var/lib/kubelet/pods/21f95b5c-4c31-4836-a8ba-41dc2949edea/volumes" Oct 29 23:25:41.555734 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:41.555697 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3b55d0dc-sbbh9"] Oct 29 23:25:41.806127 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:41.806091 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b55d0dc-sbbh9" event={"ID":"288b2965-dc7b-4096-81dd-5369e2f9f3ce","Type":"ContainerStarted","Data":"a085d6158917a2f76532abeccb63a3ab6c07104cbe7f96321aa95330a7f432f8"} Oct 29 23:25:42.816261 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:42.816223 5057 patch_prober.go:29] interesting pod/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-pk92s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 23:25:42.816740 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:42.816280 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-pk92s" podUID="9033e6ee-4fd2-41a2-8a7d-132b6ca20128" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 23:25:42.818833 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:42.818803 5057 generic.go:356] "Generic (PLEG): container finished" podID="288b2965-dc7b-4096-81dd-5369e2f9f3ce" containerID="4078490a9d95f128f42ba59fe2fed27076cf6d44f3035f5cb99906de470a5bf8" exitCode=0 Oct 29 23:25:42.818968 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:42.818891 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b55d0dc-sbbh9" event={"ID":"288b2965-dc7b-4096-81dd-5369e2f9f3ce","Type":"ContainerDied","Data":"4078490a9d95f128f42ba59fe2fed27076cf6d44f3035f5cb99906de470a5bf8"} Oct 29 23:25:43.003959 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:43.003925 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:25:43.034476 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:43.034440 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd70c84b-f9pp7"] Oct 29 23:25:43.041201 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:43.041172 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd70c84b-f9pp7" Oct 29 23:25:43.061561 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:43.061531 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd70c84b-f9pp7"] Oct 29 23:25:43.121973 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:43.121936 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9d7f5\" (UniqueName: \"kubernetes.io/projected/dbd19eb3-9ab3-473c-9a3c-1f63abca271c-kube-api-access-9d7f5\") pod \"maas-api-revoke-sub-cd70c84b-f9pp7\" (UID: \"dbd19eb3-9ab3-473c-9a3c-1f63abca271c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd70c84b-f9pp7" Oct 29 23:25:43.122920 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:43.122082 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbd19eb3-9ab3-473c-9a3c-1f63abca271c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cd70c84b-f9pp7\" (UID: \"dbd19eb3-9ab3-473c-9a3c-1f63abca271c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd70c84b-f9pp7" Oct 29 23:25:43.122920 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:43.122143 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbd19eb3-9ab3-473c-9a3c-1f63abca271c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cd70c84b-f9pp7\" (UID: \"dbd19eb3-9ab3-473c-9a3c-1f63abca271c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd70c84b-f9pp7" Oct 29 23:25:43.223374 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:43.223344 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbd19eb3-9ab3-473c-9a3c-1f63abca271c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cd70c84b-f9pp7\" (UID: \"dbd19eb3-9ab3-473c-9a3c-1f63abca271c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd70c84b-f9pp7" Oct 29 23:25:43.223541 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:43.223425 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbd19eb3-9ab3-473c-9a3c-1f63abca271c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cd70c84b-f9pp7\" (UID: \"dbd19eb3-9ab3-473c-9a3c-1f63abca271c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd70c84b-f9pp7" Oct 29 23:25:43.223665 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:43.223638 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9d7f5\" (UniqueName: \"kubernetes.io/projected/dbd19eb3-9ab3-473c-9a3c-1f63abca271c-kube-api-access-9d7f5\") pod \"maas-api-revoke-sub-cd70c84b-f9pp7\" (UID: \"dbd19eb3-9ab3-473c-9a3c-1f63abca271c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd70c84b-f9pp7" Oct 29 23:25:43.223999 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:43.223974 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbd19eb3-9ab3-473c-9a3c-1f63abca271c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cd70c84b-f9pp7\" (UID: \"dbd19eb3-9ab3-473c-9a3c-1f63abca271c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd70c84b-f9pp7" Oct 29 23:25:43.225800 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:43.225775 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbd19eb3-9ab3-473c-9a3c-1f63abca271c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cd70c84b-f9pp7\" (UID: \"dbd19eb3-9ab3-473c-9a3c-1f63abca271c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd70c84b-f9pp7" Oct 29 23:25:43.246808 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:43.246782 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9d7f5\" (UniqueName: \"kubernetes.io/projected/dbd19eb3-9ab3-473c-9a3c-1f63abca271c-kube-api-access-9d7f5\") pod \"maas-api-revoke-sub-cd70c84b-f9pp7\" (UID: \"dbd19eb3-9ab3-473c-9a3c-1f63abca271c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd70c84b-f9pp7" Oct 29 23:25:43.367425 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:43.367316 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd70c84b-f9pp7" Oct 29 23:25:43.640152 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:43.640072 5057 scope.go:122] "RemoveContainer" containerID="481b7d38fc7cfd35eba8cf8f267e267c4656064cf9eda4c3d791042a227b2b9c" Oct 29 23:25:43.654931 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:43.654906 5057 scope.go:122] "RemoveContainer" containerID="5f7d9ba17db099a54b9f7485a67459c041680ce4f3dd962e9a3386b5b9788f78" Oct 29 23:25:43.689213 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:43.689182 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd70c84b-f9pp7"] Oct 29 23:25:43.731513 ip-10-0-140-74 kubenswrapper[5057]: W1029 23:25:43.731469 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbd19eb3_9ab3_473c_9a3c_1f63abca271c.slice/crio-b095c2126a0245b886a934509f2e5d72c0e0a28010d60fda28af6cded10d8937 WatchSource:0}: Error finding container b095c2126a0245b886a934509f2e5d72c0e0a28010d60fda28af6cded10d8937: Status 404 returned error can't find the container with id b095c2126a0245b886a934509f2e5d72c0e0a28010d60fda28af6cded10d8937 Oct 29 23:25:43.844408 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:43.844376 5057 generic.go:356] "Generic (PLEG): container finished" podID="9033e6ee-4fd2-41a2-8a7d-132b6ca20128" containerID="9c6dd6a2b4c881e2526567b87b68d980e665674e84932f76ecd320f922734f62" exitCode=0 Oct 29 23:25:43.844784 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:43.844697 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-pk92s" event={"ID":"9033e6ee-4fd2-41a2-8a7d-132b6ca20128","Type":"ContainerDied","Data":"9c6dd6a2b4c881e2526567b87b68d980e665674e84932f76ecd320f922734f62"} Oct 29 23:25:43.847134 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:43.847072 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd70c84b-f9pp7" event={"ID":"dbd19eb3-9ab3-473c-9a3c-1f63abca271c","Type":"ContainerStarted","Data":"b095c2126a0245b886a934509f2e5d72c0e0a28010d60fda28af6cded10d8937"} Oct 29 23:25:43.983760 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:43.983603 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-pk92s" Oct 29 23:25:44.039394 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:44.038746 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9033e6ee-4fd2-41a2-8a7d-132b6ca20128-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9033e6ee-4fd2-41a2-8a7d-132b6ca20128-workload-certs\") pod \"9033e6ee-4fd2-41a2-8a7d-132b6ca20128\" (UID: \"9033e6ee-4fd2-41a2-8a7d-132b6ca20128\") " Oct 29 23:25:44.039394 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:44.039042 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9033e6ee-4fd2-41a2-8a7d-132b6ca20128-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9033e6ee-4fd2-41a2-8a7d-132b6ca20128-istio-podinfo\") pod \"9033e6ee-4fd2-41a2-8a7d-132b6ca20128\" (UID: \"9033e6ee-4fd2-41a2-8a7d-132b6ca20128\") " Oct 29 23:25:44.039394 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:44.039087 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9033e6ee-4fd2-41a2-8a7d-132b6ca20128-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9033e6ee-4fd2-41a2-8a7d-132b6ca20128-istio-envoy\") pod \"9033e6ee-4fd2-41a2-8a7d-132b6ca20128\" (UID: \"9033e6ee-4fd2-41a2-8a7d-132b6ca20128\") " Oct 29 23:25:44.039394 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:44.039135 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9033e6ee-4fd2-41a2-8a7d-132b6ca20128-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9033e6ee-4fd2-41a2-8a7d-132b6ca20128-istio-data\") pod \"9033e6ee-4fd2-41a2-8a7d-132b6ca20128\" (UID: \"9033e6ee-4fd2-41a2-8a7d-132b6ca20128\") " Oct 29 23:25:44.039394 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:44.039180 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9033e6ee-4fd2-41a2-8a7d-132b6ca20128-kube-api-access-wrkk5\" (UniqueName: \"kubernetes.io/projected/9033e6ee-4fd2-41a2-8a7d-132b6ca20128-kube-api-access-wrkk5\") pod \"9033e6ee-4fd2-41a2-8a7d-132b6ca20128\" (UID: \"9033e6ee-4fd2-41a2-8a7d-132b6ca20128\") " Oct 29 23:25:44.039394 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:44.039215 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9033e6ee-4fd2-41a2-8a7d-132b6ca20128-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9033e6ee-4fd2-41a2-8a7d-132b6ca20128-workload-socket\") pod \"9033e6ee-4fd2-41a2-8a7d-132b6ca20128\" (UID: \"9033e6ee-4fd2-41a2-8a7d-132b6ca20128\") " Oct 29 23:25:44.039394 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:44.039305 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9033e6ee-4fd2-41a2-8a7d-132b6ca20128-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9033e6ee-4fd2-41a2-8a7d-132b6ca20128-istiod-ca-cert\") pod \"9033e6ee-4fd2-41a2-8a7d-132b6ca20128\" (UID: \"9033e6ee-4fd2-41a2-8a7d-132b6ca20128\") " Oct 29 23:25:44.039950 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:44.039486 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9033e6ee-4fd2-41a2-8a7d-132b6ca20128-istio-token\" (UniqueName: \"kubernetes.io/projected/9033e6ee-4fd2-41a2-8a7d-132b6ca20128-istio-token\") pod \"9033e6ee-4fd2-41a2-8a7d-132b6ca20128\" (UID: \"9033e6ee-4fd2-41a2-8a7d-132b6ca20128\") " Oct 29 23:25:44.039950 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:44.039559 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9033e6ee-4fd2-41a2-8a7d-132b6ca20128-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9033e6ee-4fd2-41a2-8a7d-132b6ca20128-credential-socket\") pod \"9033e6ee-4fd2-41a2-8a7d-132b6ca20128\" (UID: \"9033e6ee-4fd2-41a2-8a7d-132b6ca20128\") " Oct 29 23:25:44.042413 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:44.040725 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9033e6ee-4fd2-41a2-8a7d-132b6ca20128-credential-socket" pod "9033e6ee-4fd2-41a2-8a7d-132b6ca20128" (UID: "9033e6ee-4fd2-41a2-8a7d-132b6ca20128"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:25:44.042413 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:44.040960 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9033e6ee-4fd2-41a2-8a7d-132b6ca20128-workload-certs" pod "9033e6ee-4fd2-41a2-8a7d-132b6ca20128" (UID: "9033e6ee-4fd2-41a2-8a7d-132b6ca20128"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:25:44.046400 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:44.042705 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9033e6ee-4fd2-41a2-8a7d-132b6ca20128-workload-socket" pod "9033e6ee-4fd2-41a2-8a7d-132b6ca20128" (UID: "9033e6ee-4fd2-41a2-8a7d-132b6ca20128"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:25:44.046400 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:44.043081 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9033e6ee-4fd2-41a2-8a7d-132b6ca20128-istiod-ca-cert" pod "9033e6ee-4fd2-41a2-8a7d-132b6ca20128" (UID: "9033e6ee-4fd2-41a2-8a7d-132b6ca20128"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:25:44.046400 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:44.043534 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9033e6ee-4fd2-41a2-8a7d-132b6ca20128-istio-data" pod "9033e6ee-4fd2-41a2-8a7d-132b6ca20128" (UID: "9033e6ee-4fd2-41a2-8a7d-132b6ca20128"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:25:44.046400 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:44.043847 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9033e6ee-4fd2-41a2-8a7d-132b6ca20128-istio-podinfo" pod "9033e6ee-4fd2-41a2-8a7d-132b6ca20128" (UID: "9033e6ee-4fd2-41a2-8a7d-132b6ca20128"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:25:44.052360 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:44.051821 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9033e6ee-4fd2-41a2-8a7d-132b6ca20128-istio-token" pod "9033e6ee-4fd2-41a2-8a7d-132b6ca20128" (UID: "9033e6ee-4fd2-41a2-8a7d-132b6ca20128"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:25:44.052813 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:44.052725 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9033e6ee-4fd2-41a2-8a7d-132b6ca20128-kube-api-access-wrkk5" pod "9033e6ee-4fd2-41a2-8a7d-132b6ca20128" (UID: "9033e6ee-4fd2-41a2-8a7d-132b6ca20128"). InnerVolumeSpecName "kube-api-access-wrkk5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:25:44.053124 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:44.053097 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9033e6ee-4fd2-41a2-8a7d-132b6ca20128-istio-envoy" pod "9033e6ee-4fd2-41a2-8a7d-132b6ca20128" (UID: "9033e6ee-4fd2-41a2-8a7d-132b6ca20128"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:25:44.067812 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:44.064273 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b55d0dc-sbbh9" Oct 29 23:25:44.141571 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:44.141536 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/288b2965-dc7b-4096-81dd-5369e2f9f3ce-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/288b2965-dc7b-4096-81dd-5369e2f9f3ce-maas-api-openshift-ca\") pod \"288b2965-dc7b-4096-81dd-5369e2f9f3ce\" (UID: \"288b2965-dc7b-4096-81dd-5369e2f9f3ce\") " Oct 29 23:25:44.141763 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:44.141647 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/288b2965-dc7b-4096-81dd-5369e2f9f3ce-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/288b2965-dc7b-4096-81dd-5369e2f9f3ce-maas-api-xks-ca\") pod \"288b2965-dc7b-4096-81dd-5369e2f9f3ce\" (UID: \"288b2965-dc7b-4096-81dd-5369e2f9f3ce\") " Oct 29 23:25:44.141763 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:44.141721 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/288b2965-dc7b-4096-81dd-5369e2f9f3ce-kube-api-access-67xrh\" (UniqueName: \"kubernetes.io/projected/288b2965-dc7b-4096-81dd-5369e2f9f3ce-kube-api-access-67xrh\") pod \"288b2965-dc7b-4096-81dd-5369e2f9f3ce\" (UID: \"288b2965-dc7b-4096-81dd-5369e2f9f3ce\") " Oct 29 23:25:44.141969 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:44.141938 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/288b2965-dc7b-4096-81dd-5369e2f9f3ce-maas-api-openshift-ca" pod "288b2965-dc7b-4096-81dd-5369e2f9f3ce" (UID: "288b2965-dc7b-4096-81dd-5369e2f9f3ce"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:25:44.143854 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:44.143824 5057 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9033e6ee-4fd2-41a2-8a7d-132b6ca20128-istio-podinfo\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:25:44.143979 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:44.143856 5057 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9033e6ee-4fd2-41a2-8a7d-132b6ca20128-istio-envoy\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:25:44.143979 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:44.143872 5057 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/288b2965-dc7b-4096-81dd-5369e2f9f3ce-maas-api-openshift-ca\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:25:44.143979 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:44.143886 5057 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9033e6ee-4fd2-41a2-8a7d-132b6ca20128-istio-data\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:25:44.143979 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:44.143899 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wrkk5\" (UniqueName: \"kubernetes.io/projected/9033e6ee-4fd2-41a2-8a7d-132b6ca20128-kube-api-access-wrkk5\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:25:44.143979 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:44.143912 5057 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9033e6ee-4fd2-41a2-8a7d-132b6ca20128-workload-socket\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:25:44.143979 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:44.143925 5057 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9033e6ee-4fd2-41a2-8a7d-132b6ca20128-istiod-ca-cert\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:25:44.143979 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:44.143937 5057 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9033e6ee-4fd2-41a2-8a7d-132b6ca20128-istio-token\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:25:44.143979 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:44.143950 5057 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9033e6ee-4fd2-41a2-8a7d-132b6ca20128-credential-socket\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:25:44.143979 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:44.143964 5057 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9033e6ee-4fd2-41a2-8a7d-132b6ca20128-workload-certs\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:25:44.144682 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:44.144657 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/288b2965-dc7b-4096-81dd-5369e2f9f3ce-kube-api-access-67xrh" pod "288b2965-dc7b-4096-81dd-5369e2f9f3ce" (UID: "288b2965-dc7b-4096-81dd-5369e2f9f3ce"). InnerVolumeSpecName "kube-api-access-67xrh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:25:44.145296 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:44.145266 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/288b2965-dc7b-4096-81dd-5369e2f9f3ce-maas-api-xks-ca" pod "288b2965-dc7b-4096-81dd-5369e2f9f3ce" (UID: "288b2965-dc7b-4096-81dd-5369e2f9f3ce"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:25:44.245282 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:44.245247 5057 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/288b2965-dc7b-4096-81dd-5369e2f9f3ce-maas-api-xks-ca\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:25:44.245440 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:44.245287 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-67xrh\" (UniqueName: \"kubernetes.io/projected/288b2965-dc7b-4096-81dd-5369e2f9f3ce-kube-api-access-67xrh\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:25:44.386290 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:44.386253 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:25:44.497316 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:44.497234 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:25:44.712346 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:44.712293 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:25:44.818404 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:44.818285 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:25:44.950754 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:44.950715 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-pk92s" event={"ID":"9033e6ee-4fd2-41a2-8a7d-132b6ca20128","Type":"ContainerDied","Data":"40b9e2c0215dee2358c1f1676159574d7c697dca9fbbda58fd701876e5dc48f9"} Oct 29 23:25:44.951286 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:44.950792 5057 scope.go:122] "RemoveContainer" containerID="9c6dd6a2b4c881e2526567b87b68d980e665674e84932f76ecd320f922734f62" Oct 29 23:25:44.951286 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:44.950878 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-pk92s" Oct 29 23:25:44.955185 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:44.954076 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b55d0dc-sbbh9" event={"ID":"288b2965-dc7b-4096-81dd-5369e2f9f3ce","Type":"ContainerDied","Data":"a085d6158917a2f76532abeccb63a3ab6c07104cbe7f96321aa95330a7f432f8"} Oct 29 23:25:44.955185 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:44.954127 5057 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a085d6158917a2f76532abeccb63a3ab6c07104cbe7f96321aa95330a7f432f8" Oct 29 23:25:44.955185 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:44.954199 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b55d0dc-sbbh9" Oct 29 23:25:44.958477 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:44.958453 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd70c84b-f9pp7" event={"ID":"dbd19eb3-9ab3-473c-9a3c-1f63abca271c","Type":"ContainerStarted","Data":"065a3eb120a6a150bf4db4a0e291f76138871419a9559ebd0507bca41e8f9a46"} Oct 29 23:25:45.020789 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:45.020731 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd70c84b-f9pp7" podStartSLOduration=2.020712038 podStartE2EDuration="2.020712038s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:25:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:25:45.017149073 +0000 UTC m=+3125.800362511" watchObservedRunningTime="2026-10-29 23:25:45.020712038 +0000 UTC m=+3125.803925464" Oct 29 23:25:45.057275 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:45.057244 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-pk92s"] Oct 29 23:25:45.068158 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:45.068132 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-pk92s"] Oct 29 23:25:45.336127 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:45.336037 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9033e6ee-4fd2-41a2-8a7d-132b6ca20128" path="/var/lib/kubelet/pods/9033e6ee-4fd2-41a2-8a7d-132b6ca20128/volumes" Oct 29 23:25:45.737063 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:45.737022 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:25:45.854771 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:45.854734 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:25:45.971275 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:45.971238 5057 generic.go:356] "Generic (PLEG): container finished" podID="dbd19eb3-9ab3-473c-9a3c-1f63abca271c" containerID="065a3eb120a6a150bf4db4a0e291f76138871419a9559ebd0507bca41e8f9a46" exitCode=0 Oct 29 23:25:45.971739 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:45.971297 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd70c84b-f9pp7" event={"ID":"dbd19eb3-9ab3-473c-9a3c-1f63abca271c","Type":"ContainerDied","Data":"065a3eb120a6a150bf4db4a0e291f76138871419a9559ebd0507bca41e8f9a46"} Oct 29 23:25:46.525877 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:46.525841 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:25:46.949939 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:46.949910 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:25:47.083732 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:47.083691 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:25:47.171456 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:47.171430 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd70c84b-f9pp7" Oct 29 23:25:47.238429 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:47.238355 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:25:47.346449 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:47.346418 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dbd19eb3-9ab3-473c-9a3c-1f63abca271c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbd19eb3-9ab3-473c-9a3c-1f63abca271c-maas-api-openshift-ca\") pod \"dbd19eb3-9ab3-473c-9a3c-1f63abca271c\" (UID: \"dbd19eb3-9ab3-473c-9a3c-1f63abca271c\") " Oct 29 23:25:47.346626 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:47.346604 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dbd19eb3-9ab3-473c-9a3c-1f63abca271c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbd19eb3-9ab3-473c-9a3c-1f63abca271c-maas-api-xks-ca\") pod \"dbd19eb3-9ab3-473c-9a3c-1f63abca271c\" (UID: \"dbd19eb3-9ab3-473c-9a3c-1f63abca271c\") " Oct 29 23:25:47.346742 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:47.346728 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dbd19eb3-9ab3-473c-9a3c-1f63abca271c-kube-api-access-9d7f5\" (UniqueName: \"kubernetes.io/projected/dbd19eb3-9ab3-473c-9a3c-1f63abca271c-kube-api-access-9d7f5\") pod \"dbd19eb3-9ab3-473c-9a3c-1f63abca271c\" (UID: \"dbd19eb3-9ab3-473c-9a3c-1f63abca271c\") " Oct 29 23:25:47.346866 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:47.346727 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbd19eb3-9ab3-473c-9a3c-1f63abca271c-maas-api-openshift-ca" pod "dbd19eb3-9ab3-473c-9a3c-1f63abca271c" (UID: "dbd19eb3-9ab3-473c-9a3c-1f63abca271c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:25:47.347299 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:47.347276 5057 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbd19eb3-9ab3-473c-9a3c-1f63abca271c-maas-api-openshift-ca\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:25:47.349257 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:47.349236 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dbd19eb3-9ab3-473c-9a3c-1f63abca271c-maas-api-xks-ca" pod "dbd19eb3-9ab3-473c-9a3c-1f63abca271c" (UID: "dbd19eb3-9ab3-473c-9a3c-1f63abca271c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:25:47.349355 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:47.349268 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbd19eb3-9ab3-473c-9a3c-1f63abca271c-kube-api-access-9d7f5" pod "dbd19eb3-9ab3-473c-9a3c-1f63abca271c" (UID: "dbd19eb3-9ab3-473c-9a3c-1f63abca271c"). InnerVolumeSpecName "kube-api-access-9d7f5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:25:47.449503 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:47.449471 5057 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbd19eb3-9ab3-473c-9a3c-1f63abca271c-maas-api-xks-ca\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:25:47.449503 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:47.449505 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9d7f5\" (UniqueName: \"kubernetes.io/projected/dbd19eb3-9ab3-473c-9a3c-1f63abca271c-kube-api-access-9d7f5\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:25:47.991128 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:47.991094 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd70c84b-f9pp7" Oct 29 23:25:47.991323 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:47.991121 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd70c84b-f9pp7" event={"ID":"dbd19eb3-9ab3-473c-9a3c-1f63abca271c","Type":"ContainerDied","Data":"b095c2126a0245b886a934509f2e5d72c0e0a28010d60fda28af6cded10d8937"} Oct 29 23:25:47.991323 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:47.991204 5057 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b095c2126a0245b886a934509f2e5d72c0e0a28010d60fda28af6cded10d8937" Oct 29 23:25:48.284179 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:48.284090 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:25:49.030322 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:25:49.030286 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d2dnn" containerName="extractor" Oct 29 23:25:49.078101 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:49.078067 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6210df6f-lcb4g"] Oct 29 23:25:49.079085 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:49.079067 5057 cpu_manager.go:414] "containerMap: removing container" podUID="288b2965-dc7b-4096-81dd-5369e2f9f3ce" containerName="revoke-keys" Oct 29 23:25:49.079085 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:49.079085 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="288b2965-dc7b-4096-81dd-5369e2f9f3ce" containerName="revoke-keys" Oct 29 23:25:49.079210 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:49.079102 5057 cpu_manager.go:414] "containerMap: removing container" podUID="9033e6ee-4fd2-41a2-8a7d-132b6ca20128" containerName="istio-proxy" Oct 29 23:25:49.079210 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:49.079109 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9033e6ee-4fd2-41a2-8a7d-132b6ca20128" containerName="istio-proxy" Oct 29 23:25:49.079210 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:49.079153 5057 cpu_manager.go:414] "containerMap: removing container" podUID="dbd19eb3-9ab3-473c-9a3c-1f63abca271c" containerName="revoke-keys" Oct 29 23:25:49.079210 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:49.079159 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dbd19eb3-9ab3-473c-9a3c-1f63abca271c" containerName="revoke-keys" Oct 29 23:25:49.079349 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:49.079289 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="288b2965-dc7b-4096-81dd-5369e2f9f3ce" containerName="revoke-keys" Oct 29 23:25:49.079349 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:49.079298 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="9033e6ee-4fd2-41a2-8a7d-132b6ca20128" containerName="istio-proxy" Oct 29 23:25:49.079349 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:49.079314 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="dbd19eb3-9ab3-473c-9a3c-1f63abca271c" containerName="revoke-keys" Oct 29 23:25:49.083772 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:49.083753 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6210df6f-lcb4g" Oct 29 23:25:49.109012 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:49.108978 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6210df6f-lcb4g"] Oct 29 23:25:49.109266 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:49.109249 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:25:49.109418 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:49.109390 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dmx57\"" Oct 29 23:25:49.282148 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:49.282068 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hrn8\" (UniqueName: \"kubernetes.io/projected/40d8fc20-6f9a-42af-b255-ce95dd53761b-kube-api-access-4hrn8\") pod \"maas-api-revoke-sub-6210df6f-lcb4g\" (UID: \"40d8fc20-6f9a-42af-b255-ce95dd53761b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6210df6f-lcb4g" Oct 29 23:25:49.282299 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:49.282213 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40d8fc20-6f9a-42af-b255-ce95dd53761b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6210df6f-lcb4g\" (UID: \"40d8fc20-6f9a-42af-b255-ce95dd53761b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6210df6f-lcb4g" Oct 29 23:25:49.282399 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:49.282380 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40d8fc20-6f9a-42af-b255-ce95dd53761b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6210df6f-lcb4g\" (UID: \"40d8fc20-6f9a-42af-b255-ce95dd53761b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6210df6f-lcb4g" Oct 29 23:25:49.383529 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:49.383496 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40d8fc20-6f9a-42af-b255-ce95dd53761b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6210df6f-lcb4g\" (UID: \"40d8fc20-6f9a-42af-b255-ce95dd53761b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6210df6f-lcb4g" Oct 29 23:25:49.383994 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:49.383626 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hrn8\" (UniqueName: \"kubernetes.io/projected/40d8fc20-6f9a-42af-b255-ce95dd53761b-kube-api-access-4hrn8\") pod \"maas-api-revoke-sub-6210df6f-lcb4g\" (UID: \"40d8fc20-6f9a-42af-b255-ce95dd53761b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6210df6f-lcb4g" Oct 29 23:25:49.383994 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:49.383767 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40d8fc20-6f9a-42af-b255-ce95dd53761b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6210df6f-lcb4g\" (UID: \"40d8fc20-6f9a-42af-b255-ce95dd53761b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6210df6f-lcb4g" Oct 29 23:25:49.384199 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:49.384177 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40d8fc20-6f9a-42af-b255-ce95dd53761b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6210df6f-lcb4g\" (UID: \"40d8fc20-6f9a-42af-b255-ce95dd53761b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6210df6f-lcb4g" Oct 29 23:25:49.386128 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:49.386097 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40d8fc20-6f9a-42af-b255-ce95dd53761b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6210df6f-lcb4g\" (UID: \"40d8fc20-6f9a-42af-b255-ce95dd53761b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6210df6f-lcb4g" Oct 29 23:25:49.406552 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:49.406527 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hrn8\" (UniqueName: \"kubernetes.io/projected/40d8fc20-6f9a-42af-b255-ce95dd53761b-kube-api-access-4hrn8\") pod \"maas-api-revoke-sub-6210df6f-lcb4g\" (UID: \"40d8fc20-6f9a-42af-b255-ce95dd53761b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6210df6f-lcb4g" Oct 29 23:25:49.706081 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:49.706048 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6210df6f-lcb4g" Oct 29 23:25:49.710582 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:49.710549 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:25:50.017997 ip-10-0-140-74 kubenswrapper[5057]: W1029 23:25:50.017956 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40d8fc20_6f9a_42af_b255_ce95dd53761b.slice/crio-7f3baa9e3228afee4cad6f6460f186327fc410715a3a08744a57ddbb157603ff WatchSource:0}: Error finding container 7f3baa9e3228afee4cad6f6460f186327fc410715a3a08744a57ddbb157603ff: Status 404 returned error can't find the container with id 7f3baa9e3228afee4cad6f6460f186327fc410715a3a08744a57ddbb157603ff Oct 29 23:25:50.019628 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:50.019479 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6210df6f-lcb4g"] Oct 29 23:25:50.465514 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:50.465459 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:25:50.591764 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:50.591687 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:25:50.701930 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:50.701895 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:25:50.820052 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:50.820017 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:25:51.024395 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:51.024363 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6210df6f-lcb4g" event={"ID":"40d8fc20-6f9a-42af-b255-ce95dd53761b","Type":"ContainerStarted","Data":"2aa3748d9c17668fdf2b746b7c8bd3a199c63187f15fa925fd200885e2de4744"} Oct 29 23:25:51.024581 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:51.024416 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6210df6f-lcb4g" event={"ID":"40d8fc20-6f9a-42af-b255-ce95dd53761b","Type":"ContainerStarted","Data":"7f3baa9e3228afee4cad6f6460f186327fc410715a3a08744a57ddbb157603ff"} Oct 29 23:25:51.067320 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:51.067262 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6210df6f-lcb4g" podStartSLOduration=2.067247958 podStartE2EDuration="2.067247958s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:25:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:25:51.064179388 +0000 UTC m=+3131.847392831" watchObservedRunningTime="2026-10-29 23:25:51.067247958 +0000 UTC m=+3131.850461384" Oct 29 23:25:51.086108 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:51.086077 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:25:51.925549 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:51.925500 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:25:52.040598 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:52.040522 5057 generic.go:356] "Generic (PLEG): container finished" podID="40d8fc20-6f9a-42af-b255-ce95dd53761b" containerID="2aa3748d9c17668fdf2b746b7c8bd3a199c63187f15fa925fd200885e2de4744" exitCode=0 Oct 29 23:25:52.040598 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:52.040599 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6210df6f-lcb4g" event={"ID":"40d8fc20-6f9a-42af-b255-ce95dd53761b","Type":"ContainerDied","Data":"2aa3748d9c17668fdf2b746b7c8bd3a199c63187f15fa925fd200885e2de4744"} Oct 29 23:25:52.512816 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:52.512780 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96f97c8a-5hxsl"] Oct 29 23:25:52.518135 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:52.518114 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96f97c8a-5hxsl" Oct 29 23:25:52.543381 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:52.543352 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96f97c8a-5hxsl"] Oct 29 23:25:52.558070 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:52.558036 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7l2q\" (UniqueName: \"kubernetes.io/projected/baaefff9-3dbc-4f65-80bb-f02ac48ae739-kube-api-access-p7l2q\") pod \"maas-api-revoke-sub-96f97c8a-5hxsl\" (UID: \"baaefff9-3dbc-4f65-80bb-f02ac48ae739\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96f97c8a-5hxsl" Oct 29 23:25:52.558207 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:52.558093 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/baaefff9-3dbc-4f65-80bb-f02ac48ae739-maas-api-xks-ca\") pod \"maas-api-revoke-sub-96f97c8a-5hxsl\" (UID: \"baaefff9-3dbc-4f65-80bb-f02ac48ae739\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96f97c8a-5hxsl" Oct 29 23:25:52.558256 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:52.558234 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/baaefff9-3dbc-4f65-80bb-f02ac48ae739-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-96f97c8a-5hxsl\" (UID: \"baaefff9-3dbc-4f65-80bb-f02ac48ae739\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96f97c8a-5hxsl" Oct 29 23:25:52.603051 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:52.603020 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:25:52.659766 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:52.659732 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7l2q\" (UniqueName: \"kubernetes.io/projected/baaefff9-3dbc-4f65-80bb-f02ac48ae739-kube-api-access-p7l2q\") pod \"maas-api-revoke-sub-96f97c8a-5hxsl\" (UID: \"baaefff9-3dbc-4f65-80bb-f02ac48ae739\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96f97c8a-5hxsl" Oct 29 23:25:52.659982 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:52.659802 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/baaefff9-3dbc-4f65-80bb-f02ac48ae739-maas-api-xks-ca\") pod \"maas-api-revoke-sub-96f97c8a-5hxsl\" (UID: \"baaefff9-3dbc-4f65-80bb-f02ac48ae739\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96f97c8a-5hxsl" Oct 29 23:25:52.659982 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:52.659967 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/baaefff9-3dbc-4f65-80bb-f02ac48ae739-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-96f97c8a-5hxsl\" (UID: \"baaefff9-3dbc-4f65-80bb-f02ac48ae739\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96f97c8a-5hxsl" Oct 29 23:25:52.660951 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:52.660922 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/baaefff9-3dbc-4f65-80bb-f02ac48ae739-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-96f97c8a-5hxsl\" (UID: \"baaefff9-3dbc-4f65-80bb-f02ac48ae739\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96f97c8a-5hxsl" Oct 29 23:25:52.662774 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:52.662747 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/baaefff9-3dbc-4f65-80bb-f02ac48ae739-maas-api-xks-ca\") pod \"maas-api-revoke-sub-96f97c8a-5hxsl\" (UID: \"baaefff9-3dbc-4f65-80bb-f02ac48ae739\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96f97c8a-5hxsl" Oct 29 23:25:52.684822 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:52.684787 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7l2q\" (UniqueName: \"kubernetes.io/projected/baaefff9-3dbc-4f65-80bb-f02ac48ae739-kube-api-access-p7l2q\") pod \"maas-api-revoke-sub-96f97c8a-5hxsl\" (UID: \"baaefff9-3dbc-4f65-80bb-f02ac48ae739\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96f97c8a-5hxsl" Oct 29 23:25:52.838094 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:52.838007 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96f97c8a-5hxsl" Oct 29 23:25:53.187281 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:53.187245 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96f97c8a-5hxsl"] Oct 29 23:25:53.226674 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:53.226634 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6210df6f-lcb4g" Oct 29 23:25:53.271615 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:53.271580 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40d8fc20-6f9a-42af-b255-ce95dd53761b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40d8fc20-6f9a-42af-b255-ce95dd53761b-maas-api-openshift-ca\") pod \"40d8fc20-6f9a-42af-b255-ce95dd53761b\" (UID: \"40d8fc20-6f9a-42af-b255-ce95dd53761b\") " Oct 29 23:25:53.271807 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:53.271674 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40d8fc20-6f9a-42af-b255-ce95dd53761b-kube-api-access-4hrn8\" (UniqueName: \"kubernetes.io/projected/40d8fc20-6f9a-42af-b255-ce95dd53761b-kube-api-access-4hrn8\") pod \"40d8fc20-6f9a-42af-b255-ce95dd53761b\" (UID: \"40d8fc20-6f9a-42af-b255-ce95dd53761b\") " Oct 29 23:25:53.271807 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:53.271718 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/40d8fc20-6f9a-42af-b255-ce95dd53761b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40d8fc20-6f9a-42af-b255-ce95dd53761b-maas-api-xks-ca\") pod \"40d8fc20-6f9a-42af-b255-ce95dd53761b\" (UID: \"40d8fc20-6f9a-42af-b255-ce95dd53761b\") " Oct 29 23:25:53.273456 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:53.273426 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40d8fc20-6f9a-42af-b255-ce95dd53761b-maas-api-openshift-ca" pod "40d8fc20-6f9a-42af-b255-ce95dd53761b" (UID: "40d8fc20-6f9a-42af-b255-ce95dd53761b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:25:53.275318 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:53.275291 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40d8fc20-6f9a-42af-b255-ce95dd53761b-kube-api-access-4hrn8" pod "40d8fc20-6f9a-42af-b255-ce95dd53761b" (UID: "40d8fc20-6f9a-42af-b255-ce95dd53761b"). InnerVolumeSpecName "kube-api-access-4hrn8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:25:53.275530 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:53.275313 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40d8fc20-6f9a-42af-b255-ce95dd53761b-maas-api-xks-ca" pod "40d8fc20-6f9a-42af-b255-ce95dd53761b" (UID: "40d8fc20-6f9a-42af-b255-ce95dd53761b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:25:53.374652 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:53.374571 5057 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40d8fc20-6f9a-42af-b255-ce95dd53761b-maas-api-openshift-ca\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:25:53.374652 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:53.374603 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4hrn8\" (UniqueName: \"kubernetes.io/projected/40d8fc20-6f9a-42af-b255-ce95dd53761b-kube-api-access-4hrn8\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:25:53.374652 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:53.374617 5057 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40d8fc20-6f9a-42af-b255-ce95dd53761b-maas-api-xks-ca\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:25:54.059151 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:54.059121 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6210df6f-lcb4g" Oct 29 23:25:54.059374 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:54.059158 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6210df6f-lcb4g" event={"ID":"40d8fc20-6f9a-42af-b255-ce95dd53761b","Type":"ContainerDied","Data":"7f3baa9e3228afee4cad6f6460f186327fc410715a3a08744a57ddbb157603ff"} Oct 29 23:25:54.059374 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:54.059214 5057 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7f3baa9e3228afee4cad6f6460f186327fc410715a3a08744a57ddbb157603ff" Oct 29 23:25:54.061032 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:54.061004 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96f97c8a-5hxsl" event={"ID":"baaefff9-3dbc-4f65-80bb-f02ac48ae739","Type":"ContainerStarted","Data":"3dd80ee3e0a5d80a630f55b79ec459e16e2a890babd9b1c6368bc8cb1360d9b6"} Oct 29 23:25:54.061176 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:54.061053 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96f97c8a-5hxsl" event={"ID":"baaefff9-3dbc-4f65-80bb-f02ac48ae739","Type":"ContainerStarted","Data":"9dd3a6376c69561d9c31cfddf5f7610d154045db01085b0aa63fe3e545ee43a9"} Oct 29 23:25:54.102126 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:54.102083 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96f97c8a-5hxsl" podStartSLOduration=2.102070716 podStartE2EDuration="2.102070716s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:25:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:25:54.101542746 +0000 UTC m=+3134.884756173" watchObservedRunningTime="2026-10-29 23:25:54.102070716 +0000 UTC m=+3134.885284142" Oct 29 23:25:54.192881 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:54.192851 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:25:55.076596 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:55.076560 5057 generic.go:356] "Generic (PLEG): container finished" podID="baaefff9-3dbc-4f65-80bb-f02ac48ae739" containerID="3dd80ee3e0a5d80a630f55b79ec459e16e2a890babd9b1c6368bc8cb1360d9b6" exitCode=0 Oct 29 23:25:55.076818 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:55.076663 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96f97c8a-5hxsl" event={"ID":"baaefff9-3dbc-4f65-80bb-f02ac48ae739","Type":"ContainerDied","Data":"3dd80ee3e0a5d80a630f55b79ec459e16e2a890babd9b1c6368bc8cb1360d9b6"} Oct 29 23:25:55.717312 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:55.717272 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:25:56.284809 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:56.284785 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96f97c8a-5hxsl" Oct 29 23:25:56.320401 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:56.320370 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/baaefff9-3dbc-4f65-80bb-f02ac48ae739-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/baaefff9-3dbc-4f65-80bb-f02ac48ae739-maas-api-openshift-ca\") pod \"baaefff9-3dbc-4f65-80bb-f02ac48ae739\" (UID: \"baaefff9-3dbc-4f65-80bb-f02ac48ae739\") " Oct 29 23:25:56.320573 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:56.320432 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/baaefff9-3dbc-4f65-80bb-f02ac48ae739-kube-api-access-p7l2q\" (UniqueName: \"kubernetes.io/projected/baaefff9-3dbc-4f65-80bb-f02ac48ae739-kube-api-access-p7l2q\") pod \"baaefff9-3dbc-4f65-80bb-f02ac48ae739\" (UID: \"baaefff9-3dbc-4f65-80bb-f02ac48ae739\") " Oct 29 23:25:56.320647 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:56.320582 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/baaefff9-3dbc-4f65-80bb-f02ac48ae739-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/baaefff9-3dbc-4f65-80bb-f02ac48ae739-maas-api-xks-ca\") pod \"baaefff9-3dbc-4f65-80bb-f02ac48ae739\" (UID: \"baaefff9-3dbc-4f65-80bb-f02ac48ae739\") " Oct 29 23:25:56.320798 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:56.320767 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/baaefff9-3dbc-4f65-80bb-f02ac48ae739-maas-api-openshift-ca" pod "baaefff9-3dbc-4f65-80bb-f02ac48ae739" (UID: "baaefff9-3dbc-4f65-80bb-f02ac48ae739"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:25:56.321237 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:56.321218 5057 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/baaefff9-3dbc-4f65-80bb-f02ac48ae739-maas-api-openshift-ca\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:25:56.322916 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:56.322886 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/baaefff9-3dbc-4f65-80bb-f02ac48ae739-maas-api-xks-ca" pod "baaefff9-3dbc-4f65-80bb-f02ac48ae739" (UID: "baaefff9-3dbc-4f65-80bb-f02ac48ae739"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:25:56.323195 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:56.323166 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/baaefff9-3dbc-4f65-80bb-f02ac48ae739-kube-api-access-p7l2q" pod "baaefff9-3dbc-4f65-80bb-f02ac48ae739" (UID: "baaefff9-3dbc-4f65-80bb-f02ac48ae739"). InnerVolumeSpecName "kube-api-access-p7l2q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:25:56.422463 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:56.422416 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p7l2q\" (UniqueName: \"kubernetes.io/projected/baaefff9-3dbc-4f65-80bb-f02ac48ae739-kube-api-access-p7l2q\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:25:56.422463 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:56.422452 5057 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/baaefff9-3dbc-4f65-80bb-f02ac48ae739-maas-api-xks-ca\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:25:57.098942 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:57.098910 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96f97c8a-5hxsl" Oct 29 23:25:57.099394 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:57.098939 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96f97c8a-5hxsl" event={"ID":"baaefff9-3dbc-4f65-80bb-f02ac48ae739","Type":"ContainerDied","Data":"9dd3a6376c69561d9c31cfddf5f7610d154045db01085b0aa63fe3e545ee43a9"} Oct 29 23:25:57.099394 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:57.098990 5057 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9dd3a6376c69561d9c31cfddf5f7610d154045db01085b0aa63fe3e545ee43a9" Oct 29 23:25:57.442906 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:57.442823 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:25:58.035104 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:58.035070 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:25:58.177559 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:58.177512 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:25:58.286969 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:58.286882 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:25:58.523780 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:58.523747 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:25:59.075478 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:59.075444 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1a8107-b79sh"] Oct 29 23:25:59.076443 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:59.076418 5057 cpu_manager.go:414] "containerMap: removing container" podUID="baaefff9-3dbc-4f65-80bb-f02ac48ae739" containerName="revoke-keys" Oct 29 23:25:59.076443 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:59.076438 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="baaefff9-3dbc-4f65-80bb-f02ac48ae739" containerName="revoke-keys" Oct 29 23:25:59.076590 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:59.076456 5057 cpu_manager.go:414] "containerMap: removing container" podUID="40d8fc20-6f9a-42af-b255-ce95dd53761b" containerName="revoke-keys" Oct 29 23:25:59.076590 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:59.076462 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40d8fc20-6f9a-42af-b255-ce95dd53761b" containerName="revoke-keys" Oct 29 23:25:59.076672 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:59.076661 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="baaefff9-3dbc-4f65-80bb-f02ac48ae739" containerName="revoke-keys" Oct 29 23:25:59.076705 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:59.076676 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="40d8fc20-6f9a-42af-b255-ce95dd53761b" containerName="revoke-keys" Oct 29 23:25:59.089806 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:59.089785 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1a8107-b79sh" Oct 29 23:25:59.100043 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:59.100016 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1a8107-b79sh"] Oct 29 23:25:59.173912 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:59.173881 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:25:59.174244 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:59.174223 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dmx57\"" Oct 29 23:25:59.174776 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:59.174752 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8v54j\" (UniqueName: \"kubernetes.io/projected/e0ea5d85-1751-4c56-ab9c-b51366e93322-kube-api-access-8v54j\") pod \"maas-api-revoke-keys-e2e-inf-b-1a8107-b79sh\" (UID: \"e0ea5d85-1751-4c56-ab9c-b51366e93322\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1a8107-b79sh" Oct 29 23:25:59.174871 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:59.174801 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0ea5d85-1751-4c56-ab9c-b51366e93322-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-1a8107-b79sh\" (UID: \"e0ea5d85-1751-4c56-ab9c-b51366e93322\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1a8107-b79sh" Oct 29 23:25:59.174935 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:59.174913 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0ea5d85-1751-4c56-ab9c-b51366e93322-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-1a8107-b79sh\" (UID: \"e0ea5d85-1751-4c56-ab9c-b51366e93322\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1a8107-b79sh" Oct 29 23:25:59.276121 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:59.276089 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8v54j\" (UniqueName: \"kubernetes.io/projected/e0ea5d85-1751-4c56-ab9c-b51366e93322-kube-api-access-8v54j\") pod \"maas-api-revoke-keys-e2e-inf-b-1a8107-b79sh\" (UID: \"e0ea5d85-1751-4c56-ab9c-b51366e93322\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1a8107-b79sh" Oct 29 23:25:59.276602 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:59.276134 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0ea5d85-1751-4c56-ab9c-b51366e93322-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-1a8107-b79sh\" (UID: \"e0ea5d85-1751-4c56-ab9c-b51366e93322\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1a8107-b79sh" Oct 29 23:25:59.276602 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:59.276181 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0ea5d85-1751-4c56-ab9c-b51366e93322-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-1a8107-b79sh\" (UID: \"e0ea5d85-1751-4c56-ab9c-b51366e93322\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1a8107-b79sh" Oct 29 23:25:59.276863 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:59.276839 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0ea5d85-1751-4c56-ab9c-b51366e93322-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-1a8107-b79sh\" (UID: \"e0ea5d85-1751-4c56-ab9c-b51366e93322\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1a8107-b79sh" Oct 29 23:25:59.278723 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:59.278693 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0ea5d85-1751-4c56-ab9c-b51366e93322-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-1a8107-b79sh\" (UID: \"e0ea5d85-1751-4c56-ab9c-b51366e93322\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1a8107-b79sh" Oct 29 23:25:59.299375 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:59.299317 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8v54j\" (UniqueName: \"kubernetes.io/projected/e0ea5d85-1751-4c56-ab9c-b51366e93322-kube-api-access-8v54j\") pod \"maas-api-revoke-keys-e2e-inf-b-1a8107-b79sh\" (UID: \"e0ea5d85-1751-4c56-ab9c-b51366e93322\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1a8107-b79sh" Oct 29 23:25:59.366067 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:59.365988 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:25:59.409120 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:59.409089 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1a8107-b79sh" Oct 29 23:25:59.474460 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:59.474420 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:25:59.743216 ip-10-0-140-74 kubenswrapper[5057]: W1029 23:25:59.743184 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0ea5d85_1751_4c56_ab9c_b51366e93322.slice/crio-dd75323d2dc12ba8b0eab49e8340d1996750274457dea3d3b3ac339a4dd4af3c WatchSource:0}: Error finding container dd75323d2dc12ba8b0eab49e8340d1996750274457dea3d3b3ac339a4dd4af3c: Status 404 returned error can't find the container with id dd75323d2dc12ba8b0eab49e8340d1996750274457dea3d3b3ac339a4dd4af3c Oct 29 23:25:59.743538 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:25:59.743515 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1a8107-b79sh"] Oct 29 23:26:00.151277 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:00.151242 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1a8107-b79sh" event={"ID":"e0ea5d85-1751-4c56-ab9c-b51366e93322","Type":"ContainerStarted","Data":"dd75323d2dc12ba8b0eab49e8340d1996750274457dea3d3b3ac339a4dd4af3c"} Oct 29 23:26:00.843680 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:00.843637 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:26:01.169424 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:01.163359 5057 generic.go:356] "Generic (PLEG): container finished" podID="e0ea5d85-1751-4c56-ab9c-b51366e93322" containerID="11461d8af346eecf8f180f312bb29405a39f5fc1a63f366784be0cb441e5f349" exitCode=0 Oct 29 23:26:01.169424 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:01.163476 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1a8107-b79sh" event={"ID":"e0ea5d85-1751-4c56-ab9c-b51366e93322","Type":"ContainerDied","Data":"11461d8af346eecf8f180f312bb29405a39f5fc1a63f366784be0cb441e5f349"} Oct 29 23:26:01.548989 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:01.548958 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:26:01.663686 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:01.663644 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:26:01.705715 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:01.705679 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b2b71e-vbzwv"] Oct 29 23:26:01.711917 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:01.711891 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b2b71e-vbzwv"] Oct 29 23:26:01.774421 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:01.774375 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:26:02.162561 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:02.162526 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:26:02.367500 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:02.367476 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1a8107-b79sh" Oct 29 23:26:02.537296 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:02.537265 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0ea5d85-1751-4c56-ab9c-b51366e93322-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0ea5d85-1751-4c56-ab9c-b51366e93322-maas-api-openshift-ca\") pod \"e0ea5d85-1751-4c56-ab9c-b51366e93322\" (UID: \"e0ea5d85-1751-4c56-ab9c-b51366e93322\") " Oct 29 23:26:02.537495 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:02.537477 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0ea5d85-1751-4c56-ab9c-b51366e93322-kube-api-access-8v54j\" (UniqueName: \"kubernetes.io/projected/e0ea5d85-1751-4c56-ab9c-b51366e93322-kube-api-access-8v54j\") pod \"e0ea5d85-1751-4c56-ab9c-b51366e93322\" (UID: \"e0ea5d85-1751-4c56-ab9c-b51366e93322\") " Oct 29 23:26:02.537575 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:02.537561 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e0ea5d85-1751-4c56-ab9c-b51366e93322-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0ea5d85-1751-4c56-ab9c-b51366e93322-maas-api-xks-ca\") pod \"e0ea5d85-1751-4c56-ab9c-b51366e93322\" (UID: \"e0ea5d85-1751-4c56-ab9c-b51366e93322\") " Oct 29 23:26:02.537658 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:02.537630 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0ea5d85-1751-4c56-ab9c-b51366e93322-maas-api-openshift-ca" pod "e0ea5d85-1751-4c56-ab9c-b51366e93322" (UID: "e0ea5d85-1751-4c56-ab9c-b51366e93322"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:26:02.538245 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:02.538224 5057 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0ea5d85-1751-4c56-ab9c-b51366e93322-maas-api-openshift-ca\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:26:02.539559 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:02.539539 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0ea5d85-1751-4c56-ab9c-b51366e93322-maas-api-xks-ca" pod "e0ea5d85-1751-4c56-ab9c-b51366e93322" (UID: "e0ea5d85-1751-4c56-ab9c-b51366e93322"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:26:02.539666 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:02.539542 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0ea5d85-1751-4c56-ab9c-b51366e93322-kube-api-access-8v54j" pod "e0ea5d85-1751-4c56-ab9c-b51366e93322" (UID: "e0ea5d85-1751-4c56-ab9c-b51366e93322"). InnerVolumeSpecName "kube-api-access-8v54j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:26:02.639466 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:02.639438 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8v54j\" (UniqueName: \"kubernetes.io/projected/e0ea5d85-1751-4c56-ab9c-b51366e93322-kube-api-access-8v54j\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:26:02.639466 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:02.639463 5057 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0ea5d85-1751-4c56-ab9c-b51366e93322-maas-api-xks-ca\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:26:03.184762 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:03.184731 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1a8107-b79sh" Oct 29 23:26:03.184762 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:03.184754 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1a8107-b79sh" event={"ID":"e0ea5d85-1751-4c56-ab9c-b51366e93322","Type":"ContainerDied","Data":"dd75323d2dc12ba8b0eab49e8340d1996750274457dea3d3b3ac339a4dd4af3c"} Oct 29 23:26:03.185307 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:03.184834 5057 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dd75323d2dc12ba8b0eab49e8340d1996750274457dea3d3b3ac339a4dd4af3c" Oct 29 23:26:03.331912 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:03.331879 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="35e18ec0-85f5-497b-8f44-907262416627" path="/var/lib/kubelet/pods/35e18ec0-85f5-497b-8f44-907262416627/volumes" Oct 29 23:26:03.560250 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:03.560219 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:26:05.146290 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:05.146255 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:26:05.542002 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:05.541966 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:26:06.724964 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:06.724917 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:26:08.282321 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:08.282281 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:26:08.444197 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:08.444155 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:26:08.570859 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:08.570772 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:26:08.916795 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:08.916706 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:26:09.533712 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:09.533681 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-1a8107-79c55c5798-wmrhd"] Oct 29 23:26:09.567781 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:09.567747 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-1a8107-6db566b5cf-6jd5j"] Oct 29 23:26:09.669033 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:09.668994 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:26:09.785710 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:09.785611 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:26:09.959615 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:09.959580 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:26:10.106087 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:10.105993 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:26:10.476997 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:10.475290 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-f3d24e75-openshift-default-7b4d9c69f7-rd4pc"] Oct 29 23:26:10.476997 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:10.476885 5057 cpu_manager.go:414] "containerMap: removing container" podUID="e0ea5d85-1751-4c56-ab9c-b51366e93322" containerName="revoke-keys" Oct 29 23:26:10.476997 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:10.476906 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0ea5d85-1751-4c56-ab9c-b51366e93322" containerName="revoke-keys" Oct 29 23:26:10.477308 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:10.477157 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0ea5d85-1751-4c56-ab9c-b51366e93322" containerName="revoke-keys" Oct 29 23:26:10.498088 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:10.498061 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f3d24e75-openshift-default-7b4d9c69f7-rd4pc" Oct 29 23:26:10.515620 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:10.515593 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-f3d24e75-openshift-default-dockercfg-2h597\"" Oct 29 23:26:10.517038 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:10.517015 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-f3d24e75-openshift-default-7b4d9c69f7-rd4pc"] Oct 29 23:26:10.554177 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:10.554104 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:26:10.571301 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:10.571266 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c02e87f-b552-45f4-97d2-d54429170748-credential-socket\") pod \"e2e-mt-f3d24e75-openshift-default-7b4d9c69f7-rd4pc\" (UID: \"9c02e87f-b552-45f4-97d2-d54429170748\") " pod="openshift-ingress/e2e-mt-f3d24e75-openshift-default-7b4d9c69f7-rd4pc" Oct 29 23:26:10.571490 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:10.571460 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7rw9\" (UniqueName: \"kubernetes.io/projected/9c02e87f-b552-45f4-97d2-d54429170748-kube-api-access-n7rw9\") pod \"e2e-mt-f3d24e75-openshift-default-7b4d9c69f7-rd4pc\" (UID: \"9c02e87f-b552-45f4-97d2-d54429170748\") " pod="openshift-ingress/e2e-mt-f3d24e75-openshift-default-7b4d9c69f7-rd4pc" Oct 29 23:26:10.571490 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:10.571483 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c02e87f-b552-45f4-97d2-d54429170748-istio-data\") pod \"e2e-mt-f3d24e75-openshift-default-7b4d9c69f7-rd4pc\" (UID: \"9c02e87f-b552-45f4-97d2-d54429170748\") " pod="openshift-ingress/e2e-mt-f3d24e75-openshift-default-7b4d9c69f7-rd4pc" Oct 29 23:26:10.571651 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:10.571503 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c02e87f-b552-45f4-97d2-d54429170748-istio-envoy\") pod \"e2e-mt-f3d24e75-openshift-default-7b4d9c69f7-rd4pc\" (UID: \"9c02e87f-b552-45f4-97d2-d54429170748\") " pod="openshift-ingress/e2e-mt-f3d24e75-openshift-default-7b4d9c69f7-rd4pc" Oct 29 23:26:10.571651 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:10.571556 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c02e87f-b552-45f4-97d2-d54429170748-workload-certs\") pod \"e2e-mt-f3d24e75-openshift-default-7b4d9c69f7-rd4pc\" (UID: \"9c02e87f-b552-45f4-97d2-d54429170748\") " pod="openshift-ingress/e2e-mt-f3d24e75-openshift-default-7b4d9c69f7-rd4pc" Oct 29 23:26:10.571651 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:10.571630 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c02e87f-b552-45f4-97d2-d54429170748-istiod-ca-cert\") pod \"e2e-mt-f3d24e75-openshift-default-7b4d9c69f7-rd4pc\" (UID: \"9c02e87f-b552-45f4-97d2-d54429170748\") " pod="openshift-ingress/e2e-mt-f3d24e75-openshift-default-7b4d9c69f7-rd4pc" Oct 29 23:26:10.571829 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:10.571746 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c02e87f-b552-45f4-97d2-d54429170748-workload-socket\") pod \"e2e-mt-f3d24e75-openshift-default-7b4d9c69f7-rd4pc\" (UID: \"9c02e87f-b552-45f4-97d2-d54429170748\") " pod="openshift-ingress/e2e-mt-f3d24e75-openshift-default-7b4d9c69f7-rd4pc" Oct 29 23:26:10.571829 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:10.571773 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c02e87f-b552-45f4-97d2-d54429170748-istio-podinfo\") pod \"e2e-mt-f3d24e75-openshift-default-7b4d9c69f7-rd4pc\" (UID: \"9c02e87f-b552-45f4-97d2-d54429170748\") " pod="openshift-ingress/e2e-mt-f3d24e75-openshift-default-7b4d9c69f7-rd4pc" Oct 29 23:26:10.571953 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:10.571848 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c02e87f-b552-45f4-97d2-d54429170748-istio-token\") pod \"e2e-mt-f3d24e75-openshift-default-7b4d9c69f7-rd4pc\" (UID: \"9c02e87f-b552-45f4-97d2-d54429170748\") " pod="openshift-ingress/e2e-mt-f3d24e75-openshift-default-7b4d9c69f7-rd4pc" Oct 29 23:26:10.659314 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:10.659282 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:26:10.673765 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:10.673729 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7rw9\" (UniqueName: \"kubernetes.io/projected/9c02e87f-b552-45f4-97d2-d54429170748-kube-api-access-n7rw9\") pod \"e2e-mt-f3d24e75-openshift-default-7b4d9c69f7-rd4pc\" (UID: \"9c02e87f-b552-45f4-97d2-d54429170748\") " pod="openshift-ingress/e2e-mt-f3d24e75-openshift-default-7b4d9c69f7-rd4pc" Oct 29 23:26:10.673944 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:10.673772 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c02e87f-b552-45f4-97d2-d54429170748-istio-data\") pod \"e2e-mt-f3d24e75-openshift-default-7b4d9c69f7-rd4pc\" (UID: \"9c02e87f-b552-45f4-97d2-d54429170748\") " pod="openshift-ingress/e2e-mt-f3d24e75-openshift-default-7b4d9c69f7-rd4pc" Oct 29 23:26:10.673944 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:10.673807 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c02e87f-b552-45f4-97d2-d54429170748-istio-envoy\") pod \"e2e-mt-f3d24e75-openshift-default-7b4d9c69f7-rd4pc\" (UID: \"9c02e87f-b552-45f4-97d2-d54429170748\") " pod="openshift-ingress/e2e-mt-f3d24e75-openshift-default-7b4d9c69f7-rd4pc" Oct 29 23:26:10.673944 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:10.673849 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c02e87f-b552-45f4-97d2-d54429170748-workload-certs\") pod \"e2e-mt-f3d24e75-openshift-default-7b4d9c69f7-rd4pc\" (UID: \"9c02e87f-b552-45f4-97d2-d54429170748\") " pod="openshift-ingress/e2e-mt-f3d24e75-openshift-default-7b4d9c69f7-rd4pc" Oct 29 23:26:10.673944 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:10.673896 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c02e87f-b552-45f4-97d2-d54429170748-istiod-ca-cert\") pod \"e2e-mt-f3d24e75-openshift-default-7b4d9c69f7-rd4pc\" (UID: \"9c02e87f-b552-45f4-97d2-d54429170748\") " pod="openshift-ingress/e2e-mt-f3d24e75-openshift-default-7b4d9c69f7-rd4pc" Oct 29 23:26:10.674170 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:10.674041 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c02e87f-b552-45f4-97d2-d54429170748-workload-socket\") pod \"e2e-mt-f3d24e75-openshift-default-7b4d9c69f7-rd4pc\" (UID: \"9c02e87f-b552-45f4-97d2-d54429170748\") " pod="openshift-ingress/e2e-mt-f3d24e75-openshift-default-7b4d9c69f7-rd4pc" Oct 29 23:26:10.674170 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:10.674082 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c02e87f-b552-45f4-97d2-d54429170748-istio-podinfo\") pod \"e2e-mt-f3d24e75-openshift-default-7b4d9c69f7-rd4pc\" (UID: \"9c02e87f-b552-45f4-97d2-d54429170748\") " pod="openshift-ingress/e2e-mt-f3d24e75-openshift-default-7b4d9c69f7-rd4pc" Oct 29 23:26:10.674170 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:10.674156 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c02e87f-b552-45f4-97d2-d54429170748-istio-token\") pod \"e2e-mt-f3d24e75-openshift-default-7b4d9c69f7-rd4pc\" (UID: \"9c02e87f-b552-45f4-97d2-d54429170748\") " pod="openshift-ingress/e2e-mt-f3d24e75-openshift-default-7b4d9c69f7-rd4pc" Oct 29 23:26:10.674343 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:10.674253 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c02e87f-b552-45f4-97d2-d54429170748-credential-socket\") pod \"e2e-mt-f3d24e75-openshift-default-7b4d9c69f7-rd4pc\" (UID: \"9c02e87f-b552-45f4-97d2-d54429170748\") " pod="openshift-ingress/e2e-mt-f3d24e75-openshift-default-7b4d9c69f7-rd4pc" Oct 29 23:26:10.674398 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:10.674253 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c02e87f-b552-45f4-97d2-d54429170748-workload-certs\") pod \"e2e-mt-f3d24e75-openshift-default-7b4d9c69f7-rd4pc\" (UID: \"9c02e87f-b552-45f4-97d2-d54429170748\") " pod="openshift-ingress/e2e-mt-f3d24e75-openshift-default-7b4d9c69f7-rd4pc" Oct 29 23:26:10.674475 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:10.674452 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c02e87f-b552-45f4-97d2-d54429170748-workload-socket\") pod \"e2e-mt-f3d24e75-openshift-default-7b4d9c69f7-rd4pc\" (UID: \"9c02e87f-b552-45f4-97d2-d54429170748\") " pod="openshift-ingress/e2e-mt-f3d24e75-openshift-default-7b4d9c69f7-rd4pc" Oct 29 23:26:10.674697 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:10.674662 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c02e87f-b552-45f4-97d2-d54429170748-istio-data\") pod \"e2e-mt-f3d24e75-openshift-default-7b4d9c69f7-rd4pc\" (UID: \"9c02e87f-b552-45f4-97d2-d54429170748\") " pod="openshift-ingress/e2e-mt-f3d24e75-openshift-default-7b4d9c69f7-rd4pc" Oct 29 23:26:10.674762 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:10.674679 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c02e87f-b552-45f4-97d2-d54429170748-credential-socket\") pod \"e2e-mt-f3d24e75-openshift-default-7b4d9c69f7-rd4pc\" (UID: \"9c02e87f-b552-45f4-97d2-d54429170748\") " pod="openshift-ingress/e2e-mt-f3d24e75-openshift-default-7b4d9c69f7-rd4pc" Oct 29 23:26:10.675047 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:10.675022 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c02e87f-b552-45f4-97d2-d54429170748-istiod-ca-cert\") pod \"e2e-mt-f3d24e75-openshift-default-7b4d9c69f7-rd4pc\" (UID: \"9c02e87f-b552-45f4-97d2-d54429170748\") " pod="openshift-ingress/e2e-mt-f3d24e75-openshift-default-7b4d9c69f7-rd4pc" Oct 29 23:26:10.676314 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:10.676290 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c02e87f-b552-45f4-97d2-d54429170748-istio-envoy\") pod \"e2e-mt-f3d24e75-openshift-default-7b4d9c69f7-rd4pc\" (UID: \"9c02e87f-b552-45f4-97d2-d54429170748\") " pod="openshift-ingress/e2e-mt-f3d24e75-openshift-default-7b4d9c69f7-rd4pc" Oct 29 23:26:10.677264 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:10.677243 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c02e87f-b552-45f4-97d2-d54429170748-istio-podinfo\") pod \"e2e-mt-f3d24e75-openshift-default-7b4d9c69f7-rd4pc\" (UID: \"9c02e87f-b552-45f4-97d2-d54429170748\") " pod="openshift-ingress/e2e-mt-f3d24e75-openshift-default-7b4d9c69f7-rd4pc" Oct 29 23:26:10.697724 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:10.697694 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c02e87f-b552-45f4-97d2-d54429170748-istio-token\") pod \"e2e-mt-f3d24e75-openshift-default-7b4d9c69f7-rd4pc\" (UID: \"9c02e87f-b552-45f4-97d2-d54429170748\") " pod="openshift-ingress/e2e-mt-f3d24e75-openshift-default-7b4d9c69f7-rd4pc" Oct 29 23:26:10.698301 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:10.698278 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7rw9\" (UniqueName: \"kubernetes.io/projected/9c02e87f-b552-45f4-97d2-d54429170748-kube-api-access-n7rw9\") pod \"e2e-mt-f3d24e75-openshift-default-7b4d9c69f7-rd4pc\" (UID: \"9c02e87f-b552-45f4-97d2-d54429170748\") " pod="openshift-ingress/e2e-mt-f3d24e75-openshift-default-7b4d9c69f7-rd4pc" Oct 29 23:26:10.839282 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:10.839189 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f3d24e75-openshift-default-7b4d9c69f7-rd4pc" Oct 29 23:26:11.179488 ip-10-0-140-74 kubenswrapper[5057]: W1029 23:26:11.179455 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c02e87f_b552_45f4_97d2_d54429170748.slice/crio-cac90193515f3f2f4c933e310428ad76b4c1b17cfb09d3c7488c517c8e08f545 WatchSource:0}: Error finding container cac90193515f3f2f4c933e310428ad76b4c1b17cfb09d3c7488c517c8e08f545: Status 404 returned error can't find the container with id cac90193515f3f2f4c933e310428ad76b4c1b17cfb09d3c7488c517c8e08f545 Oct 29 23:26:11.180127 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:11.180084 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-f3d24e75-openshift-default-7b4d9c69f7-rd4pc"] Oct 29 23:26:11.182621 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:11.182587 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 23:26:11.182728 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:11.182651 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 23:26:11.182728 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:11.182685 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 23:26:11.299614 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:11.299567 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f3d24e75-openshift-default-7b4d9c69f7-rd4pc" event={"ID":"9c02e87f-b552-45f4-97d2-d54429170748","Type":"ContainerStarted","Data":"ffe99f5d1703980ea3c268cb83c005ab697c8dacbb0e060ef257ad2e96a03df2"} Oct 29 23:26:11.299804 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:11.299648 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f3d24e75-openshift-default-7b4d9c69f7-rd4pc" event={"ID":"9c02e87f-b552-45f4-97d2-d54429170748","Type":"ContainerStarted","Data":"cac90193515f3f2f4c933e310428ad76b4c1b17cfb09d3c7488c517c8e08f545"} Oct 29 23:26:11.351294 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:11.351242 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-f3d24e75-openshift-default-7b4d9c69f7-rd4pc" podStartSLOduration=1.351228205 podStartE2EDuration="1.351228205s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:26:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:26:11.345025819 +0000 UTC m=+3152.128239246" watchObservedRunningTime="2026-10-29 23:26:11.351228205 +0000 UTC m=+3152.134441628" Oct 29 23:26:11.837645 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:11.837611 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v2tfm"] Oct 29 23:26:11.839927 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:11.839901 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-f3d24e75-openshift-default-7b4d9c69f7-rd4pc" Oct 29 23:26:11.931466 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:11.931425 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:26:12.896123 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:12.896082 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-f3d24e75-openshift-default-7b4d9c69f7-rd4pc" Oct 29 23:26:13.321255 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:13.320260 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-f3d24e75-openshift-default-7b4d9c69f7-rd4pc" Oct 29 23:26:13.339789 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:13.339760 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-f3d24e75-openshift-default-7b4d9c69f7-rd4pc" Oct 29 23:26:14.006354 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:14.000275 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:26:14.128517 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:14.128478 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:26:14.534097 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:14.534058 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-1a8107-79c55c5798-wmrhd" podUID="7a4256ac-b349-4288-a6c9-92b4ed74e3e6" containerName="payload-processing" containerID="cri-o://2a646f693a7588e1442fe7d7821877f1e595bcad97a1eff4feffbd0266d87083" gracePeriod=25 Oct 29 23:26:14.568216 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:14.568171 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1a8107-6db566b5cf-6jd5j" podUID="56e0eac5-6393-4933-a3ed-4e80fd849bf5" containerName="payload-pre-processing" containerID="cri-o://06c6274d405753bba9393c5739324878e51136a5c2e3db78aa8096cc9f28c564" gracePeriod=25 Oct 29 23:26:14.626710 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:14.626674 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:26:15.592353 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:15.572096 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:26:15.817355 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:15.813803 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:26:15.943923 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:15.943821 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a37090-fz9b8"] Oct 29 23:26:15.981850 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:15.980763 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:26:16.362434 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:16.362404 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a37090-fz9b8"] Oct 29 23:26:16.367352 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:16.363298 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a37090-fz9b8" Oct 29 23:26:16.382655 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:16.382626 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dmx57\"" Oct 29 23:26:16.382904 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:16.382889 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:26:16.385317 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:16.385295 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-f3d24e75-58f4755655-nsl26"] Oct 29 23:26:16.511544 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:16.511509 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-f3d24e75-58f4755655-nsl26"] Oct 29 23:26:16.511796 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:16.511670 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-f3d24e75-58f4755655-nsl26" Oct 29 23:26:16.527369 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:16.526480 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-f3d24e75-dockercfg-fqvqk\"" Oct 29 23:26:16.527369 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:16.527289 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-f3d24e75\"" Oct 29 23:26:16.649749 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:16.649657 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f3d24e75-b57596bdd-tvq82"] Oct 29 23:26:16.975273 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:16.975239 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f3d24e75-b57596bdd-tvq82"] Oct 29 23:26:16.975671 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:16.975650 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-f3d24e75-b57596bdd-tvq82" Oct 29 23:26:17.002765 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:17.002729 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9pn26\" (UniqueName: \"kubernetes.io/projected/1cf4cf1a-74e0-4632-97d3-0b743a746ab8-kube-api-access-9pn26\") pod \"maas-api-revoke-keys-e2e-inf-a-a37090-fz9b8\" (UID: \"1cf4cf1a-74e0-4632-97d3-0b743a746ab8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a37090-fz9b8" Oct 29 23:26:17.003155 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:17.003127 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1cf4cf1a-74e0-4632-97d3-0b743a746ab8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-a37090-fz9b8\" (UID: \"1cf4cf1a-74e0-4632-97d3-0b743a746ab8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a37090-fz9b8" Oct 29 23:26:17.003544 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:17.003518 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1cf4cf1a-74e0-4632-97d3-0b743a746ab8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-a37090-fz9b8\" (UID: \"1cf4cf1a-74e0-4632-97d3-0b743a746ab8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a37090-fz9b8" Oct 29 23:26:17.117382 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:17.104951 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/268baf9a-3c0a-4ac5-a170-e69a7f3d1cfc-plugins-config-volume\") pod \"payload-processing-e2e-mt-f3d24e75-b57596bdd-tvq82\" (UID: \"268baf9a-3c0a-4ac5-a170-e69a7f3d1cfc\") " pod="openshift-ingress/payload-processing-e2e-mt-f3d24e75-b57596bdd-tvq82" Oct 29 23:26:17.117382 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:17.105028 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pn26\" (UniqueName: \"kubernetes.io/projected/1cf4cf1a-74e0-4632-97d3-0b743a746ab8-kube-api-access-9pn26\") pod \"maas-api-revoke-keys-e2e-inf-a-a37090-fz9b8\" (UID: \"1cf4cf1a-74e0-4632-97d3-0b743a746ab8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a37090-fz9b8" Oct 29 23:26:17.117382 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:17.105143 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0c5087d-e510-4841-a25c-dd6908304d61-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-f3d24e75-58f4755655-nsl26\" (UID: \"e0c5087d-e510-4841-a25c-dd6908304d61\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-f3d24e75-58f4755655-nsl26" Oct 29 23:26:17.117382 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:17.105198 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1cf4cf1a-74e0-4632-97d3-0b743a746ab8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-a37090-fz9b8\" (UID: \"1cf4cf1a-74e0-4632-97d3-0b743a746ab8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a37090-fz9b8" Oct 29 23:26:17.118670 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:17.105320 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2hzw\" (UniqueName: \"kubernetes.io/projected/e0c5087d-e510-4841-a25c-dd6908304d61-kube-api-access-j2hzw\") pod \"payload-pre-processing-e2e-mt-f3d24e75-58f4755655-nsl26\" (UID: \"e0c5087d-e510-4841-a25c-dd6908304d61\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-f3d24e75-58f4755655-nsl26" Oct 29 23:26:17.118670 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:17.117950 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1cf4cf1a-74e0-4632-97d3-0b743a746ab8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-a37090-fz9b8\" (UID: \"1cf4cf1a-74e0-4632-97d3-0b743a746ab8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a37090-fz9b8" Oct 29 23:26:17.118670 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:17.117996 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hc7dd\" (UniqueName: \"kubernetes.io/projected/268baf9a-3c0a-4ac5-a170-e69a7f3d1cfc-kube-api-access-hc7dd\") pod \"payload-processing-e2e-mt-f3d24e75-b57596bdd-tvq82\" (UID: \"268baf9a-3c0a-4ac5-a170-e69a7f3d1cfc\") " pod="openshift-ingress/payload-processing-e2e-mt-f3d24e75-b57596bdd-tvq82" Oct 29 23:26:17.123534 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:17.123505 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1cf4cf1a-74e0-4632-97d3-0b743a746ab8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-a37090-fz9b8\" (UID: \"1cf4cf1a-74e0-4632-97d3-0b743a746ab8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a37090-fz9b8" Oct 29 23:26:17.140960 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:17.140904 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1cf4cf1a-74e0-4632-97d3-0b743a746ab8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-a37090-fz9b8\" (UID: \"1cf4cf1a-74e0-4632-97d3-0b743a746ab8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a37090-fz9b8" Oct 29 23:26:17.155254 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:17.155002 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pn26\" (UniqueName: \"kubernetes.io/projected/1cf4cf1a-74e0-4632-97d3-0b743a746ab8-kube-api-access-9pn26\") pod \"maas-api-revoke-keys-e2e-inf-a-a37090-fz9b8\" (UID: \"1cf4cf1a-74e0-4632-97d3-0b743a746ab8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a37090-fz9b8" Oct 29 23:26:17.220403 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:17.219168 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2hzw\" (UniqueName: \"kubernetes.io/projected/e0c5087d-e510-4841-a25c-dd6908304d61-kube-api-access-j2hzw\") pod \"payload-pre-processing-e2e-mt-f3d24e75-58f4755655-nsl26\" (UID: \"e0c5087d-e510-4841-a25c-dd6908304d61\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-f3d24e75-58f4755655-nsl26" Oct 29 23:26:17.220403 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:17.219280 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hc7dd\" (UniqueName: \"kubernetes.io/projected/268baf9a-3c0a-4ac5-a170-e69a7f3d1cfc-kube-api-access-hc7dd\") pod \"payload-processing-e2e-mt-f3d24e75-b57596bdd-tvq82\" (UID: \"268baf9a-3c0a-4ac5-a170-e69a7f3d1cfc\") " pod="openshift-ingress/payload-processing-e2e-mt-f3d24e75-b57596bdd-tvq82" Oct 29 23:26:17.220403 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:17.219422 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/268baf9a-3c0a-4ac5-a170-e69a7f3d1cfc-plugins-config-volume\") pod \"payload-processing-e2e-mt-f3d24e75-b57596bdd-tvq82\" (UID: \"268baf9a-3c0a-4ac5-a170-e69a7f3d1cfc\") " pod="openshift-ingress/payload-processing-e2e-mt-f3d24e75-b57596bdd-tvq82" Oct 29 23:26:17.220403 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:17.219554 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0c5087d-e510-4841-a25c-dd6908304d61-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-f3d24e75-58f4755655-nsl26\" (UID: \"e0c5087d-e510-4841-a25c-dd6908304d61\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-f3d24e75-58f4755655-nsl26" Oct 29 23:26:17.221110 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:17.221063 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0c5087d-e510-4841-a25c-dd6908304d61-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-f3d24e75-58f4755655-nsl26\" (UID: \"e0c5087d-e510-4841-a25c-dd6908304d61\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-f3d24e75-58f4755655-nsl26" Oct 29 23:26:17.221691 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:17.221641 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/268baf9a-3c0a-4ac5-a170-e69a7f3d1cfc-plugins-config-volume\") pod \"payload-processing-e2e-mt-f3d24e75-b57596bdd-tvq82\" (UID: \"268baf9a-3c0a-4ac5-a170-e69a7f3d1cfc\") " pod="openshift-ingress/payload-processing-e2e-mt-f3d24e75-b57596bdd-tvq82" Oct 29 23:26:17.263709 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:17.263616 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2hzw\" (UniqueName: \"kubernetes.io/projected/e0c5087d-e510-4841-a25c-dd6908304d61-kube-api-access-j2hzw\") pod \"payload-pre-processing-e2e-mt-f3d24e75-58f4755655-nsl26\" (UID: \"e0c5087d-e510-4841-a25c-dd6908304d61\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-f3d24e75-58f4755655-nsl26" Oct 29 23:26:17.269070 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:17.269041 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hc7dd\" (UniqueName: \"kubernetes.io/projected/268baf9a-3c0a-4ac5-a170-e69a7f3d1cfc-kube-api-access-hc7dd\") pod \"payload-processing-e2e-mt-f3d24e75-b57596bdd-tvq82\" (UID: \"268baf9a-3c0a-4ac5-a170-e69a7f3d1cfc\") " pod="openshift-ingress/payload-processing-e2e-mt-f3d24e75-b57596bdd-tvq82" Oct 29 23:26:17.286057 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:17.286033 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-f3d24e75-58f4755655-nsl26" Oct 29 23:26:17.335358 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:17.331433 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-f3d24e75-b57596bdd-tvq82" Oct 29 23:26:17.405379 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:17.396631 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a37090-fz9b8" Oct 29 23:26:17.646199 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:17.646113 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:26:17.918877 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:17.918790 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:26:17.953246 ip-10-0-140-74 kubenswrapper[5057]: W1029 23:26:17.953206 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0c5087d_e510_4841_a25c_dd6908304d61.slice/crio-adbabbc86a40a5939f68830b9e881a6711fab2d9569041b91850243f6f0b628b WatchSource:0}: Error finding container adbabbc86a40a5939f68830b9e881a6711fab2d9569041b91850243f6f0b628b: Status 404 returned error can't find the container with id adbabbc86a40a5939f68830b9e881a6711fab2d9569041b91850243f6f0b628b Oct 29 23:26:17.953520 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:17.953496 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-f3d24e75-58f4755655-nsl26"] Oct 29 23:26:17.987112 ip-10-0-140-74 kubenswrapper[5057]: W1029 23:26:17.985588 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1cf4cf1a_74e0_4632_97d3_0b743a746ab8.slice/crio-cab9959a2686b490cc6f7fcb604be9a2d4906bceb4fd43bd3069bd4277f6d6c3 WatchSource:0}: Error finding container cab9959a2686b490cc6f7fcb604be9a2d4906bceb4fd43bd3069bd4277f6d6c3: Status 404 returned error can't find the container with id cab9959a2686b490cc6f7fcb604be9a2d4906bceb4fd43bd3069bd4277f6d6c3 Oct 29 23:26:17.991635 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:17.991038 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a37090-fz9b8"] Oct 29 23:26:18.026314 ip-10-0-140-74 kubenswrapper[5057]: W1029 23:26:18.026282 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod268baf9a_3c0a_4ac5_a170_e69a7f3d1cfc.slice/crio-d5a2a9ee7130e42a5ed0b2ffc7a20971aa2ad18328bcd35de9a8b43d9baa22bf WatchSource:0}: Error finding container d5a2a9ee7130e42a5ed0b2ffc7a20971aa2ad18328bcd35de9a8b43d9baa22bf: Status 404 returned error can't find the container with id d5a2a9ee7130e42a5ed0b2ffc7a20971aa2ad18328bcd35de9a8b43d9baa22bf Oct 29 23:26:18.043670 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:18.043632 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f3d24e75-b57596bdd-tvq82"] Oct 29 23:26:18.569588 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:18.569509 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:26:18.671383 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:18.671298 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a37090-fz9b8" event={"ID":"1cf4cf1a-74e0-4632-97d3-0b743a746ab8","Type":"ContainerStarted","Data":"36739b33ecc51a349147d6192bb8cf0f89e26939c04d2a125105ec2d1b6027a8"} Oct 29 23:26:18.671876 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:18.671413 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a37090-fz9b8" event={"ID":"1cf4cf1a-74e0-4632-97d3-0b743a746ab8","Type":"ContainerStarted","Data":"cab9959a2686b490cc6f7fcb604be9a2d4906bceb4fd43bd3069bd4277f6d6c3"} Oct 29 23:26:18.673487 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:18.673453 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-f3d24e75-58f4755655-nsl26" event={"ID":"e0c5087d-e510-4841-a25c-dd6908304d61","Type":"ContainerStarted","Data":"99c27e53364f12b11d95e1846f068b9065620c840d8c0dc435b3d27a3905e14f"} Oct 29 23:26:18.673611 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:18.673519 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-f3d24e75-58f4755655-nsl26" Oct 29 23:26:18.673611 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:18.673543 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-f3d24e75-58f4755655-nsl26" event={"ID":"e0c5087d-e510-4841-a25c-dd6908304d61","Type":"ContainerStarted","Data":"adbabbc86a40a5939f68830b9e881a6711fab2d9569041b91850243f6f0b628b"} Oct 29 23:26:18.676004 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:18.675975 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-f3d24e75-b57596bdd-tvq82" event={"ID":"268baf9a-3c0a-4ac5-a170-e69a7f3d1cfc","Type":"ContainerStarted","Data":"d8b88aeb5bbc5de07c873491bf590b2a6bf6a803d45584ceb105830a4de88a3c"} Oct 29 23:26:18.676119 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:18.676028 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-f3d24e75-b57596bdd-tvq82" event={"ID":"268baf9a-3c0a-4ac5-a170-e69a7f3d1cfc","Type":"ContainerStarted","Data":"d5a2a9ee7130e42a5ed0b2ffc7a20971aa2ad18328bcd35de9a8b43d9baa22bf"} Oct 29 23:26:18.676382 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:18.676319 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-f3d24e75-b57596bdd-tvq82" Oct 29 23:26:18.725536 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:18.722899 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a37090-fz9b8" podStartSLOduration=3.722879845 podStartE2EDuration="3.722879845s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:26:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:26:18.718580743 +0000 UTC m=+3159.501794170" watchObservedRunningTime="2026-10-29 23:26:18.722879845 +0000 UTC m=+3159.506093290" Oct 29 23:26:18.768613 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:18.768554 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-f3d24e75-58f4755655-nsl26" podStartSLOduration=3.768535926 podStartE2EDuration="3.768535926s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:26:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:26:18.766225967 +0000 UTC m=+3159.549439394" watchObservedRunningTime="2026-10-29 23:26:18.768535926 +0000 UTC m=+3159.551749352" Oct 29 23:26:18.862318 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:18.862193 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-f3d24e75-b57596bdd-tvq82" podStartSLOduration=2.862175942 podStartE2EDuration="2.862175942s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:26:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:26:18.859708643 +0000 UTC m=+3159.642922095" watchObservedRunningTime="2026-10-29 23:26:18.862175942 +0000 UTC m=+3159.645389369" Oct 29 23:26:19.065077 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:26:19.065033 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d2dnn" containerName="extractor" Oct 29 23:26:19.109978 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:19.109943 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d7861bca-cz6gc"] Oct 29 23:26:19.117821 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:19.117754 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7861bca-cz6gc" Oct 29 23:26:19.137069 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:19.137041 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d7861bca-cz6gc"] Oct 29 23:26:19.157666 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:19.157629 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d884577-666d-418f-91be-0d0744c3bf09-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d7861bca-cz6gc\" (UID: \"9d884577-666d-418f-91be-0d0744c3bf09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7861bca-cz6gc" Oct 29 23:26:19.157858 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:19.157839 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d884577-666d-418f-91be-0d0744c3bf09-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d7861bca-cz6gc\" (UID: \"9d884577-666d-418f-91be-0d0744c3bf09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7861bca-cz6gc" Oct 29 23:26:19.157946 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:19.157907 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5txqq\" (UniqueName: \"kubernetes.io/projected/9d884577-666d-418f-91be-0d0744c3bf09-kube-api-access-5txqq\") pod \"maas-api-revoke-sub-d7861bca-cz6gc\" (UID: \"9d884577-666d-418f-91be-0d0744c3bf09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7861bca-cz6gc" Oct 29 23:26:19.260069 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:19.260032 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d884577-666d-418f-91be-0d0744c3bf09-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d7861bca-cz6gc\" (UID: \"9d884577-666d-418f-91be-0d0744c3bf09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7861bca-cz6gc" Oct 29 23:26:19.260258 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:19.260240 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d884577-666d-418f-91be-0d0744c3bf09-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d7861bca-cz6gc\" (UID: \"9d884577-666d-418f-91be-0d0744c3bf09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7861bca-cz6gc" Oct 29 23:26:19.260384 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:19.260322 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5txqq\" (UniqueName: \"kubernetes.io/projected/9d884577-666d-418f-91be-0d0744c3bf09-kube-api-access-5txqq\") pod \"maas-api-revoke-sub-d7861bca-cz6gc\" (UID: \"9d884577-666d-418f-91be-0d0744c3bf09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7861bca-cz6gc" Oct 29 23:26:19.260792 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:19.260767 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d884577-666d-418f-91be-0d0744c3bf09-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d7861bca-cz6gc\" (UID: \"9d884577-666d-418f-91be-0d0744c3bf09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7861bca-cz6gc" Oct 29 23:26:19.262555 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:19.262530 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d884577-666d-418f-91be-0d0744c3bf09-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d7861bca-cz6gc\" (UID: \"9d884577-666d-418f-91be-0d0744c3bf09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7861bca-cz6gc" Oct 29 23:26:19.283404 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:19.283313 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5txqq\" (UniqueName: \"kubernetes.io/projected/9d884577-666d-418f-91be-0d0744c3bf09-kube-api-access-5txqq\") pod \"maas-api-revoke-sub-d7861bca-cz6gc\" (UID: \"9d884577-666d-418f-91be-0d0744c3bf09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7861bca-cz6gc" Oct 29 23:26:19.439447 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:19.439356 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7861bca-cz6gc" Oct 29 23:26:19.691274 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:19.691175 5057 generic.go:356] "Generic (PLEG): container finished" podID="1cf4cf1a-74e0-4632-97d3-0b743a746ab8" containerID="36739b33ecc51a349147d6192bb8cf0f89e26939c04d2a125105ec2d1b6027a8" exitCode=0 Oct 29 23:26:19.691274 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:19.691246 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a37090-fz9b8" event={"ID":"1cf4cf1a-74e0-4632-97d3-0b743a746ab8","Type":"ContainerDied","Data":"36739b33ecc51a349147d6192bb8cf0f89e26939c04d2a125105ec2d1b6027a8"} Oct 29 23:26:19.797383 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:19.790966 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:26:19.800084 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:19.800051 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d7861bca-cz6gc"] Oct 29 23:26:19.806318 ip-10-0-140-74 kubenswrapper[5057]: W1029 23:26:19.806259 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d884577_666d_418f_91be_0d0744c3bf09.slice/crio-ec862839847893fb313cfbcb7178c25ae3f371541c72cb07c04255e9da0195dd WatchSource:0}: Error finding container ec862839847893fb313cfbcb7178c25ae3f371541c72cb07c04255e9da0195dd: Status 404 returned error can't find the container with id ec862839847893fb313cfbcb7178c25ae3f371541c72cb07c04255e9da0195dd Oct 29 23:26:20.706419 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:20.706373 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7861bca-cz6gc" event={"ID":"9d884577-666d-418f-91be-0d0744c3bf09","Type":"ContainerStarted","Data":"9fc0bb2728b43e98c9e96557107c448d5860a2b2fd14e4c5a91e72db5ac4c269"} Oct 29 23:26:20.707148 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:20.706441 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7861bca-cz6gc" event={"ID":"9d884577-666d-418f-91be-0d0744c3bf09","Type":"ContainerStarted","Data":"ec862839847893fb313cfbcb7178c25ae3f371541c72cb07c04255e9da0195dd"} Oct 29 23:26:20.751911 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:20.751848 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7861bca-cz6gc" podStartSLOduration=1.75183264 podStartE2EDuration="1.75183264s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:26:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:26:20.745514938 +0000 UTC m=+3161.528728376" watchObservedRunningTime="2026-10-29 23:26:20.75183264 +0000 UTC m=+3161.535046062" Oct 29 23:26:20.944576 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:20.944547 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a37090-fz9b8" Oct 29 23:26:20.998880 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:20.995038 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1cf4cf1a-74e0-4632-97d3-0b743a746ab8-kube-api-access-9pn26\" (UniqueName: \"kubernetes.io/projected/1cf4cf1a-74e0-4632-97d3-0b743a746ab8-kube-api-access-9pn26\") pod \"1cf4cf1a-74e0-4632-97d3-0b743a746ab8\" (UID: \"1cf4cf1a-74e0-4632-97d3-0b743a746ab8\") " Oct 29 23:26:20.998880 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:20.995112 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1cf4cf1a-74e0-4632-97d3-0b743a746ab8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1cf4cf1a-74e0-4632-97d3-0b743a746ab8-maas-api-xks-ca\") pod \"1cf4cf1a-74e0-4632-97d3-0b743a746ab8\" (UID: \"1cf4cf1a-74e0-4632-97d3-0b743a746ab8\") " Oct 29 23:26:20.998880 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:20.995550 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1cf4cf1a-74e0-4632-97d3-0b743a746ab8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1cf4cf1a-74e0-4632-97d3-0b743a746ab8-maas-api-openshift-ca\") pod \"1cf4cf1a-74e0-4632-97d3-0b743a746ab8\" (UID: \"1cf4cf1a-74e0-4632-97d3-0b743a746ab8\") " Oct 29 23:26:20.998880 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:20.996548 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1cf4cf1a-74e0-4632-97d3-0b743a746ab8-maas-api-openshift-ca" pod "1cf4cf1a-74e0-4632-97d3-0b743a746ab8" (UID: "1cf4cf1a-74e0-4632-97d3-0b743a746ab8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:26:21.001190 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:21.001158 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1cf4cf1a-74e0-4632-97d3-0b743a746ab8-maas-api-xks-ca" pod "1cf4cf1a-74e0-4632-97d3-0b743a746ab8" (UID: "1cf4cf1a-74e0-4632-97d3-0b743a746ab8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:26:21.001536 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:21.001500 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1cf4cf1a-74e0-4632-97d3-0b743a746ab8-kube-api-access-9pn26" pod "1cf4cf1a-74e0-4632-97d3-0b743a746ab8" (UID: "1cf4cf1a-74e0-4632-97d3-0b743a746ab8"). InnerVolumeSpecName "kube-api-access-9pn26". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:26:21.098609 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:21.098571 5057 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1cf4cf1a-74e0-4632-97d3-0b743a746ab8-maas-api-openshift-ca\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:26:21.098609 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:21.098608 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9pn26\" (UniqueName: \"kubernetes.io/projected/1cf4cf1a-74e0-4632-97d3-0b743a746ab8-kube-api-access-9pn26\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:26:21.098809 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:21.098624 5057 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1cf4cf1a-74e0-4632-97d3-0b743a746ab8-maas-api-xks-ca\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:26:21.368194 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:21.368163 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:26:21.718195 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:21.718151 5057 generic.go:356] "Generic (PLEG): container finished" podID="9d884577-666d-418f-91be-0d0744c3bf09" containerID="9fc0bb2728b43e98c9e96557107c448d5860a2b2fd14e4c5a91e72db5ac4c269" exitCode=0 Oct 29 23:26:21.718683 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:21.718239 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7861bca-cz6gc" event={"ID":"9d884577-666d-418f-91be-0d0744c3bf09","Type":"ContainerDied","Data":"9fc0bb2728b43e98c9e96557107c448d5860a2b2fd14e4c5a91e72db5ac4c269"} Oct 29 23:26:21.719889 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:21.719866 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a37090-fz9b8" event={"ID":"1cf4cf1a-74e0-4632-97d3-0b743a746ab8","Type":"ContainerDied","Data":"cab9959a2686b490cc6f7fcb604be9a2d4906bceb4fd43bd3069bd4277f6d6c3"} Oct 29 23:26:21.720053 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:21.719904 5057 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cab9959a2686b490cc6f7fcb604be9a2d4906bceb4fd43bd3069bd4277f6d6c3" Oct 29 23:26:21.720053 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:21.719917 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a37090-fz9b8" Oct 29 23:26:22.127954 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:22.127852 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1a8107-6db566b5cf-6jd5j" podUID="56e0eac5-6393-4933-a3ed-4e80fd849bf5" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:26:22.887101 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:22.887060 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:26:22.917242 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:22.917215 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7861bca-cz6gc" Oct 29 23:26:23.032673 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:23.032587 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d884577-666d-418f-91be-0d0744c3bf09-kube-api-access-5txqq\" (UniqueName: \"kubernetes.io/projected/9d884577-666d-418f-91be-0d0744c3bf09-kube-api-access-5txqq\") pod \"9d884577-666d-418f-91be-0d0744c3bf09\" (UID: \"9d884577-666d-418f-91be-0d0744c3bf09\") " Oct 29 23:26:23.032815 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:23.032725 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9d884577-666d-418f-91be-0d0744c3bf09-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d884577-666d-418f-91be-0d0744c3bf09-maas-api-xks-ca\") pod \"9d884577-666d-418f-91be-0d0744c3bf09\" (UID: \"9d884577-666d-418f-91be-0d0744c3bf09\") " Oct 29 23:26:23.032922 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:23.032909 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d884577-666d-418f-91be-0d0744c3bf09-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d884577-666d-418f-91be-0d0744c3bf09-maas-api-openshift-ca\") pod \"9d884577-666d-418f-91be-0d0744c3bf09\" (UID: \"9d884577-666d-418f-91be-0d0744c3bf09\") " Oct 29 23:26:23.033367 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:23.033305 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d884577-666d-418f-91be-0d0744c3bf09-maas-api-openshift-ca" pod "9d884577-666d-418f-91be-0d0744c3bf09" (UID: "9d884577-666d-418f-91be-0d0744c3bf09"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:26:23.033627 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:23.033611 5057 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d884577-666d-418f-91be-0d0744c3bf09-maas-api-openshift-ca\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:26:23.034741 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:23.034721 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d884577-666d-418f-91be-0d0744c3bf09-maas-api-xks-ca" pod "9d884577-666d-418f-91be-0d0744c3bf09" (UID: "9d884577-666d-418f-91be-0d0744c3bf09"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:26:23.034846 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:23.034738 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d884577-666d-418f-91be-0d0744c3bf09-kube-api-access-5txqq" pod "9d884577-666d-418f-91be-0d0744c3bf09" (UID: "9d884577-666d-418f-91be-0d0744c3bf09"). InnerVolumeSpecName "kube-api-access-5txqq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:26:23.135172 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:23.135133 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5txqq\" (UniqueName: \"kubernetes.io/projected/9d884577-666d-418f-91be-0d0744c3bf09-kube-api-access-5txqq\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:26:23.135172 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:23.135168 5057 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d884577-666d-418f-91be-0d0744c3bf09-maas-api-xks-ca\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:26:23.150492 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:23.150449 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-1a8107-79c55c5798-wmrhd" podUID="7a4256ac-b349-4288-a6c9-92b4ed74e3e6" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:26:23.741407 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:23.741373 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7861bca-cz6gc" Oct 29 23:26:23.741407 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:23.741396 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7861bca-cz6gc" event={"ID":"9d884577-666d-418f-91be-0d0744c3bf09","Type":"ContainerDied","Data":"ec862839847893fb313cfbcb7178c25ae3f371541c72cb07c04255e9da0195dd"} Oct 29 23:26:23.741617 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:23.741442 5057 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ec862839847893fb313cfbcb7178c25ae3f371541c72cb07c04255e9da0195dd" Oct 29 23:26:24.506847 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:24.506809 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:26:26.069173 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:26.069139 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a37090-58c6cbb588-lj97b"] Oct 29 23:26:26.096174 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:26.096135 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:26:26.103087 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:26.103055 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a37090-84ddb8588c-n4gn5"] Oct 29 23:26:26.213392 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:26.213356 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:26:26.320637 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:26.319435 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:26:27.740017 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:27.739982 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:26:29.305751 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:29.305716 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:26:29.694747 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:29.694660 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-f3d24e75-58f4755655-nsl26" Oct 29 23:26:29.694747 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:29.694739 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-f3d24e75-b57596bdd-tvq82" Oct 29 23:26:30.896874 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:30.896839 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:26:31.070257 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:31.070219 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-a37090-58c6cbb588-lj97b" podUID="159d07ab-681d-4431-9d7d-e08bcee96624" containerName="payload-processing" containerID="cri-o://36f85597af859c5e4303f518d0d31d815a44e513f74da021785b2120bba5b1e3" gracePeriod=25 Oct 29 23:26:31.103773 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:31.103713 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a37090-84ddb8588c-n4gn5" podUID="eb4eaa12-cde8-43d8-9af0-9534a90f9c6b" containerName="payload-pre-processing" containerID="cri-o://20440706175fed5a0dc35c1ba1e6aa0329ba5032d54d7f576193d92285a8d72a" gracePeriod=25 Oct 29 23:26:31.133900 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:31.133853 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-058e69ab-ngh6t"] Oct 29 23:26:31.145751 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:31.145717 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-058e69ab-ngh6t"] Oct 29 23:26:31.334654 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:31.334619 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9e1f7d1-fb93-44da-9b30-8c27769dbbc9" path="/var/lib/kubelet/pods/c9e1f7d1-fb93-44da-9b30-8c27769dbbc9/volumes" Oct 29 23:26:32.126885 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:32.126846 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1a8107-6db566b5cf-6jd5j" podUID="56e0eac5-6393-4933-a3ed-4e80fd849bf5" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:26:32.368527 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:32.368415 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:26:33.095309 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:33.095271 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:26:33.149404 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:33.149367 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-1a8107-79c55c5798-wmrhd" podUID="7a4256ac-b349-4288-a6c9-92b4ed74e3e6" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:26:33.210856 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:33.210819 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:26:33.232294 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:33.232262 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a37090-fz9b8"] Oct 29 23:26:33.243649 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:33.243612 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a37090-fz9b8"] Oct 29 23:26:33.315357 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:33.315296 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:26:33.335497 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:33.335464 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1cf4cf1a-74e0-4632-97d3-0b743a746ab8" path="/var/lib/kubelet/pods/1cf4cf1a-74e0-4632-97d3-0b743a746ab8/volumes" Oct 29 23:26:33.733948 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:33.733907 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:26:34.901258 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:34.901232 5057 generic.go:356] "Generic (PLEG): container finished" podID="56e0eac5-6393-4933-a3ed-4e80fd849bf5" containerID="06c6274d405753bba9393c5739324878e51136a5c2e3db78aa8096cc9f28c564" exitCode=0 Oct 29 23:26:34.901679 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:34.901307 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1a8107-6db566b5cf-6jd5j" event={"ID":"56e0eac5-6393-4933-a3ed-4e80fd849bf5","Type":"ContainerDied","Data":"06c6274d405753bba9393c5739324878e51136a5c2e3db78aa8096cc9f28c564"} Oct 29 23:26:34.903025 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:34.903006 5057 generic.go:356] "Generic (PLEG): container finished" podID="7a4256ac-b349-4288-a6c9-92b4ed74e3e6" containerID="2a646f693a7588e1442fe7d7821877f1e595bcad97a1eff4feffbd0266d87083" exitCode=0 Oct 29 23:26:34.903137 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:34.903078 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-1a8107-79c55c5798-wmrhd" event={"ID":"7a4256ac-b349-4288-a6c9-92b4ed74e3e6","Type":"ContainerDied","Data":"2a646f693a7588e1442fe7d7821877f1e595bcad97a1eff4feffbd0266d87083"} Oct 29 23:26:34.903182 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:34.903133 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-1a8107-79c55c5798-wmrhd" event={"ID":"7a4256ac-b349-4288-a6c9-92b4ed74e3e6","Type":"ContainerDied","Data":"027af2c9815f3f4181d723844f96b5e94acaea272651de6649c40dfdd58797f2"} Oct 29 23:26:34.903182 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:34.903157 5057 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="027af2c9815f3f4181d723844f96b5e94acaea272651de6649c40dfdd58797f2" Oct 29 23:26:34.917396 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:34.917379 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-1a8107-79c55c5798-wmrhd" Oct 29 23:26:34.921682 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:34.921662 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1a8107-6db566b5cf-6jd5j" Oct 29 23:26:34.922191 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:34.922172 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a4256ac-b349-4288-a6c9-92b4ed74e3e6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a4256ac-b349-4288-a6c9-92b4ed74e3e6-plugins-config-volume\") pod \"7a4256ac-b349-4288-a6c9-92b4ed74e3e6\" (UID: \"7a4256ac-b349-4288-a6c9-92b4ed74e3e6\") " Oct 29 23:26:34.922299 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:34.922286 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a4256ac-b349-4288-a6c9-92b4ed74e3e6-kube-api-access-d58sl\" (UniqueName: \"kubernetes.io/projected/7a4256ac-b349-4288-a6c9-92b4ed74e3e6-kube-api-access-d58sl\") pod \"7a4256ac-b349-4288-a6c9-92b4ed74e3e6\" (UID: \"7a4256ac-b349-4288-a6c9-92b4ed74e3e6\") " Oct 29 23:26:34.922523 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:34.922504 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a4256ac-b349-4288-a6c9-92b4ed74e3e6-plugins-config-volume" pod "7a4256ac-b349-4288-a6c9-92b4ed74e3e6" (UID: "7a4256ac-b349-4288-a6c9-92b4ed74e3e6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:26:34.922861 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:34.922843 5057 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a4256ac-b349-4288-a6c9-92b4ed74e3e6-plugins-config-volume\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:26:34.924363 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:34.924325 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a4256ac-b349-4288-a6c9-92b4ed74e3e6-kube-api-access-d58sl" pod "7a4256ac-b349-4288-a6c9-92b4ed74e3e6" (UID: "7a4256ac-b349-4288-a6c9-92b4ed74e3e6"). InnerVolumeSpecName "kube-api-access-d58sl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:26:35.023453 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:35.023421 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56e0eac5-6393-4933-a3ed-4e80fd849bf5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56e0eac5-6393-4933-a3ed-4e80fd849bf5-plugins-config-volume\") pod \"56e0eac5-6393-4933-a3ed-4e80fd849bf5\" (UID: \"56e0eac5-6393-4933-a3ed-4e80fd849bf5\") " Oct 29 23:26:35.023657 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:35.023568 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56e0eac5-6393-4933-a3ed-4e80fd849bf5-kube-api-access-7q5wb\" (UniqueName: \"kubernetes.io/projected/56e0eac5-6393-4933-a3ed-4e80fd849bf5-kube-api-access-7q5wb\") pod \"56e0eac5-6393-4933-a3ed-4e80fd849bf5\" (UID: \"56e0eac5-6393-4933-a3ed-4e80fd849bf5\") " Oct 29 23:26:35.024127 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:35.024102 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d58sl\" (UniqueName: \"kubernetes.io/projected/7a4256ac-b349-4288-a6c9-92b4ed74e3e6-kube-api-access-d58sl\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:26:35.024481 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:35.024449 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56e0eac5-6393-4933-a3ed-4e80fd849bf5-plugins-config-volume" pod "56e0eac5-6393-4933-a3ed-4e80fd849bf5" (UID: "56e0eac5-6393-4933-a3ed-4e80fd849bf5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:26:35.025947 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:35.025920 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56e0eac5-6393-4933-a3ed-4e80fd849bf5-kube-api-access-7q5wb" pod "56e0eac5-6393-4933-a3ed-4e80fd849bf5" (UID: "56e0eac5-6393-4933-a3ed-4e80fd849bf5"). InnerVolumeSpecName "kube-api-access-7q5wb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:26:35.053960 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:35.053918 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:26:35.125940 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:35.125856 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7q5wb\" (UniqueName: \"kubernetes.io/projected/56e0eac5-6393-4933-a3ed-4e80fd849bf5-kube-api-access-7q5wb\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:26:35.125940 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:35.125895 5057 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56e0eac5-6393-4933-a3ed-4e80fd849bf5-plugins-config-volume\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:26:35.916949 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:35.916871 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-1a8107-79c55c5798-wmrhd" Oct 29 23:26:35.917737 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:35.917718 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1a8107-6db566b5cf-6jd5j" Oct 29 23:26:35.925084 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:35.918279 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1a8107-6db566b5cf-6jd5j" event={"ID":"56e0eac5-6393-4933-a3ed-4e80fd849bf5","Type":"ContainerDied","Data":"3a32422386f7178883cd2d4741a229b570956e12385ce7aaefb413123ba8cb41"} Oct 29 23:26:35.925084 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:35.918355 5057 scope.go:122] "RemoveContainer" containerID="06c6274d405753bba9393c5739324878e51136a5c2e3db78aa8096cc9f28c564" Oct 29 23:26:35.987188 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:35.987119 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-1a8107-79c55c5798-wmrhd"] Oct 29 23:26:35.989531 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:35.989502 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:26:35.998748 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:35.998720 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-1a8107-79c55c5798-wmrhd"] Oct 29 23:26:36.025438 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:36.025408 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-1a8107-6db566b5cf-6jd5j"] Oct 29 23:26:36.030418 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:36.030386 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-1a8107-6db566b5cf-6jd5j"] Oct 29 23:26:36.710064 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:36.710029 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:26:37.331664 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:37.331624 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56e0eac5-6393-4933-a3ed-4e80fd849bf5" path="/var/lib/kubelet/pods/56e0eac5-6393-4933-a3ed-4e80fd849bf5/volumes" Oct 29 23:26:37.332244 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:37.332225 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a4256ac-b349-4288-a6c9-92b4ed74e3e6" path="/var/lib/kubelet/pods/7a4256ac-b349-4288-a6c9-92b4ed74e3e6/volumes" Oct 29 23:26:38.254011 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:38.253974 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:26:38.520212 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:38.520130 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-a37090-58c6cbb588-lj97b" podUID="159d07ab-681d-4431-9d7d-e08bcee96624" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:26:38.520212 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:38.520152 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a37090-84ddb8588c-n4gn5" podUID="eb4eaa12-cde8-43d8-9af0-9534a90f9c6b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:26:38.818367 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:38.818249 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-dcacb9-openshift-default-54b777975-58gd9"] Oct 29 23:26:38.819305 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:38.819287 5057 cpu_manager.go:414] "containerMap: removing container" podUID="7a4256ac-b349-4288-a6c9-92b4ed74e3e6" containerName="payload-processing" Oct 29 23:26:38.819399 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:38.819307 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a4256ac-b349-4288-a6c9-92b4ed74e3e6" containerName="payload-processing" Oct 29 23:26:38.819399 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:38.819340 5057 cpu_manager.go:414] "containerMap: removing container" podUID="9d884577-666d-418f-91be-0d0744c3bf09" containerName="revoke-keys" Oct 29 23:26:38.819399 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:38.819347 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d884577-666d-418f-91be-0d0744c3bf09" containerName="revoke-keys" Oct 29 23:26:38.819399 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:38.819368 5057 cpu_manager.go:414] "containerMap: removing container" podUID="56e0eac5-6393-4933-a3ed-4e80fd849bf5" containerName="payload-pre-processing" Oct 29 23:26:38.819399 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:38.819373 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56e0eac5-6393-4933-a3ed-4e80fd849bf5" containerName="payload-pre-processing" Oct 29 23:26:38.819608 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:38.819431 5057 cpu_manager.go:414] "containerMap: removing container" podUID="1cf4cf1a-74e0-4632-97d3-0b743a746ab8" containerName="revoke-keys" Oct 29 23:26:38.819608 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:38.819449 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1cf4cf1a-74e0-4632-97d3-0b743a746ab8" containerName="revoke-keys" Oct 29 23:26:38.819724 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:38.819663 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="1cf4cf1a-74e0-4632-97d3-0b743a746ab8" containerName="revoke-keys" Oct 29 23:26:38.819724 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:38.819679 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a4256ac-b349-4288-a6c9-92b4ed74e3e6" containerName="payload-processing" Oct 29 23:26:38.819724 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:38.819691 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d884577-666d-418f-91be-0d0744c3bf09" containerName="revoke-keys" Oct 29 23:26:38.819724 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:38.819701 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="56e0eac5-6393-4933-a3ed-4e80fd849bf5" containerName="payload-pre-processing" Oct 29 23:26:38.825460 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:38.825442 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-dcacb9-openshift-default-54b777975-58gd9" Oct 29 23:26:38.845924 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:38.845902 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-dcacb9-openshift-default-dockercfg-4g4gh\"" Oct 29 23:26:38.861590 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:38.861559 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-dcacb9-openshift-default-54b777975-58gd9"] Oct 29 23:26:38.882857 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:38.882822 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c8c58af-2d80-4b3e-901f-a0a65338b5d7-istio-token\") pod \"e2e-inf-a-dcacb9-openshift-default-54b777975-58gd9\" (UID: \"3c8c58af-2d80-4b3e-901f-a0a65338b5d7\") " pod="openshift-ingress/e2e-inf-a-dcacb9-openshift-default-54b777975-58gd9" Oct 29 23:26:38.882999 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:38.882932 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c8c58af-2d80-4b3e-901f-a0a65338b5d7-istio-envoy\") pod \"e2e-inf-a-dcacb9-openshift-default-54b777975-58gd9\" (UID: \"3c8c58af-2d80-4b3e-901f-a0a65338b5d7\") " pod="openshift-ingress/e2e-inf-a-dcacb9-openshift-default-54b777975-58gd9" Oct 29 23:26:38.883073 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:38.882997 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4z9hn\" (UniqueName: \"kubernetes.io/projected/3c8c58af-2d80-4b3e-901f-a0a65338b5d7-kube-api-access-4z9hn\") pod \"e2e-inf-a-dcacb9-openshift-default-54b777975-58gd9\" (UID: \"3c8c58af-2d80-4b3e-901f-a0a65338b5d7\") " pod="openshift-ingress/e2e-inf-a-dcacb9-openshift-default-54b777975-58gd9" Oct 29 23:26:38.883141 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:38.883119 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c8c58af-2d80-4b3e-901f-a0a65338b5d7-istio-podinfo\") pod \"e2e-inf-a-dcacb9-openshift-default-54b777975-58gd9\" (UID: \"3c8c58af-2d80-4b3e-901f-a0a65338b5d7\") " pod="openshift-ingress/e2e-inf-a-dcacb9-openshift-default-54b777975-58gd9" Oct 29 23:26:38.883204 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:38.883181 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c8c58af-2d80-4b3e-901f-a0a65338b5d7-workload-socket\") pod \"e2e-inf-a-dcacb9-openshift-default-54b777975-58gd9\" (UID: \"3c8c58af-2d80-4b3e-901f-a0a65338b5d7\") " pod="openshift-ingress/e2e-inf-a-dcacb9-openshift-default-54b777975-58gd9" Oct 29 23:26:38.883304 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:38.883288 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c8c58af-2d80-4b3e-901f-a0a65338b5d7-credential-socket\") pod \"e2e-inf-a-dcacb9-openshift-default-54b777975-58gd9\" (UID: \"3c8c58af-2d80-4b3e-901f-a0a65338b5d7\") " pod="openshift-ingress/e2e-inf-a-dcacb9-openshift-default-54b777975-58gd9" Oct 29 23:26:38.883388 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:38.883364 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c8c58af-2d80-4b3e-901f-a0a65338b5d7-istiod-ca-cert\") pod \"e2e-inf-a-dcacb9-openshift-default-54b777975-58gd9\" (UID: \"3c8c58af-2d80-4b3e-901f-a0a65338b5d7\") " pod="openshift-ingress/e2e-inf-a-dcacb9-openshift-default-54b777975-58gd9" Oct 29 23:26:38.883488 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:38.883395 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c8c58af-2d80-4b3e-901f-a0a65338b5d7-istio-data\") pod \"e2e-inf-a-dcacb9-openshift-default-54b777975-58gd9\" (UID: \"3c8c58af-2d80-4b3e-901f-a0a65338b5d7\") " pod="openshift-ingress/e2e-inf-a-dcacb9-openshift-default-54b777975-58gd9" Oct 29 23:26:38.883532 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:38.883501 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c8c58af-2d80-4b3e-901f-a0a65338b5d7-workload-certs\") pod \"e2e-inf-a-dcacb9-openshift-default-54b777975-58gd9\" (UID: \"3c8c58af-2d80-4b3e-901f-a0a65338b5d7\") " pod="openshift-ingress/e2e-inf-a-dcacb9-openshift-default-54b777975-58gd9" Oct 29 23:26:38.985555 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:38.985524 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c8c58af-2d80-4b3e-901f-a0a65338b5d7-istio-token\") pod \"e2e-inf-a-dcacb9-openshift-default-54b777975-58gd9\" (UID: \"3c8c58af-2d80-4b3e-901f-a0a65338b5d7\") " pod="openshift-ingress/e2e-inf-a-dcacb9-openshift-default-54b777975-58gd9" Oct 29 23:26:38.985749 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:38.985577 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c8c58af-2d80-4b3e-901f-a0a65338b5d7-istio-envoy\") pod \"e2e-inf-a-dcacb9-openshift-default-54b777975-58gd9\" (UID: \"3c8c58af-2d80-4b3e-901f-a0a65338b5d7\") " pod="openshift-ingress/e2e-inf-a-dcacb9-openshift-default-54b777975-58gd9" Oct 29 23:26:38.985749 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:38.985625 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4z9hn\" (UniqueName: \"kubernetes.io/projected/3c8c58af-2d80-4b3e-901f-a0a65338b5d7-kube-api-access-4z9hn\") pod \"e2e-inf-a-dcacb9-openshift-default-54b777975-58gd9\" (UID: \"3c8c58af-2d80-4b3e-901f-a0a65338b5d7\") " pod="openshift-ingress/e2e-inf-a-dcacb9-openshift-default-54b777975-58gd9" Oct 29 23:26:38.985749 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:38.985658 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c8c58af-2d80-4b3e-901f-a0a65338b5d7-istio-podinfo\") pod \"e2e-inf-a-dcacb9-openshift-default-54b777975-58gd9\" (UID: \"3c8c58af-2d80-4b3e-901f-a0a65338b5d7\") " pod="openshift-ingress/e2e-inf-a-dcacb9-openshift-default-54b777975-58gd9" Oct 29 23:26:38.985749 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:38.985688 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c8c58af-2d80-4b3e-901f-a0a65338b5d7-workload-socket\") pod \"e2e-inf-a-dcacb9-openshift-default-54b777975-58gd9\" (UID: \"3c8c58af-2d80-4b3e-901f-a0a65338b5d7\") " pod="openshift-ingress/e2e-inf-a-dcacb9-openshift-default-54b777975-58gd9" Oct 29 23:26:38.986002 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:38.985763 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c8c58af-2d80-4b3e-901f-a0a65338b5d7-credential-socket\") pod \"e2e-inf-a-dcacb9-openshift-default-54b777975-58gd9\" (UID: \"3c8c58af-2d80-4b3e-901f-a0a65338b5d7\") " pod="openshift-ingress/e2e-inf-a-dcacb9-openshift-default-54b777975-58gd9" Oct 29 23:26:38.986002 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:38.985790 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c8c58af-2d80-4b3e-901f-a0a65338b5d7-istiod-ca-cert\") pod \"e2e-inf-a-dcacb9-openshift-default-54b777975-58gd9\" (UID: \"3c8c58af-2d80-4b3e-901f-a0a65338b5d7\") " pod="openshift-ingress/e2e-inf-a-dcacb9-openshift-default-54b777975-58gd9" Oct 29 23:26:38.986002 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:38.985817 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c8c58af-2d80-4b3e-901f-a0a65338b5d7-istio-data\") pod \"e2e-inf-a-dcacb9-openshift-default-54b777975-58gd9\" (UID: \"3c8c58af-2d80-4b3e-901f-a0a65338b5d7\") " pod="openshift-ingress/e2e-inf-a-dcacb9-openshift-default-54b777975-58gd9" Oct 29 23:26:38.986002 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:38.985882 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c8c58af-2d80-4b3e-901f-a0a65338b5d7-workload-certs\") pod \"e2e-inf-a-dcacb9-openshift-default-54b777975-58gd9\" (UID: \"3c8c58af-2d80-4b3e-901f-a0a65338b5d7\") " pod="openshift-ingress/e2e-inf-a-dcacb9-openshift-default-54b777975-58gd9" Oct 29 23:26:38.986214 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:38.986171 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c8c58af-2d80-4b3e-901f-a0a65338b5d7-credential-socket\") pod \"e2e-inf-a-dcacb9-openshift-default-54b777975-58gd9\" (UID: \"3c8c58af-2d80-4b3e-901f-a0a65338b5d7\") " pod="openshift-ingress/e2e-inf-a-dcacb9-openshift-default-54b777975-58gd9" Oct 29 23:26:38.986273 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:38.986208 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c8c58af-2d80-4b3e-901f-a0a65338b5d7-workload-socket\") pod \"e2e-inf-a-dcacb9-openshift-default-54b777975-58gd9\" (UID: \"3c8c58af-2d80-4b3e-901f-a0a65338b5d7\") " pod="openshift-ingress/e2e-inf-a-dcacb9-openshift-default-54b777975-58gd9" Oct 29 23:26:38.986428 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:38.986410 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c8c58af-2d80-4b3e-901f-a0a65338b5d7-istio-data\") pod \"e2e-inf-a-dcacb9-openshift-default-54b777975-58gd9\" (UID: \"3c8c58af-2d80-4b3e-901f-a0a65338b5d7\") " pod="openshift-ingress/e2e-inf-a-dcacb9-openshift-default-54b777975-58gd9" Oct 29 23:26:38.986514 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:38.986489 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c8c58af-2d80-4b3e-901f-a0a65338b5d7-workload-certs\") pod \"e2e-inf-a-dcacb9-openshift-default-54b777975-58gd9\" (UID: \"3c8c58af-2d80-4b3e-901f-a0a65338b5d7\") " pod="openshift-ingress/e2e-inf-a-dcacb9-openshift-default-54b777975-58gd9" Oct 29 23:26:38.986722 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:38.986705 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c8c58af-2d80-4b3e-901f-a0a65338b5d7-istiod-ca-cert\") pod \"e2e-inf-a-dcacb9-openshift-default-54b777975-58gd9\" (UID: \"3c8c58af-2d80-4b3e-901f-a0a65338b5d7\") " pod="openshift-ingress/e2e-inf-a-dcacb9-openshift-default-54b777975-58gd9" Oct 29 23:26:38.988273 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:38.988252 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c8c58af-2d80-4b3e-901f-a0a65338b5d7-istio-podinfo\") pod \"e2e-inf-a-dcacb9-openshift-default-54b777975-58gd9\" (UID: \"3c8c58af-2d80-4b3e-901f-a0a65338b5d7\") " pod="openshift-ingress/e2e-inf-a-dcacb9-openshift-default-54b777975-58gd9" Oct 29 23:26:38.988400 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:38.988381 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c8c58af-2d80-4b3e-901f-a0a65338b5d7-istio-envoy\") pod \"e2e-inf-a-dcacb9-openshift-default-54b777975-58gd9\" (UID: \"3c8c58af-2d80-4b3e-901f-a0a65338b5d7\") " pod="openshift-ingress/e2e-inf-a-dcacb9-openshift-default-54b777975-58gd9" Oct 29 23:26:39.008655 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:39.008626 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c8c58af-2d80-4b3e-901f-a0a65338b5d7-istio-token\") pod \"e2e-inf-a-dcacb9-openshift-default-54b777975-58gd9\" (UID: \"3c8c58af-2d80-4b3e-901f-a0a65338b5d7\") " pod="openshift-ingress/e2e-inf-a-dcacb9-openshift-default-54b777975-58gd9" Oct 29 23:26:39.008754 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:39.008725 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4z9hn\" (UniqueName: \"kubernetes.io/projected/3c8c58af-2d80-4b3e-901f-a0a65338b5d7-kube-api-access-4z9hn\") pod \"e2e-inf-a-dcacb9-openshift-default-54b777975-58gd9\" (UID: \"3c8c58af-2d80-4b3e-901f-a0a65338b5d7\") " pod="openshift-ingress/e2e-inf-a-dcacb9-openshift-default-54b777975-58gd9" Oct 29 23:26:39.142027 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:39.141941 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-dcacb9-openshift-default-54b777975-58gd9" Oct 29 23:26:39.467523 ip-10-0-140-74 kubenswrapper[5057]: W1029 23:26:39.467494 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c8c58af_2d80_4b3e_901f_a0a65338b5d7.slice/crio-87c7f93616689bf37fb669f4724805448037cac4b53976d90f2dc8ea1b3a2c46 WatchSource:0}: Error finding container 87c7f93616689bf37fb669f4724805448037cac4b53976d90f2dc8ea1b3a2c46: Status 404 returned error can't find the container with id 87c7f93616689bf37fb669f4724805448037cac4b53976d90f2dc8ea1b3a2c46 Oct 29 23:26:39.468207 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:39.468100 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-dcacb9-openshift-default-54b777975-58gd9"] Oct 29 23:26:39.470635 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:39.470600 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 23:26:39.470740 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:39.470669 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 23:26:39.470740 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:39.470711 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 23:26:39.855395 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:39.855289 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:26:39.965858 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:39.965817 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-dcacb9-openshift-default-54b777975-58gd9" event={"ID":"3c8c58af-2d80-4b3e-901f-a0a65338b5d7","Type":"ContainerStarted","Data":"7b92baa13c881e9839e072042cbcb7a8ca8d9c87ac1cc91dccad45334fe527d5"} Oct 29 23:26:39.966068 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:39.965911 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-dcacb9-openshift-default-54b777975-58gd9" event={"ID":"3c8c58af-2d80-4b3e-901f-a0a65338b5d7","Type":"ContainerStarted","Data":"87c7f93616689bf37fb669f4724805448037cac4b53976d90f2dc8ea1b3a2c46"} Oct 29 23:26:40.015476 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:40.015411 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-dcacb9-openshift-default-54b777975-58gd9" podStartSLOduration=2.015391176 podStartE2EDuration="2.015391176s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:26:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:26:40.009709522 +0000 UTC m=+3180.792922950" watchObservedRunningTime="2026-10-29 23:26:40.015391176 +0000 UTC m=+3180.798604619" Oct 29 23:26:40.142281 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:40.142187 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-dcacb9-openshift-default-54b777975-58gd9" Oct 29 23:26:40.144315 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:40.144287 5057 patch_prober.go:29] interesting pod/e2e-inf-a-dcacb9-openshift-default-54b777975-58gd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.255:15021/healthz/ready\": dial tcp 10.132.0.255:15021: connect: connection refused" start-of-body= Oct 29 23:26:40.144460 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:40.144364 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-dcacb9-openshift-default-54b777975-58gd9" podUID="3c8c58af-2d80-4b3e-901f-a0a65338b5d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.255:15021/healthz/ready\": dial tcp 10.132.0.255:15021: connect: connection refused" Oct 29 23:26:40.226049 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:40.225993 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:26:40.346849 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:40.346812 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:26:40.610064 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:40.609777 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:26:41.142735 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:41.142699 5057 patch_prober.go:29] interesting pod/e2e-inf-a-dcacb9-openshift-default-54b777975-58gd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.255:15021/healthz/ready\": dial tcp 10.132.0.255:15021: connect: connection refused" start-of-body= Oct 29 23:26:41.143214 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:41.142767 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-dcacb9-openshift-default-54b777975-58gd9" podUID="3c8c58af-2d80-4b3e-901f-a0a65338b5d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.255:15021/healthz/ready\": dial tcp 10.132.0.255:15021: connect: connection refused" Oct 29 23:26:41.146519 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:41.146493 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-024aa6-wzpcp"] Oct 29 23:26:41.151940 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:41.151919 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-024aa6-wzpcp" Oct 29 23:26:41.169464 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:41.169443 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-024aa6-wzpcp"] Oct 29 23:26:41.171759 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:41.171732 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:26:41.174262 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:41.174241 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dmx57\"" Oct 29 23:26:41.241868 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:41.241830 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec6f58bd-af5d-40f6-a333-d355d71fe880-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-024aa6-wzpcp\" (UID: \"ec6f58bd-af5d-40f6-a333-d355d71fe880\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-024aa6-wzpcp" Oct 29 23:26:41.242055 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:41.241983 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qt2q4\" (UniqueName: \"kubernetes.io/projected/ec6f58bd-af5d-40f6-a333-d355d71fe880-kube-api-access-qt2q4\") pod \"maas-api-revoke-keys-e2e-inf-b-024aa6-wzpcp\" (UID: \"ec6f58bd-af5d-40f6-a333-d355d71fe880\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-024aa6-wzpcp" Oct 29 23:26:41.242137 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:41.242121 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec6f58bd-af5d-40f6-a333-d355d71fe880-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-024aa6-wzpcp\" (UID: \"ec6f58bd-af5d-40f6-a333-d355d71fe880\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-024aa6-wzpcp" Oct 29 23:26:41.343819 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:41.343790 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qt2q4\" (UniqueName: \"kubernetes.io/projected/ec6f58bd-af5d-40f6-a333-d355d71fe880-kube-api-access-qt2q4\") pod \"maas-api-revoke-keys-e2e-inf-b-024aa6-wzpcp\" (UID: \"ec6f58bd-af5d-40f6-a333-d355d71fe880\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-024aa6-wzpcp" Oct 29 23:26:41.343985 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:41.343899 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec6f58bd-af5d-40f6-a333-d355d71fe880-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-024aa6-wzpcp\" (UID: \"ec6f58bd-af5d-40f6-a333-d355d71fe880\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-024aa6-wzpcp" Oct 29 23:26:41.344061 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:41.344049 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec6f58bd-af5d-40f6-a333-d355d71fe880-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-024aa6-wzpcp\" (UID: \"ec6f58bd-af5d-40f6-a333-d355d71fe880\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-024aa6-wzpcp" Oct 29 23:26:41.344571 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:41.344552 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec6f58bd-af5d-40f6-a333-d355d71fe880-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-024aa6-wzpcp\" (UID: \"ec6f58bd-af5d-40f6-a333-d355d71fe880\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-024aa6-wzpcp" Oct 29 23:26:41.346262 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:41.346240 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec6f58bd-af5d-40f6-a333-d355d71fe880-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-024aa6-wzpcp\" (UID: \"ec6f58bd-af5d-40f6-a333-d355d71fe880\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-024aa6-wzpcp" Oct 29 23:26:41.375029 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:41.375000 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qt2q4\" (UniqueName: \"kubernetes.io/projected/ec6f58bd-af5d-40f6-a333-d355d71fe880-kube-api-access-qt2q4\") pod \"maas-api-revoke-keys-e2e-inf-b-024aa6-wzpcp\" (UID: \"ec6f58bd-af5d-40f6-a333-d355d71fe880\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-024aa6-wzpcp" Oct 29 23:26:41.468837 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:41.468806 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-024aa6-wzpcp" Oct 29 23:26:41.621147 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:41.618847 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:26:41.816255 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:41.816225 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-024aa6-wzpcp"] Oct 29 23:26:42.011008 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:42.010976 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-024aa6-wzpcp" event={"ID":"ec6f58bd-af5d-40f6-a333-d355d71fe880","Type":"ContainerStarted","Data":"2b0a7c06184a54228a9256bfc916a6e9e40883da7405c5dc64120ec720d9dfca"} Oct 29 23:26:42.143579 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:42.143503 5057 patch_prober.go:29] interesting pod/e2e-inf-a-dcacb9-openshift-default-54b777975-58gd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.255:15021/healthz/ready\": dial tcp 10.132.0.255:15021: connect: connection refused" start-of-body= Oct 29 23:26:42.143579 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:42.143554 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-dcacb9-openshift-default-54b777975-58gd9" podUID="3c8c58af-2d80-4b3e-901f-a0a65338b5d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.255:15021/healthz/ready\": dial tcp 10.132.0.255:15021: connect: connection refused" Oct 29 23:26:43.039724 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:43.038557 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-024aa6-wzpcp" event={"ID":"ec6f58bd-af5d-40f6-a333-d355d71fe880","Type":"ContainerStarted","Data":"1c9205531c5cdc0460fb0cd069a46026bdb0d8e482c58a02ff1dcd97d2996f1b"} Oct 29 23:26:43.063761 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:43.063723 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:26:43.123195 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:43.122720 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-024aa6-wzpcp" podStartSLOduration=2.12270043 podStartE2EDuration="2.12270043s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:26:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:26:43.115988237 +0000 UTC m=+3183.899201701" watchObservedRunningTime="2026-10-29 23:26:43.12270043 +0000 UTC m=+3183.905913857" Oct 29 23:26:43.144569 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:43.143975 5057 patch_prober.go:29] interesting pod/e2e-inf-a-dcacb9-openshift-default-54b777975-58gd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.255:15021/healthz/ready\": dial tcp 10.132.0.255:15021: connect: connection refused" start-of-body= Oct 29 23:26:43.144569 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:43.144015 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-dcacb9-openshift-default-54b777975-58gd9" podUID="3c8c58af-2d80-4b3e-901f-a0a65338b5d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.255:15021/healthz/ready\": dial tcp 10.132.0.255:15021: connect: connection refused" Oct 29 23:26:43.172513 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:43.172473 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:26:43.869098 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:43.869052 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:26:43.900356 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:43.896471 5057 scope.go:122] "RemoveContainer" containerID="fb02052ef8934d6de0c89be624228de4a404489ba44daa99355046cc5a7341bb" Oct 29 23:26:43.922062 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:43.922026 5057 scope.go:122] "RemoveContainer" containerID="2a646f693a7588e1442fe7d7821877f1e595bcad97a1eff4feffbd0266d87083" Oct 29 23:26:44.011015 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:44.010941 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:26:44.050672 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:44.050634 5057 generic.go:356] "Generic (PLEG): container finished" podID="ec6f58bd-af5d-40f6-a333-d355d71fe880" containerID="1c9205531c5cdc0460fb0cd069a46026bdb0d8e482c58a02ff1dcd97d2996f1b" exitCode=0 Oct 29 23:26:44.050823 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:44.050680 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-024aa6-wzpcp" event={"ID":"ec6f58bd-af5d-40f6-a333-d355d71fe880","Type":"ContainerDied","Data":"1c9205531c5cdc0460fb0cd069a46026bdb0d8e482c58a02ff1dcd97d2996f1b"} Oct 29 23:26:44.075275 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:44.075244 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1a8107-b79sh"] Oct 29 23:26:44.084360 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:44.083612 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1a8107-b79sh"] Oct 29 23:26:44.143053 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:44.142960 5057 patch_prober.go:29] interesting pod/e2e-inf-a-dcacb9-openshift-default-54b777975-58gd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.255:15021/healthz/ready\": dial tcp 10.132.0.255:15021: connect: connection refused" start-of-body= Oct 29 23:26:44.143053 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:44.143024 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-dcacb9-openshift-default-54b777975-58gd9" podUID="3c8c58af-2d80-4b3e-901f-a0a65338b5d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.255:15021/healthz/ready\": dial tcp 10.132.0.255:15021: connect: connection refused" Oct 29 23:26:44.250148 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:44.250116 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:26:44.538735 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:44.538692 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:26:45.142994 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:45.142956 5057 patch_prober.go:29] interesting pod/e2e-inf-a-dcacb9-openshift-default-54b777975-58gd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.255:15021/healthz/ready\": dial tcp 10.132.0.255:15021: connect: connection refused" start-of-body= Oct 29 23:26:45.143181 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:45.143006 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-dcacb9-openshift-default-54b777975-58gd9" podUID="3c8c58af-2d80-4b3e-901f-a0a65338b5d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.255:15021/healthz/ready\": dial tcp 10.132.0.255:15021: connect: connection refused" Oct 29 23:26:45.317668 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:45.317599 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-024aa6-wzpcp" Oct 29 23:26:45.332709 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:45.332682 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0ea5d85-1751-4c56-ab9c-b51366e93322" path="/var/lib/kubelet/pods/e0ea5d85-1751-4c56-ab9c-b51366e93322/volumes" Oct 29 23:26:45.451947 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:45.451910 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ec6f58bd-af5d-40f6-a333-d355d71fe880-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec6f58bd-af5d-40f6-a333-d355d71fe880-maas-api-xks-ca\") pod \"ec6f58bd-af5d-40f6-a333-d355d71fe880\" (UID: \"ec6f58bd-af5d-40f6-a333-d355d71fe880\") " Oct 29 23:26:45.452146 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:45.451979 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec6f58bd-af5d-40f6-a333-d355d71fe880-kube-api-access-qt2q4\" (UniqueName: \"kubernetes.io/projected/ec6f58bd-af5d-40f6-a333-d355d71fe880-kube-api-access-qt2q4\") pod \"ec6f58bd-af5d-40f6-a333-d355d71fe880\" (UID: \"ec6f58bd-af5d-40f6-a333-d355d71fe880\") " Oct 29 23:26:45.452146 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:45.452094 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec6f58bd-af5d-40f6-a333-d355d71fe880-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec6f58bd-af5d-40f6-a333-d355d71fe880-maas-api-openshift-ca\") pod \"ec6f58bd-af5d-40f6-a333-d355d71fe880\" (UID: \"ec6f58bd-af5d-40f6-a333-d355d71fe880\") " Oct 29 23:26:45.452717 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:45.452674 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec6f58bd-af5d-40f6-a333-d355d71fe880-maas-api-openshift-ca" pod "ec6f58bd-af5d-40f6-a333-d355d71fe880" (UID: "ec6f58bd-af5d-40f6-a333-d355d71fe880"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:26:45.453356 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:45.453319 5057 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec6f58bd-af5d-40f6-a333-d355d71fe880-maas-api-openshift-ca\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:26:45.454312 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:45.454282 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec6f58bd-af5d-40f6-a333-d355d71fe880-maas-api-xks-ca" pod "ec6f58bd-af5d-40f6-a333-d355d71fe880" (UID: "ec6f58bd-af5d-40f6-a333-d355d71fe880"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:26:45.454449 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:45.454360 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec6f58bd-af5d-40f6-a333-d355d71fe880-kube-api-access-qt2q4" pod "ec6f58bd-af5d-40f6-a333-d355d71fe880" (UID: "ec6f58bd-af5d-40f6-a333-d355d71fe880"). InnerVolumeSpecName "kube-api-access-qt2q4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:26:45.554769 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:45.554736 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qt2q4\" (UniqueName: \"kubernetes.io/projected/ec6f58bd-af5d-40f6-a333-d355d71fe880-kube-api-access-qt2q4\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:26:45.554769 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:45.554766 5057 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec6f58bd-af5d-40f6-a333-d355d71fe880-maas-api-xks-ca\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:26:45.918572 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:45.918529 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:26:46.072087 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:46.072055 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-024aa6-wzpcp" Oct 29 23:26:46.072087 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:46.072077 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-024aa6-wzpcp" event={"ID":"ec6f58bd-af5d-40f6-a333-d355d71fe880","Type":"ContainerDied","Data":"2b0a7c06184a54228a9256bfc916a6e9e40883da7405c5dc64120ec720d9dfca"} Oct 29 23:26:46.072373 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:46.072129 5057 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2b0a7c06184a54228a9256bfc916a6e9e40883da7405c5dc64120ec720d9dfca" Oct 29 23:26:46.148093 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:46.148062 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-dcacb9-openshift-default-54b777975-58gd9" Oct 29 23:26:46.148513 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:46.148489 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-dcacb9-openshift-default-54b777975-58gd9" Oct 29 23:26:46.149207 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:46.149187 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-dcacb9-openshift-default-54b777975-58gd9" Oct 29 23:26:47.615093 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:47.615051 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:26:48.519465 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:48.519422 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a37090-84ddb8588c-n4gn5" podUID="eb4eaa12-cde8-43d8-9af0-9534a90f9c6b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:26:48.519680 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:48.519436 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-a37090-58c6cbb588-lj97b" podUID="159d07ab-681d-4431-9d7d-e08bcee96624" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:26:49.029366 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:26:49.029312 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d2dnn" containerName="extractor" Oct 29 23:26:49.106905 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:49.106863 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:26:50.819705 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:50.819611 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:26:50.965477 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:50.965435 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:26:51.086140 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:51.086093 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:26:51.335439 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:51.324797 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:26:51.352262 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:51.352225 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-1a8107-openshift-default-7fff599fbf-rtjrx"] Oct 29 23:26:51.352525 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:51.352474 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-1a8107-openshift-default-7fff599fbf-rtjrx" podUID="1a1427f2-b35c-4355-9997-cde832700d7f" containerName="istio-proxy" containerID="cri-o://21b23ab4ce637a39eb4a94f9a00935ce39aaeca6fc3c2da9178ee83903e3ebd2" gracePeriod=30 Oct 29 23:26:51.652683 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:51.652652 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-a37090-58c6cbb588-lj97b" Oct 29 23:26:51.720288 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:51.720259 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a37090-84ddb8588c-n4gn5" Oct 29 23:26:51.764699 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:51.764652 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/159d07ab-681d-4431-9d7d-e08bcee96624-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/159d07ab-681d-4431-9d7d-e08bcee96624-plugins-config-volume\") pod \"159d07ab-681d-4431-9d7d-e08bcee96624\" (UID: \"159d07ab-681d-4431-9d7d-e08bcee96624\") " Oct 29 23:26:51.764898 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:51.764745 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/159d07ab-681d-4431-9d7d-e08bcee96624-kube-api-access-dxk8j\" (UniqueName: \"kubernetes.io/projected/159d07ab-681d-4431-9d7d-e08bcee96624-kube-api-access-dxk8j\") pod \"159d07ab-681d-4431-9d7d-e08bcee96624\" (UID: \"159d07ab-681d-4431-9d7d-e08bcee96624\") " Oct 29 23:26:51.765237 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:51.765190 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/159d07ab-681d-4431-9d7d-e08bcee96624-plugins-config-volume" pod "159d07ab-681d-4431-9d7d-e08bcee96624" (UID: "159d07ab-681d-4431-9d7d-e08bcee96624"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:26:51.766183 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:51.766150 5057 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/159d07ab-681d-4431-9d7d-e08bcee96624-plugins-config-volume\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:26:51.769070 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:51.769043 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/159d07ab-681d-4431-9d7d-e08bcee96624-kube-api-access-dxk8j" pod "159d07ab-681d-4431-9d7d-e08bcee96624" (UID: "159d07ab-681d-4431-9d7d-e08bcee96624"). InnerVolumeSpecName "kube-api-access-dxk8j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:26:51.798543 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:51.798518 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-024aa6-5599897986-xltss"] Oct 29 23:26:51.837112 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:51.837078 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-024aa6-b99d57cd7-pslx4"] Oct 29 23:26:51.867592 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:51.867531 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb4eaa12-cde8-43d8-9af0-9534a90f9c6b-kube-api-access-r7gk5\" (UniqueName: \"kubernetes.io/projected/eb4eaa12-cde8-43d8-9af0-9534a90f9c6b-kube-api-access-r7gk5\") pod \"eb4eaa12-cde8-43d8-9af0-9534a90f9c6b\" (UID: \"eb4eaa12-cde8-43d8-9af0-9534a90f9c6b\") " Oct 29 23:26:51.867592 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:51.867570 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb4eaa12-cde8-43d8-9af0-9534a90f9c6b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb4eaa12-cde8-43d8-9af0-9534a90f9c6b-plugins-config-volume\") pod \"eb4eaa12-cde8-43d8-9af0-9534a90f9c6b\" (UID: \"eb4eaa12-cde8-43d8-9af0-9534a90f9c6b\") " Oct 29 23:26:51.868008 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:51.867977 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb4eaa12-cde8-43d8-9af0-9534a90f9c6b-plugins-config-volume" pod "eb4eaa12-cde8-43d8-9af0-9534a90f9c6b" (UID: "eb4eaa12-cde8-43d8-9af0-9534a90f9c6b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:26:51.868086 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:51.868049 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dxk8j\" (UniqueName: \"kubernetes.io/projected/159d07ab-681d-4431-9d7d-e08bcee96624-kube-api-access-dxk8j\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:26:51.869605 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:51.869585 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb4eaa12-cde8-43d8-9af0-9534a90f9c6b-kube-api-access-r7gk5" pod "eb4eaa12-cde8-43d8-9af0-9534a90f9c6b" (UID: "eb4eaa12-cde8-43d8-9af0-9534a90f9c6b"). InnerVolumeSpecName "kube-api-access-r7gk5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:26:51.960733 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:51.960692 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:26:51.972397 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:51.969750 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r7gk5\" (UniqueName: \"kubernetes.io/projected/eb4eaa12-cde8-43d8-9af0-9534a90f9c6b-kube-api-access-r7gk5\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:26:51.972397 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:51.969785 5057 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb4eaa12-cde8-43d8-9af0-9534a90f9c6b-plugins-config-volume\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:26:52.159725 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:52.150567 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:26:52.159725 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:52.159325 5057 generic.go:356] "Generic (PLEG): container finished" podID="159d07ab-681d-4431-9d7d-e08bcee96624" containerID="36f85597af859c5e4303f518d0d31d815a44e513f74da021785b2120bba5b1e3" exitCode=0 Oct 29 23:26:52.159725 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:52.159611 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-a37090-58c6cbb588-lj97b" event={"ID":"159d07ab-681d-4431-9d7d-e08bcee96624","Type":"ContainerDied","Data":"36f85597af859c5e4303f518d0d31d815a44e513f74da021785b2120bba5b1e3"} Oct 29 23:26:52.159725 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:52.159677 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-a37090-58c6cbb588-lj97b" event={"ID":"159d07ab-681d-4431-9d7d-e08bcee96624","Type":"ContainerDied","Data":"f3798d6b125fc942cadd4271397f2469da96052179a385b870d4d43738036e8a"} Oct 29 23:26:52.159725 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:52.159709 5057 scope.go:122] "RemoveContainer" containerID="36f85597af859c5e4303f518d0d31d815a44e513f74da021785b2120bba5b1e3" Oct 29 23:26:52.160097 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:52.159928 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-a37090-58c6cbb588-lj97b" Oct 29 23:26:52.172560 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:52.169108 5057 generic.go:356] "Generic (PLEG): container finished" podID="eb4eaa12-cde8-43d8-9af0-9534a90f9c6b" containerID="20440706175fed5a0dc35c1ba1e6aa0329ba5032d54d7f576193d92285a8d72a" exitCode=0 Oct 29 23:26:52.191079 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:52.172882 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a37090-84ddb8588c-n4gn5" event={"ID":"eb4eaa12-cde8-43d8-9af0-9534a90f9c6b","Type":"ContainerDied","Data":"20440706175fed5a0dc35c1ba1e6aa0329ba5032d54d7f576193d92285a8d72a"} Oct 29 23:26:52.191079 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:52.173018 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a37090-84ddb8588c-n4gn5" event={"ID":"eb4eaa12-cde8-43d8-9af0-9534a90f9c6b","Type":"ContainerDied","Data":"a39a8bae9b75b04d7e49a24bf51f46637908d417f4e8c8e276e8a8e9f7353f5e"} Oct 29 23:26:52.191079 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:52.177517 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a37090-84ddb8588c-n4gn5" Oct 29 23:26:52.254805 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:52.254766 5057 scope.go:122] "RemoveContainer" containerID="36f85597af859c5e4303f518d0d31d815a44e513f74da021785b2120bba5b1e3" Oct 29 23:26:52.255272 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:26:52.255237 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36f85597af859c5e4303f518d0d31d815a44e513f74da021785b2120bba5b1e3\": container with ID starting with 36f85597af859c5e4303f518d0d31d815a44e513f74da021785b2120bba5b1e3 not found: ID does not exist" containerID="36f85597af859c5e4303f518d0d31d815a44e513f74da021785b2120bba5b1e3" Oct 29 23:26:52.255410 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:52.255283 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36f85597af859c5e4303f518d0d31d815a44e513f74da021785b2120bba5b1e3"} err="failed to get container status \"36f85597af859c5e4303f518d0d31d815a44e513f74da021785b2120bba5b1e3\": rpc error: code = NotFound desc = could not find container \"36f85597af859c5e4303f518d0d31d815a44e513f74da021785b2120bba5b1e3\": container with ID starting with 36f85597af859c5e4303f518d0d31d815a44e513f74da021785b2120bba5b1e3 not found: ID does not exist" Oct 29 23:26:52.255410 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:52.255311 5057 scope.go:122] "RemoveContainer" containerID="20440706175fed5a0dc35c1ba1e6aa0329ba5032d54d7f576193d92285a8d72a" Oct 29 23:26:52.289055 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:52.289033 5057 scope.go:122] "RemoveContainer" containerID="20440706175fed5a0dc35c1ba1e6aa0329ba5032d54d7f576193d92285a8d72a" Oct 29 23:26:52.290229 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:26:52.290202 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20440706175fed5a0dc35c1ba1e6aa0329ba5032d54d7f576193d92285a8d72a\": container with ID starting with 20440706175fed5a0dc35c1ba1e6aa0329ba5032d54d7f576193d92285a8d72a not found: ID does not exist" containerID="20440706175fed5a0dc35c1ba1e6aa0329ba5032d54d7f576193d92285a8d72a" Oct 29 23:26:52.290510 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:52.290483 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20440706175fed5a0dc35c1ba1e6aa0329ba5032d54d7f576193d92285a8d72a"} err="failed to get container status \"20440706175fed5a0dc35c1ba1e6aa0329ba5032d54d7f576193d92285a8d72a\": rpc error: code = NotFound desc = could not find container \"20440706175fed5a0dc35c1ba1e6aa0329ba5032d54d7f576193d92285a8d72a\": container with ID starting with 20440706175fed5a0dc35c1ba1e6aa0329ba5032d54d7f576193d92285a8d72a not found: ID does not exist" Oct 29 23:26:52.292382 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:52.292219 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:26:52.320467 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:52.320443 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a37090-84ddb8588c-n4gn5"] Oct 29 23:26:52.337386 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:52.337365 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a37090-84ddb8588c-n4gn5"] Oct 29 23:26:52.368123 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:52.368094 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a37090-58c6cbb588-lj97b"] Oct 29 23:26:52.372981 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:52.372957 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a37090-58c6cbb588-lj97b"] Oct 29 23:26:52.402510 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:52.402474 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:26:52.624127 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:52.624092 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:26:52.797867 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:52.797833 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:26:53.334123 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:53.334088 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="159d07ab-681d-4431-9d7d-e08bcee96624" path="/var/lib/kubelet/pods/159d07ab-681d-4431-9d7d-e08bcee96624/volumes" Oct 29 23:26:53.334621 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:53.334583 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb4eaa12-cde8-43d8-9af0-9534a90f9c6b" path="/var/lib/kubelet/pods/eb4eaa12-cde8-43d8-9af0-9534a90f9c6b/volumes" Oct 29 23:26:53.701094 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:53.701064 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-dcacb9-767c887c7b-jrk26"] Oct 29 23:26:53.723419 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:53.723385 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-dcacb9-86d7d89559-xk4m6"] Oct 29 23:26:53.897192 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:53.897157 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:26:54.006223 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:54.006121 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:26:54.451046 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:54.451008 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:26:56.320626 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:56.320590 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:26:56.690859 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:56.690826 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-1a8107-openshift-default-7fff599fbf-rtjrx" Oct 29 23:26:56.772088 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:56.772055 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a1427f2-b35c-4355-9997-cde832700d7f-kube-api-access-9nz8t\" (UniqueName: \"kubernetes.io/projected/1a1427f2-b35c-4355-9997-cde832700d7f-kube-api-access-9nz8t\") pod \"1a1427f2-b35c-4355-9997-cde832700d7f\" (UID: \"1a1427f2-b35c-4355-9997-cde832700d7f\") " Oct 29 23:26:56.772297 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:56.772188 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a1427f2-b35c-4355-9997-cde832700d7f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a1427f2-b35c-4355-9997-cde832700d7f-workload-socket\") pod \"1a1427f2-b35c-4355-9997-cde832700d7f\" (UID: \"1a1427f2-b35c-4355-9997-cde832700d7f\") " Oct 29 23:26:56.772297 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:56.772247 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a1427f2-b35c-4355-9997-cde832700d7f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a1427f2-b35c-4355-9997-cde832700d7f-istio-data\") pod \"1a1427f2-b35c-4355-9997-cde832700d7f\" (UID: \"1a1427f2-b35c-4355-9997-cde832700d7f\") " Oct 29 23:26:56.772297 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:56.772293 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a1427f2-b35c-4355-9997-cde832700d7f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a1427f2-b35c-4355-9997-cde832700d7f-istiod-ca-cert\") pod \"1a1427f2-b35c-4355-9997-cde832700d7f\" (UID: \"1a1427f2-b35c-4355-9997-cde832700d7f\") " Oct 29 23:26:56.772601 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:56.772325 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a1427f2-b35c-4355-9997-cde832700d7f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a1427f2-b35c-4355-9997-cde832700d7f-workload-certs\") pod \"1a1427f2-b35c-4355-9997-cde832700d7f\" (UID: \"1a1427f2-b35c-4355-9997-cde832700d7f\") " Oct 29 23:26:56.772601 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:56.772553 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a1427f2-b35c-4355-9997-cde832700d7f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a1427f2-b35c-4355-9997-cde832700d7f-istio-envoy\") pod \"1a1427f2-b35c-4355-9997-cde832700d7f\" (UID: \"1a1427f2-b35c-4355-9997-cde832700d7f\") " Oct 29 23:26:56.772719 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:56.772602 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1a1427f2-b35c-4355-9997-cde832700d7f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a1427f2-b35c-4355-9997-cde832700d7f-istio-podinfo\") pod \"1a1427f2-b35c-4355-9997-cde832700d7f\" (UID: \"1a1427f2-b35c-4355-9997-cde832700d7f\") " Oct 29 23:26:56.772719 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:56.772700 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a1427f2-b35c-4355-9997-cde832700d7f-istio-token\" (UniqueName: \"kubernetes.io/projected/1a1427f2-b35c-4355-9997-cde832700d7f-istio-token\") pod \"1a1427f2-b35c-4355-9997-cde832700d7f\" (UID: \"1a1427f2-b35c-4355-9997-cde832700d7f\") " Oct 29 23:26:56.772833 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:56.772729 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a1427f2-b35c-4355-9997-cde832700d7f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a1427f2-b35c-4355-9997-cde832700d7f-credential-socket\") pod \"1a1427f2-b35c-4355-9997-cde832700d7f\" (UID: \"1a1427f2-b35c-4355-9997-cde832700d7f\") " Oct 29 23:26:56.772833 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:56.772789 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a1427f2-b35c-4355-9997-cde832700d7f-workload-certs" pod "1a1427f2-b35c-4355-9997-cde832700d7f" (UID: "1a1427f2-b35c-4355-9997-cde832700d7f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:26:56.772833 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:56.772798 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a1427f2-b35c-4355-9997-cde832700d7f-istiod-ca-cert" pod "1a1427f2-b35c-4355-9997-cde832700d7f" (UID: "1a1427f2-b35c-4355-9997-cde832700d7f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:26:56.773035 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:56.772991 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a1427f2-b35c-4355-9997-cde832700d7f-istio-data" pod "1a1427f2-b35c-4355-9997-cde832700d7f" (UID: "1a1427f2-b35c-4355-9997-cde832700d7f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:26:56.773035 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:56.773013 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a1427f2-b35c-4355-9997-cde832700d7f-credential-socket" pod "1a1427f2-b35c-4355-9997-cde832700d7f" (UID: "1a1427f2-b35c-4355-9997-cde832700d7f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:26:56.774022 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:56.773788 5057 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a1427f2-b35c-4355-9997-cde832700d7f-credential-socket\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:26:56.774022 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:56.773817 5057 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a1427f2-b35c-4355-9997-cde832700d7f-istio-data\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:26:56.774022 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:56.773831 5057 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a1427f2-b35c-4355-9997-cde832700d7f-istiod-ca-cert\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:26:56.774022 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:56.773843 5057 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a1427f2-b35c-4355-9997-cde832700d7f-workload-certs\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:26:56.774022 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:56.773971 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a1427f2-b35c-4355-9997-cde832700d7f-workload-socket" pod "1a1427f2-b35c-4355-9997-cde832700d7f" (UID: "1a1427f2-b35c-4355-9997-cde832700d7f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:26:56.775486 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:56.775449 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a1427f2-b35c-4355-9997-cde832700d7f-kube-api-access-9nz8t" pod "1a1427f2-b35c-4355-9997-cde832700d7f" (UID: "1a1427f2-b35c-4355-9997-cde832700d7f"). InnerVolumeSpecName "kube-api-access-9nz8t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:26:56.775582 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:56.775506 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a1427f2-b35c-4355-9997-cde832700d7f-istio-envoy" pod "1a1427f2-b35c-4355-9997-cde832700d7f" (UID: "1a1427f2-b35c-4355-9997-cde832700d7f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:26:56.775582 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:56.775523 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1a1427f2-b35c-4355-9997-cde832700d7f-istio-podinfo" pod "1a1427f2-b35c-4355-9997-cde832700d7f" (UID: "1a1427f2-b35c-4355-9997-cde832700d7f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:26:56.775722 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:56.775585 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a1427f2-b35c-4355-9997-cde832700d7f-istio-token" pod "1a1427f2-b35c-4355-9997-cde832700d7f" (UID: "1a1427f2-b35c-4355-9997-cde832700d7f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:26:56.799307 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:56.799105 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-024aa6-5599897986-xltss" podUID="03c6c1ca-e5e4-41c5-961c-50880f649b13" containerName="payload-processing" containerID="cri-o://52484e13b9750f79456b156ed836d5869497879cee846f908fc2a8e542def12e" gracePeriod=25 Oct 29 23:26:56.838907 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:56.838855 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-024aa6-b99d57cd7-pslx4" podUID="7a5e765c-b65b-4411-92a5-fad9521c9291" containerName="payload-pre-processing" containerID="cri-o://a40acdcd1dd204a8e3273392b221ff5ebe518c72f4fb3d8b53d68302c894c7b2" gracePeriod=25 Oct 29 23:26:56.875189 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:56.875163 5057 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a1427f2-b35c-4355-9997-cde832700d7f-workload-socket\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:26:56.875189 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:56.875189 5057 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a1427f2-b35c-4355-9997-cde832700d7f-istio-envoy\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:26:56.875313 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:56.875198 5057 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a1427f2-b35c-4355-9997-cde832700d7f-istio-podinfo\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:26:56.875313 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:56.875206 5057 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a1427f2-b35c-4355-9997-cde832700d7f-istio-token\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:26:56.875313 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:56.875215 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9nz8t\" (UniqueName: \"kubernetes.io/projected/1a1427f2-b35c-4355-9997-cde832700d7f-kube-api-access-9nz8t\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:26:57.092892 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:57.092864 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-024aa6-5599897986-xltss" Oct 29 23:26:57.179559 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:57.179460 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/03c6c1ca-e5e4-41c5-961c-50880f649b13-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03c6c1ca-e5e4-41c5-961c-50880f649b13-plugins-config-volume\") pod \"03c6c1ca-e5e4-41c5-961c-50880f649b13\" (UID: \"03c6c1ca-e5e4-41c5-961c-50880f649b13\") " Oct 29 23:26:57.179752 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:57.179584 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03c6c1ca-e5e4-41c5-961c-50880f649b13-kube-api-access-479nm\" (UniqueName: \"kubernetes.io/projected/03c6c1ca-e5e4-41c5-961c-50880f649b13-kube-api-access-479nm\") pod \"03c6c1ca-e5e4-41c5-961c-50880f649b13\" (UID: \"03c6c1ca-e5e4-41c5-961c-50880f649b13\") " Oct 29 23:26:57.180053 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:57.180025 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03c6c1ca-e5e4-41c5-961c-50880f649b13-plugins-config-volume" pod "03c6c1ca-e5e4-41c5-961c-50880f649b13" (UID: "03c6c1ca-e5e4-41c5-961c-50880f649b13"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:26:57.180507 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:57.180481 5057 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03c6c1ca-e5e4-41c5-961c-50880f649b13-plugins-config-volume\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:26:57.182740 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:57.182704 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03c6c1ca-e5e4-41c5-961c-50880f649b13-kube-api-access-479nm" pod "03c6c1ca-e5e4-41c5-961c-50880f649b13" (UID: "03c6c1ca-e5e4-41c5-961c-50880f649b13"). InnerVolumeSpecName "kube-api-access-479nm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:26:57.263950 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:57.263914 5057 generic.go:356] "Generic (PLEG): container finished" podID="1a1427f2-b35c-4355-9997-cde832700d7f" containerID="21b23ab4ce637a39eb4a94f9a00935ce39aaeca6fc3c2da9178ee83903e3ebd2" exitCode=0 Oct 29 23:26:57.264233 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:57.263975 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-1a8107-openshift-default-7fff599fbf-rtjrx" Oct 29 23:26:57.264233 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:57.264002 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-1a8107-openshift-default-7fff599fbf-rtjrx" event={"ID":"1a1427f2-b35c-4355-9997-cde832700d7f","Type":"ContainerDied","Data":"21b23ab4ce637a39eb4a94f9a00935ce39aaeca6fc3c2da9178ee83903e3ebd2"} Oct 29 23:26:57.264233 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:57.264085 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-1a8107-openshift-default-7fff599fbf-rtjrx" event={"ID":"1a1427f2-b35c-4355-9997-cde832700d7f","Type":"ContainerDied","Data":"af491a46d2a7721a8a51903b8fc682e4358302dcbc22e1c8948b63d7d2c6d7c0"} Oct 29 23:26:57.264233 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:57.264116 5057 scope.go:122] "RemoveContainer" containerID="21b23ab4ce637a39eb4a94f9a00935ce39aaeca6fc3c2da9178ee83903e3ebd2" Oct 29 23:26:57.266119 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:57.266096 5057 generic.go:356] "Generic (PLEG): container finished" podID="03c6c1ca-e5e4-41c5-961c-50880f649b13" containerID="52484e13b9750f79456b156ed836d5869497879cee846f908fc2a8e542def12e" exitCode=0 Oct 29 23:26:57.266245 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:57.266149 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-024aa6-5599897986-xltss" Oct 29 23:26:57.266245 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:57.266174 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-024aa6-5599897986-xltss" event={"ID":"03c6c1ca-e5e4-41c5-961c-50880f649b13","Type":"ContainerDied","Data":"52484e13b9750f79456b156ed836d5869497879cee846f908fc2a8e542def12e"} Oct 29 23:26:57.266245 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:57.266222 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-024aa6-5599897986-xltss" event={"ID":"03c6c1ca-e5e4-41c5-961c-50880f649b13","Type":"ContainerDied","Data":"af1c093b2d109d7383d29e55808cedf1464b9b6fb34855b8c026562809db99bd"} Oct 29 23:26:57.279235 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:57.279217 5057 scope.go:122] "RemoveContainer" containerID="21b23ab4ce637a39eb4a94f9a00935ce39aaeca6fc3c2da9178ee83903e3ebd2" Oct 29 23:26:57.279552 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:26:57.279533 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21b23ab4ce637a39eb4a94f9a00935ce39aaeca6fc3c2da9178ee83903e3ebd2\": container with ID starting with 21b23ab4ce637a39eb4a94f9a00935ce39aaeca6fc3c2da9178ee83903e3ebd2 not found: ID does not exist" containerID="21b23ab4ce637a39eb4a94f9a00935ce39aaeca6fc3c2da9178ee83903e3ebd2" Oct 29 23:26:57.279646 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:57.279559 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21b23ab4ce637a39eb4a94f9a00935ce39aaeca6fc3c2da9178ee83903e3ebd2"} err="failed to get container status \"21b23ab4ce637a39eb4a94f9a00935ce39aaeca6fc3c2da9178ee83903e3ebd2\": rpc error: code = NotFound desc = could not find container \"21b23ab4ce637a39eb4a94f9a00935ce39aaeca6fc3c2da9178ee83903e3ebd2\": container with ID starting with 21b23ab4ce637a39eb4a94f9a00935ce39aaeca6fc3c2da9178ee83903e3ebd2 not found: ID does not exist" Oct 29 23:26:57.279646 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:57.279575 5057 scope.go:122] "RemoveContainer" containerID="52484e13b9750f79456b156ed836d5869497879cee846f908fc2a8e542def12e" Oct 29 23:26:57.282365 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:57.282309 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-479nm\" (UniqueName: \"kubernetes.io/projected/03c6c1ca-e5e4-41c5-961c-50880f649b13-kube-api-access-479nm\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:26:57.282711 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:57.282684 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:26:57.292472 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:57.292449 5057 scope.go:122] "RemoveContainer" containerID="52484e13b9750f79456b156ed836d5869497879cee846f908fc2a8e542def12e" Oct 29 23:26:57.292764 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:26:57.292739 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52484e13b9750f79456b156ed836d5869497879cee846f908fc2a8e542def12e\": container with ID starting with 52484e13b9750f79456b156ed836d5869497879cee846f908fc2a8e542def12e not found: ID does not exist" containerID="52484e13b9750f79456b156ed836d5869497879cee846f908fc2a8e542def12e" Oct 29 23:26:57.292849 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:57.292772 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52484e13b9750f79456b156ed836d5869497879cee846f908fc2a8e542def12e"} err="failed to get container status \"52484e13b9750f79456b156ed836d5869497879cee846f908fc2a8e542def12e\": rpc error: code = NotFound desc = could not find container \"52484e13b9750f79456b156ed836d5869497879cee846f908fc2a8e542def12e\": container with ID starting with 52484e13b9750f79456b156ed836d5869497879cee846f908fc2a8e542def12e not found: ID does not exist" Oct 29 23:26:57.331778 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:57.331748 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-024aa6-5599897986-xltss"] Oct 29 23:26:57.331778 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:57.331783 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-024aa6-5599897986-xltss"] Oct 29 23:26:57.358461 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:57.358414 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-1a8107-openshift-default-7fff599fbf-rtjrx"] Oct 29 23:26:57.365613 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:57.365570 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-1a8107-openshift-default-7fff599fbf-rtjrx"] Oct 29 23:26:57.415850 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:57.415817 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:26:58.034666 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:58.034629 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:26:58.148205 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:58.148165 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-024aa6-b99d57cd7-pslx4" podUID="7a5e765c-b65b-4411-92a5-fad9521c9291" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:26:58.701832 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:58.701786 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-dcacb9-767c887c7b-jrk26" podUID="26ab3c63-6e66-4981-b86c-bdf7055db34f" containerName="payload-processing" containerID="cri-o://09baca9166765ceb8dd71e72fb60b7cdeabed1278bb49142c70a5587b32ab390" gracePeriod=25 Oct 29 23:26:58.725380 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:58.724475 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-dcacb9-86d7d89559-xk4m6" podUID="38161411-6a7a-4a65-aacf-e39baf965f8e" containerName="payload-pre-processing" containerID="cri-o://7af091fe73f5cc4c7a442becd64d2f40546de358686d9b023fcd45d2cef5ec3a" gracePeriod=25 Oct 29 23:26:59.015806 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:59.015696 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:26:59.127186 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:59.127142 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:26:59.238720 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:59.238684 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-3e5b21-7whs8"] Oct 29 23:26:59.239726 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:59.239705 5057 cpu_manager.go:414] "containerMap: removing container" podUID="ec6f58bd-af5d-40f6-a333-d355d71fe880" containerName="revoke-keys" Oct 29 23:26:59.239726 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:59.239724 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec6f58bd-af5d-40f6-a333-d355d71fe880" containerName="revoke-keys" Oct 29 23:26:59.239870 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:59.239739 5057 cpu_manager.go:414] "containerMap: removing container" podUID="159d07ab-681d-4431-9d7d-e08bcee96624" containerName="payload-processing" Oct 29 23:26:59.239870 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:59.239744 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="159d07ab-681d-4431-9d7d-e08bcee96624" containerName="payload-processing" Oct 29 23:26:59.239870 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:59.239757 5057 cpu_manager.go:414] "containerMap: removing container" podUID="eb4eaa12-cde8-43d8-9af0-9534a90f9c6b" containerName="payload-pre-processing" Oct 29 23:26:59.239870 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:59.239765 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb4eaa12-cde8-43d8-9af0-9534a90f9c6b" containerName="payload-pre-processing" Oct 29 23:26:59.239870 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:59.239779 5057 cpu_manager.go:414] "containerMap: removing container" podUID="1a1427f2-b35c-4355-9997-cde832700d7f" containerName="istio-proxy" Oct 29 23:26:59.239870 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:59.239787 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a1427f2-b35c-4355-9997-cde832700d7f" containerName="istio-proxy" Oct 29 23:26:59.239870 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:59.239805 5057 cpu_manager.go:414] "containerMap: removing container" podUID="03c6c1ca-e5e4-41c5-961c-50880f649b13" containerName="payload-processing" Oct 29 23:26:59.239870 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:59.239810 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03c6c1ca-e5e4-41c5-961c-50880f649b13" containerName="payload-processing" Oct 29 23:26:59.240153 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:59.240009 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec6f58bd-af5d-40f6-a333-d355d71fe880" containerName="revoke-keys" Oct 29 23:26:59.240153 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:59.240022 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="159d07ab-681d-4431-9d7d-e08bcee96624" containerName="payload-processing" Oct 29 23:26:59.240153 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:59.240030 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb4eaa12-cde8-43d8-9af0-9534a90f9c6b" containerName="payload-pre-processing" Oct 29 23:26:59.240153 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:59.240037 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a1427f2-b35c-4355-9997-cde832700d7f" containerName="istio-proxy" Oct 29 23:26:59.240153 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:59.240044 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="03c6c1ca-e5e4-41c5-961c-50880f649b13" containerName="payload-processing" Oct 29 23:26:59.245860 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:59.245836 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-3e5b21-7whs8" Oct 29 23:26:59.262991 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:59.262960 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-3e5b21-7whs8"] Oct 29 23:26:59.268176 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:59.268116 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:26:59.268176 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:59.268130 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dmx57\"" Oct 29 23:26:59.315500 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:59.315464 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b56f7c05-6607-4652-a92d-a55b5b21fcf0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-3e5b21-7whs8\" (UID: \"b56f7c05-6607-4652-a92d-a55b5b21fcf0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-3e5b21-7whs8" Oct 29 23:26:59.315697 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:59.315510 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b56f7c05-6607-4652-a92d-a55b5b21fcf0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-3e5b21-7whs8\" (UID: \"b56f7c05-6607-4652-a92d-a55b5b21fcf0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-3e5b21-7whs8" Oct 29 23:26:59.315766 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:59.315719 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9zpm\" (UniqueName: \"kubernetes.io/projected/b56f7c05-6607-4652-a92d-a55b5b21fcf0-kube-api-access-g9zpm\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-3e5b21-7whs8\" (UID: \"b56f7c05-6607-4652-a92d-a55b5b21fcf0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-3e5b21-7whs8" Oct 29 23:26:59.332905 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:59.332880 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03c6c1ca-e5e4-41c5-961c-50880f649b13" path="/var/lib/kubelet/pods/03c6c1ca-e5e4-41c5-961c-50880f649b13/volumes" Oct 29 23:26:59.333388 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:59.333371 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a1427f2-b35c-4355-9997-cde832700d7f" path="/var/lib/kubelet/pods/1a1427f2-b35c-4355-9997-cde832700d7f/volumes" Oct 29 23:26:59.417618 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:59.417566 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b56f7c05-6607-4652-a92d-a55b5b21fcf0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-3e5b21-7whs8\" (UID: \"b56f7c05-6607-4652-a92d-a55b5b21fcf0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-3e5b21-7whs8" Oct 29 23:26:59.417618 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:59.417617 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b56f7c05-6607-4652-a92d-a55b5b21fcf0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-3e5b21-7whs8\" (UID: \"b56f7c05-6607-4652-a92d-a55b5b21fcf0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-3e5b21-7whs8" Oct 29 23:26:59.418112 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:59.417865 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9zpm\" (UniqueName: \"kubernetes.io/projected/b56f7c05-6607-4652-a92d-a55b5b21fcf0-kube-api-access-g9zpm\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-3e5b21-7whs8\" (UID: \"b56f7c05-6607-4652-a92d-a55b5b21fcf0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-3e5b21-7whs8" Oct 29 23:26:59.418719 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:59.418677 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b56f7c05-6607-4652-a92d-a55b5b21fcf0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-3e5b21-7whs8\" (UID: \"b56f7c05-6607-4652-a92d-a55b5b21fcf0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-3e5b21-7whs8" Oct 29 23:26:59.421055 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:59.421024 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b56f7c05-6607-4652-a92d-a55b5b21fcf0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-3e5b21-7whs8\" (UID: \"b56f7c05-6607-4652-a92d-a55b5b21fcf0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-3e5b21-7whs8" Oct 29 23:26:59.440976 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:59.440955 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9zpm\" (UniqueName: \"kubernetes.io/projected/b56f7c05-6607-4652-a92d-a55b5b21fcf0-kube-api-access-g9zpm\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-3e5b21-7whs8\" (UID: \"b56f7c05-6607-4652-a92d-a55b5b21fcf0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-3e5b21-7whs8" Oct 29 23:26:59.442918 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:59.442890 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:26:59.553412 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:59.553303 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:26:59.563033 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:59.563008 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-3e5b21-7whs8" Oct 29 23:26:59.753104 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:59.753069 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:26:59.762263 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:59.762234 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-32717a-jsm8v"] Oct 29 23:26:59.770468 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:59.770448 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-32717a-jsm8v" Oct 29 23:26:59.781652 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:59.781625 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-32717a-jsm8v"] Oct 29 23:26:59.824835 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:59.824741 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9ed3669-6751-4a9c-8c44-ba9a7df4146a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-32717a-jsm8v\" (UID: \"d9ed3669-6751-4a9c-8c44-ba9a7df4146a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-32717a-jsm8v" Oct 29 23:26:59.825059 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:59.824972 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8m9qz\" (UniqueName: \"kubernetes.io/projected/d9ed3669-6751-4a9c-8c44-ba9a7df4146a-kube-api-access-8m9qz\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-32717a-jsm8v\" (UID: \"d9ed3669-6751-4a9c-8c44-ba9a7df4146a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-32717a-jsm8v" Oct 29 23:26:59.825059 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:59.825037 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9ed3669-6751-4a9c-8c44-ba9a7df4146a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-32717a-jsm8v\" (UID: \"d9ed3669-6751-4a9c-8c44-ba9a7df4146a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-32717a-jsm8v" Oct 29 23:26:59.869551 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:59.869510 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:26:59.892503 ip-10-0-140-74 kubenswrapper[5057]: W1029 23:26:59.892470 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb56f7c05_6607_4652_a92d_a55b5b21fcf0.slice/crio-b37bb23ec07bc3fdd9350d72cca72e99885c1fe1860a71d4c97dc525b380f9cf WatchSource:0}: Error finding container b37bb23ec07bc3fdd9350d72cca72e99885c1fe1860a71d4c97dc525b380f9cf: Status 404 returned error can't find the container with id b37bb23ec07bc3fdd9350d72cca72e99885c1fe1860a71d4c97dc525b380f9cf Oct 29 23:26:59.895794 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:59.895768 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-3e5b21-7whs8"] Oct 29 23:26:59.927017 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:59.926987 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9ed3669-6751-4a9c-8c44-ba9a7df4146a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-32717a-jsm8v\" (UID: \"d9ed3669-6751-4a9c-8c44-ba9a7df4146a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-32717a-jsm8v" Oct 29 23:26:59.927151 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:59.927115 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8m9qz\" (UniqueName: \"kubernetes.io/projected/d9ed3669-6751-4a9c-8c44-ba9a7df4146a-kube-api-access-8m9qz\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-32717a-jsm8v\" (UID: \"d9ed3669-6751-4a9c-8c44-ba9a7df4146a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-32717a-jsm8v" Oct 29 23:26:59.927224 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:59.927175 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9ed3669-6751-4a9c-8c44-ba9a7df4146a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-32717a-jsm8v\" (UID: \"d9ed3669-6751-4a9c-8c44-ba9a7df4146a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-32717a-jsm8v" Oct 29 23:26:59.927923 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:59.927899 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9ed3669-6751-4a9c-8c44-ba9a7df4146a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-32717a-jsm8v\" (UID: \"d9ed3669-6751-4a9c-8c44-ba9a7df4146a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-32717a-jsm8v" Oct 29 23:26:59.929347 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:59.929312 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9ed3669-6751-4a9c-8c44-ba9a7df4146a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-32717a-jsm8v\" (UID: \"d9ed3669-6751-4a9c-8c44-ba9a7df4146a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-32717a-jsm8v" Oct 29 23:26:59.950304 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:26:59.950280 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8m9qz\" (UniqueName: \"kubernetes.io/projected/d9ed3669-6751-4a9c-8c44-ba9a7df4146a-kube-api-access-8m9qz\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-32717a-jsm8v\" (UID: \"d9ed3669-6751-4a9c-8c44-ba9a7df4146a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-32717a-jsm8v" Oct 29 23:27:00.096037 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:00.095957 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-32717a-jsm8v" Oct 29 23:27:00.310693 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:00.310657 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-3e5b21-7whs8" event={"ID":"b56f7c05-6607-4652-a92d-a55b5b21fcf0","Type":"ContainerStarted","Data":"9d7d1d93c53e6b0dd2e85cbfbb91e3fe81dc4305337936c9f8a3cc8e554d1f9a"} Oct 29 23:27:00.310831 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:00.310723 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-3e5b21-7whs8" event={"ID":"b56f7c05-6607-4652-a92d-a55b5b21fcf0","Type":"ContainerStarted","Data":"b37bb23ec07bc3fdd9350d72cca72e99885c1fe1860a71d4c97dc525b380f9cf"} Oct 29 23:27:00.353164 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:00.353054 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-3e5b21-7whs8" podStartSLOduration=1.353035492 podStartE2EDuration="1.353035492s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:26:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:27:00.350211537 +0000 UTC m=+3201.133424962" watchObservedRunningTime="2026-10-29 23:27:00.353035492 +0000 UTC m=+3201.136248919" Oct 29 23:27:00.407002 ip-10-0-140-74 kubenswrapper[5057]: W1029 23:27:00.406936 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9ed3669_6751_4a9c_8c44_ba9a7df4146a.slice/crio-2b3a4b8417ddf3481f3cf1e0fb3fc6def26560e8be54d27556cfbfabce620d9d WatchSource:0}: Error finding container 2b3a4b8417ddf3481f3cf1e0fb3fc6def26560e8be54d27556cfbfabce620d9d: Status 404 returned error can't find the container with id 2b3a4b8417ddf3481f3cf1e0fb3fc6def26560e8be54d27556cfbfabce620d9d Oct 29 23:27:00.409227 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:00.409192 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-32717a-jsm8v"] Oct 29 23:27:01.323819 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:01.323726 5057 generic.go:356] "Generic (PLEG): container finished" podID="b56f7c05-6607-4652-a92d-a55b5b21fcf0" containerID="9d7d1d93c53e6b0dd2e85cbfbb91e3fe81dc4305337936c9f8a3cc8e554d1f9a" exitCode=0 Oct 29 23:27:01.333603 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:01.333570 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-3e5b21-7whs8" event={"ID":"b56f7c05-6607-4652-a92d-a55b5b21fcf0","Type":"ContainerDied","Data":"9d7d1d93c53e6b0dd2e85cbfbb91e3fe81dc4305337936c9f8a3cc8e554d1f9a"} Oct 29 23:27:01.333728 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:01.333630 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-32717a-jsm8v" event={"ID":"d9ed3669-6751-4a9c-8c44-ba9a7df4146a","Type":"ContainerStarted","Data":"0df22865870a0255fc37a1e98900b2ebe6a64cf652d48a94c7b571d21032fb99"} Oct 29 23:27:01.333728 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:01.333660 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-32717a-jsm8v" event={"ID":"d9ed3669-6751-4a9c-8c44-ba9a7df4146a","Type":"ContainerStarted","Data":"2b3a4b8417ddf3481f3cf1e0fb3fc6def26560e8be54d27556cfbfabce620d9d"} Oct 29 23:27:01.407504 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:01.407445 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-32717a-jsm8v" podStartSLOduration=2.407424518 podStartE2EDuration="2.407424518s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:26:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:27:01.404139027 +0000 UTC m=+3202.187352454" watchObservedRunningTime="2026-10-29 23:27:01.407424518 +0000 UTC m=+3202.190637945" Oct 29 23:27:01.725441 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:01.725386 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:27:02.340178 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:02.340143 5057 generic.go:356] "Generic (PLEG): container finished" podID="d9ed3669-6751-4a9c-8c44-ba9a7df4146a" containerID="0df22865870a0255fc37a1e98900b2ebe6a64cf652d48a94c7b571d21032fb99" exitCode=0 Oct 29 23:27:02.340662 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:02.340219 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-32717a-jsm8v" event={"ID":"d9ed3669-6751-4a9c-8c44-ba9a7df4146a","Type":"ContainerDied","Data":"0df22865870a0255fc37a1e98900b2ebe6a64cf652d48a94c7b571d21032fb99"} Oct 29 23:27:02.547924 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:02.547897 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-3e5b21-7whs8" Oct 29 23:27:02.680683 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:02.680607 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b56f7c05-6607-4652-a92d-a55b5b21fcf0-kube-api-access-g9zpm\" (UniqueName: \"kubernetes.io/projected/b56f7c05-6607-4652-a92d-a55b5b21fcf0-kube-api-access-g9zpm\") pod \"b56f7c05-6607-4652-a92d-a55b5b21fcf0\" (UID: \"b56f7c05-6607-4652-a92d-a55b5b21fcf0\") " Oct 29 23:27:02.680845 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:02.680830 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b56f7c05-6607-4652-a92d-a55b5b21fcf0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b56f7c05-6607-4652-a92d-a55b5b21fcf0-maas-api-xks-ca\") pod \"b56f7c05-6607-4652-a92d-a55b5b21fcf0\" (UID: \"b56f7c05-6607-4652-a92d-a55b5b21fcf0\") " Oct 29 23:27:02.680938 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:02.680925 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b56f7c05-6607-4652-a92d-a55b5b21fcf0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b56f7c05-6607-4652-a92d-a55b5b21fcf0-maas-api-openshift-ca\") pod \"b56f7c05-6607-4652-a92d-a55b5b21fcf0\" (UID: \"b56f7c05-6607-4652-a92d-a55b5b21fcf0\") " Oct 29 23:27:02.681365 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:02.681304 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b56f7c05-6607-4652-a92d-a55b5b21fcf0-maas-api-openshift-ca" pod "b56f7c05-6607-4652-a92d-a55b5b21fcf0" (UID: "b56f7c05-6607-4652-a92d-a55b5b21fcf0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:27:02.681507 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:02.681489 5057 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b56f7c05-6607-4652-a92d-a55b5b21fcf0-maas-api-openshift-ca\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:27:02.682844 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:02.682820 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b56f7c05-6607-4652-a92d-a55b5b21fcf0-kube-api-access-g9zpm" pod "b56f7c05-6607-4652-a92d-a55b5b21fcf0" (UID: "b56f7c05-6607-4652-a92d-a55b5b21fcf0"). InnerVolumeSpecName "kube-api-access-g9zpm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:27:02.682844 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:02.682830 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b56f7c05-6607-4652-a92d-a55b5b21fcf0-maas-api-xks-ca" pod "b56f7c05-6607-4652-a92d-a55b5b21fcf0" (UID: "b56f7c05-6607-4652-a92d-a55b5b21fcf0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:27:02.782694 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:02.782653 5057 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b56f7c05-6607-4652-a92d-a55b5b21fcf0-maas-api-xks-ca\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:27:02.782694 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:02.782687 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g9zpm\" (UniqueName: \"kubernetes.io/projected/b56f7c05-6607-4652-a92d-a55b5b21fcf0-kube-api-access-g9zpm\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:27:02.923828 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:02.923768 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-dcacb9-86d7d89559-xk4m6" podUID="38161411-6a7a-4a65-aacf-e39baf965f8e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:27:02.923828 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:02.923799 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-dcacb9-767c887c7b-jrk26" podUID="26ab3c63-6e66-4981-b86c-bdf7055db34f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:27:03.350994 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:03.350968 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-3e5b21-7whs8" Oct 29 23:27:03.351448 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:03.351003 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-3e5b21-7whs8" event={"ID":"b56f7c05-6607-4652-a92d-a55b5b21fcf0","Type":"ContainerDied","Data":"b37bb23ec07bc3fdd9350d72cca72e99885c1fe1860a71d4c97dc525b380f9cf"} Oct 29 23:27:03.351448 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:03.351036 5057 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b37bb23ec07bc3fdd9350d72cca72e99885c1fe1860a71d4c97dc525b380f9cf" Oct 29 23:27:03.398659 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:03.398625 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:27:03.554821 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:03.554793 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-32717a-jsm8v" Oct 29 23:27:03.706267 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:03.706215 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9ed3669-6751-4a9c-8c44-ba9a7df4146a-kube-api-access-8m9qz\" (UniqueName: \"kubernetes.io/projected/d9ed3669-6751-4a9c-8c44-ba9a7df4146a-kube-api-access-8m9qz\") pod \"d9ed3669-6751-4a9c-8c44-ba9a7df4146a\" (UID: \"d9ed3669-6751-4a9c-8c44-ba9a7df4146a\") " Oct 29 23:27:03.706743 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:03.706623 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d9ed3669-6751-4a9c-8c44-ba9a7df4146a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9ed3669-6751-4a9c-8c44-ba9a7df4146a-maas-api-xks-ca\") pod \"d9ed3669-6751-4a9c-8c44-ba9a7df4146a\" (UID: \"d9ed3669-6751-4a9c-8c44-ba9a7df4146a\") " Oct 29 23:27:03.707205 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:03.707174 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d9ed3669-6751-4a9c-8c44-ba9a7df4146a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9ed3669-6751-4a9c-8c44-ba9a7df4146a-maas-api-openshift-ca\") pod \"d9ed3669-6751-4a9c-8c44-ba9a7df4146a\" (UID: \"d9ed3669-6751-4a9c-8c44-ba9a7df4146a\") " Oct 29 23:27:03.708720 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:03.708698 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9ed3669-6751-4a9c-8c44-ba9a7df4146a-maas-api-openshift-ca" pod "d9ed3669-6751-4a9c-8c44-ba9a7df4146a" (UID: "d9ed3669-6751-4a9c-8c44-ba9a7df4146a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:27:03.718759 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:03.718718 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9ed3669-6751-4a9c-8c44-ba9a7df4146a-kube-api-access-8m9qz" pod "d9ed3669-6751-4a9c-8c44-ba9a7df4146a" (UID: "d9ed3669-6751-4a9c-8c44-ba9a7df4146a"). InnerVolumeSpecName "kube-api-access-8m9qz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:27:03.741617 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:03.741566 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d9ed3669-6751-4a9c-8c44-ba9a7df4146a-maas-api-xks-ca" pod "d9ed3669-6751-4a9c-8c44-ba9a7df4146a" (UID: "d9ed3669-6751-4a9c-8c44-ba9a7df4146a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:27:03.809928 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:03.809888 5057 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9ed3669-6751-4a9c-8c44-ba9a7df4146a-maas-api-openshift-ca\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:27:03.809928 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:03.809930 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8m9qz\" (UniqueName: \"kubernetes.io/projected/d9ed3669-6751-4a9c-8c44-ba9a7df4146a-kube-api-access-8m9qz\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:27:03.810189 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:03.809947 5057 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9ed3669-6751-4a9c-8c44-ba9a7df4146a-maas-api-xks-ca\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:27:04.426350 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:04.425351 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-32717a-jsm8v" event={"ID":"d9ed3669-6751-4a9c-8c44-ba9a7df4146a","Type":"ContainerDied","Data":"2b3a4b8417ddf3481f3cf1e0fb3fc6def26560e8be54d27556cfbfabce620d9d"} Oct 29 23:27:04.426350 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:04.425417 5057 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2b3a4b8417ddf3481f3cf1e0fb3fc6def26560e8be54d27556cfbfabce620d9d" Oct 29 23:27:04.426350 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:04.425488 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-32717a-jsm8v" Oct 29 23:27:05.404627 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:05.404581 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:27:08.148515 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:08.148470 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-024aa6-b99d57cd7-pslx4" podUID="7a5e765c-b65b-4411-92a5-fad9521c9291" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:27:08.169241 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:08.169202 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:27:09.657238 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:09.657207 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-3e5b21-f69958676-g78g2"] Oct 29 23:27:09.690718 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:09.690679 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-3e5b21-d7b5c69c5-ds82z"] Oct 29 23:27:09.818347 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:09.812867 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:27:09.924137 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:09.924041 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:27:11.307352 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:11.307238 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:27:11.416833 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:11.416791 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:27:11.633158 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:11.633061 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:27:12.923606 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:12.923562 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-dcacb9-767c887c7b-jrk26" podUID="26ab3c63-6e66-4981-b86c-bdf7055db34f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:27:12.923606 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:12.923594 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-dcacb9-86d7d89559-xk4m6" podUID="38161411-6a7a-4a65-aacf-e39baf965f8e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:27:13.395623 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:13.395583 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:27:14.657516 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:14.657450 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-3e5b21-f69958676-g78g2" podUID="0157b7d8-b442-4302-9fc2-4e87e55ad7aa" containerName="payload-processing" containerID="cri-o://8aaf466ab895e32406715310466b10620eeaa03863836a83b5f1fc50ad9c378b" gracePeriod=25 Oct 29 23:27:14.691460 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:14.691429 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-3e5b21-d7b5c69c5-ds82z" podUID="e05f5f87-b9fc-4dfa-b61a-ce1f776e146c" containerName="payload-pre-processing" containerID="cri-o://91078cc6e3795c4ac92f4e63a8fc9158442f9ae23309168a63e15a221a4d9edd" gracePeriod=25 Oct 29 23:27:15.007059 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:15.006624 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-3e5b21-f69958676-g78g2" Oct 29 23:27:15.057007 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:15.056982 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-3e5b21-d7b5c69c5-ds82z" Oct 29 23:27:15.065735 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:15.065687 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:27:15.106140 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:15.104819 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0157b7d8-b442-4302-9fc2-4e87e55ad7aa-kube-api-access-xdtwn\" (UniqueName: \"kubernetes.io/projected/0157b7d8-b442-4302-9fc2-4e87e55ad7aa-kube-api-access-xdtwn\") pod \"0157b7d8-b442-4302-9fc2-4e87e55ad7aa\" (UID: \"0157b7d8-b442-4302-9fc2-4e87e55ad7aa\") " Oct 29 23:27:15.106140 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:15.105150 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0157b7d8-b442-4302-9fc2-4e87e55ad7aa-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0157b7d8-b442-4302-9fc2-4e87e55ad7aa-plugins-config-volume\") pod \"0157b7d8-b442-4302-9fc2-4e87e55ad7aa\" (UID: \"0157b7d8-b442-4302-9fc2-4e87e55ad7aa\") " Oct 29 23:27:15.106911 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:15.106593 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0157b7d8-b442-4302-9fc2-4e87e55ad7aa-plugins-config-volume" pod "0157b7d8-b442-4302-9fc2-4e87e55ad7aa" (UID: "0157b7d8-b442-4302-9fc2-4e87e55ad7aa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:27:15.109372 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:15.109296 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0157b7d8-b442-4302-9fc2-4e87e55ad7aa-kube-api-access-xdtwn" pod "0157b7d8-b442-4302-9fc2-4e87e55ad7aa" (UID: "0157b7d8-b442-4302-9fc2-4e87e55ad7aa"). InnerVolumeSpecName "kube-api-access-xdtwn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:27:15.207325 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:15.207248 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e05f5f87-b9fc-4dfa-b61a-ce1f776e146c-kube-api-access-d2c7l\" (UniqueName: \"kubernetes.io/projected/e05f5f87-b9fc-4dfa-b61a-ce1f776e146c-kube-api-access-d2c7l\") pod \"e05f5f87-b9fc-4dfa-b61a-ce1f776e146c\" (UID: \"e05f5f87-b9fc-4dfa-b61a-ce1f776e146c\") " Oct 29 23:27:15.207544 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:15.207483 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e05f5f87-b9fc-4dfa-b61a-ce1f776e146c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e05f5f87-b9fc-4dfa-b61a-ce1f776e146c-plugins-config-volume\") pod \"e05f5f87-b9fc-4dfa-b61a-ce1f776e146c\" (UID: \"e05f5f87-b9fc-4dfa-b61a-ce1f776e146c\") " Oct 29 23:27:15.207873 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:15.207844 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e05f5f87-b9fc-4dfa-b61a-ce1f776e146c-plugins-config-volume" pod "e05f5f87-b9fc-4dfa-b61a-ce1f776e146c" (UID: "e05f5f87-b9fc-4dfa-b61a-ce1f776e146c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:27:15.208150 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:15.208132 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xdtwn\" (UniqueName: \"kubernetes.io/projected/0157b7d8-b442-4302-9fc2-4e87e55ad7aa-kube-api-access-xdtwn\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:27:15.208199 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:15.208154 5057 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e05f5f87-b9fc-4dfa-b61a-ce1f776e146c-plugins-config-volume\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:27:15.208199 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:15.208165 5057 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0157b7d8-b442-4302-9fc2-4e87e55ad7aa-plugins-config-volume\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:27:15.210006 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:15.209975 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e05f5f87-b9fc-4dfa-b61a-ce1f776e146c-kube-api-access-d2c7l" pod "e05f5f87-b9fc-4dfa-b61a-ce1f776e146c" (UID: "e05f5f87-b9fc-4dfa-b61a-ce1f776e146c"). InnerVolumeSpecName "kube-api-access-d2c7l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:27:15.310204 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:15.310173 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d2c7l\" (UniqueName: \"kubernetes.io/projected/e05f5f87-b9fc-4dfa-b61a-ce1f776e146c-kube-api-access-d2c7l\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:27:15.566956 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:15.566866 5057 generic.go:356] "Generic (PLEG): container finished" podID="e05f5f87-b9fc-4dfa-b61a-ce1f776e146c" containerID="91078cc6e3795c4ac92f4e63a8fc9158442f9ae23309168a63e15a221a4d9edd" exitCode=0 Oct 29 23:27:15.566956 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:15.566916 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-3e5b21-d7b5c69c5-ds82z" Oct 29 23:27:15.567182 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:15.566950 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-3e5b21-d7b5c69c5-ds82z" event={"ID":"e05f5f87-b9fc-4dfa-b61a-ce1f776e146c","Type":"ContainerDied","Data":"91078cc6e3795c4ac92f4e63a8fc9158442f9ae23309168a63e15a221a4d9edd"} Oct 29 23:27:15.567182 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:15.567006 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-3e5b21-d7b5c69c5-ds82z" event={"ID":"e05f5f87-b9fc-4dfa-b61a-ce1f776e146c","Type":"ContainerDied","Data":"d1f91f973ed09cc5b0829fb1b29a7e8f36ecbb581d417c392f32529f2ec7bb54"} Oct 29 23:27:15.567182 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:15.567034 5057 scope.go:122] "RemoveContainer" containerID="91078cc6e3795c4ac92f4e63a8fc9158442f9ae23309168a63e15a221a4d9edd" Oct 29 23:27:15.569065 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:15.569040 5057 generic.go:356] "Generic (PLEG): container finished" podID="0157b7d8-b442-4302-9fc2-4e87e55ad7aa" containerID="8aaf466ab895e32406715310466b10620eeaa03863836a83b5f1fc50ad9c378b" exitCode=0 Oct 29 23:27:15.569240 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:15.569123 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-3e5b21-f69958676-g78g2" event={"ID":"0157b7d8-b442-4302-9fc2-4e87e55ad7aa","Type":"ContainerDied","Data":"8aaf466ab895e32406715310466b10620eeaa03863836a83b5f1fc50ad9c378b"} Oct 29 23:27:15.569407 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:15.569273 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-3e5b21-f69958676-g78g2" event={"ID":"0157b7d8-b442-4302-9fc2-4e87e55ad7aa","Type":"ContainerDied","Data":"86642ea1ec030e84cb937fbdbacf3cb366591e581de8ece19dbfca0a3c7d155b"} Oct 29 23:27:15.569407 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:15.569155 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-3e5b21-f69958676-g78g2" Oct 29 23:27:15.579377 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:15.579355 5057 scope.go:122] "RemoveContainer" containerID="91078cc6e3795c4ac92f4e63a8fc9158442f9ae23309168a63e15a221a4d9edd" Oct 29 23:27:15.579685 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:27:15.579660 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91078cc6e3795c4ac92f4e63a8fc9158442f9ae23309168a63e15a221a4d9edd\": container with ID starting with 91078cc6e3795c4ac92f4e63a8fc9158442f9ae23309168a63e15a221a4d9edd not found: ID does not exist" containerID="91078cc6e3795c4ac92f4e63a8fc9158442f9ae23309168a63e15a221a4d9edd" Oct 29 23:27:15.579762 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:15.579692 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91078cc6e3795c4ac92f4e63a8fc9158442f9ae23309168a63e15a221a4d9edd"} err="failed to get container status \"91078cc6e3795c4ac92f4e63a8fc9158442f9ae23309168a63e15a221a4d9edd\": rpc error: code = NotFound desc = could not find container \"91078cc6e3795c4ac92f4e63a8fc9158442f9ae23309168a63e15a221a4d9edd\": container with ID starting with 91078cc6e3795c4ac92f4e63a8fc9158442f9ae23309168a63e15a221a4d9edd not found: ID does not exist" Oct 29 23:27:15.579762 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:15.579709 5057 scope.go:122] "RemoveContainer" containerID="8aaf466ab895e32406715310466b10620eeaa03863836a83b5f1fc50ad9c378b" Oct 29 23:27:15.590462 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:15.590425 5057 scope.go:122] "RemoveContainer" containerID="8aaf466ab895e32406715310466b10620eeaa03863836a83b5f1fc50ad9c378b" Oct 29 23:27:15.590842 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:27:15.590812 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8aaf466ab895e32406715310466b10620eeaa03863836a83b5f1fc50ad9c378b\": container with ID starting with 8aaf466ab895e32406715310466b10620eeaa03863836a83b5f1fc50ad9c378b not found: ID does not exist" containerID="8aaf466ab895e32406715310466b10620eeaa03863836a83b5f1fc50ad9c378b" Oct 29 23:27:15.590937 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:15.590852 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8aaf466ab895e32406715310466b10620eeaa03863836a83b5f1fc50ad9c378b"} err="failed to get container status \"8aaf466ab895e32406715310466b10620eeaa03863836a83b5f1fc50ad9c378b\": rpc error: code = NotFound desc = could not find container \"8aaf466ab895e32406715310466b10620eeaa03863836a83b5f1fc50ad9c378b\": container with ID starting with 8aaf466ab895e32406715310466b10620eeaa03863836a83b5f1fc50ad9c378b not found: ID does not exist" Oct 29 23:27:15.630642 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:15.630209 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-3e5b21-d7b5c69c5-ds82z"] Oct 29 23:27:15.640147 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:15.640118 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-3e5b21-d7b5c69c5-ds82z"] Oct 29 23:27:15.660966 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:15.660932 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-3e5b21-f69958676-g78g2"] Oct 29 23:27:15.666124 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:15.666094 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-3e5b21-f69958676-g78g2"] Oct 29 23:27:16.619649 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:16.619607 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:27:17.161730 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:17.161705 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-024aa6-b99d57cd7-pslx4" Oct 29 23:27:17.247417 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:17.240064 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a5e765c-b65b-4411-92a5-fad9521c9291-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a5e765c-b65b-4411-92a5-fad9521c9291-plugins-config-volume\") pod \"7a5e765c-b65b-4411-92a5-fad9521c9291\" (UID: \"7a5e765c-b65b-4411-92a5-fad9521c9291\") " Oct 29 23:27:17.247417 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:17.240238 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a5e765c-b65b-4411-92a5-fad9521c9291-kube-api-access-8hpct\" (UniqueName: \"kubernetes.io/projected/7a5e765c-b65b-4411-92a5-fad9521c9291-kube-api-access-8hpct\") pod \"7a5e765c-b65b-4411-92a5-fad9521c9291\" (UID: \"7a5e765c-b65b-4411-92a5-fad9521c9291\") " Oct 29 23:27:17.247417 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:17.241073 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a5e765c-b65b-4411-92a5-fad9521c9291-plugins-config-volume" pod "7a5e765c-b65b-4411-92a5-fad9521c9291" (UID: "7a5e765c-b65b-4411-92a5-fad9521c9291"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:27:17.247417 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:17.242658 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a5e765c-b65b-4411-92a5-fad9521c9291-kube-api-access-8hpct" pod "7a5e765c-b65b-4411-92a5-fad9521c9291" (UID: "7a5e765c-b65b-4411-92a5-fad9521c9291"). InnerVolumeSpecName "kube-api-access-8hpct". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:27:17.333967 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:17.333933 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0157b7d8-b442-4302-9fc2-4e87e55ad7aa" path="/var/lib/kubelet/pods/0157b7d8-b442-4302-9fc2-4e87e55ad7aa/volumes" Oct 29 23:27:17.334460 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:17.334444 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e05f5f87-b9fc-4dfa-b61a-ce1f776e146c" path="/var/lib/kubelet/pods/e05f5f87-b9fc-4dfa-b61a-ce1f776e146c/volumes" Oct 29 23:27:17.342830 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:17.342801 5057 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a5e765c-b65b-4411-92a5-fad9521c9291-plugins-config-volume\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:27:17.342952 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:17.342833 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8hpct\" (UniqueName: \"kubernetes.io/projected/7a5e765c-b65b-4411-92a5-fad9521c9291-kube-api-access-8hpct\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:27:17.599762 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:17.599673 5057 generic.go:356] "Generic (PLEG): container finished" podID="7a5e765c-b65b-4411-92a5-fad9521c9291" containerID="a40acdcd1dd204a8e3273392b221ff5ebe518c72f4fb3d8b53d68302c894c7b2" exitCode=0 Oct 29 23:27:17.599762 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:17.599727 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-024aa6-b99d57cd7-pslx4" Oct 29 23:27:17.599997 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:17.599756 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-024aa6-b99d57cd7-pslx4" event={"ID":"7a5e765c-b65b-4411-92a5-fad9521c9291","Type":"ContainerDied","Data":"a40acdcd1dd204a8e3273392b221ff5ebe518c72f4fb3d8b53d68302c894c7b2"} Oct 29 23:27:17.599997 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:17.599812 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-024aa6-b99d57cd7-pslx4" event={"ID":"7a5e765c-b65b-4411-92a5-fad9521c9291","Type":"ContainerDied","Data":"57bbe331c3f257180b6981b4778d0bbc0279b4531420da65936dfcb2c18e766a"} Oct 29 23:27:17.599997 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:17.599842 5057 scope.go:122] "RemoveContainer" containerID="a40acdcd1dd204a8e3273392b221ff5ebe518c72f4fb3d8b53d68302c894c7b2" Oct 29 23:27:17.615598 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:17.615574 5057 scope.go:122] "RemoveContainer" containerID="a40acdcd1dd204a8e3273392b221ff5ebe518c72f4fb3d8b53d68302c894c7b2" Oct 29 23:27:17.615932 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:27:17.615898 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a40acdcd1dd204a8e3273392b221ff5ebe518c72f4fb3d8b53d68302c894c7b2\": container with ID starting with a40acdcd1dd204a8e3273392b221ff5ebe518c72f4fb3d8b53d68302c894c7b2 not found: ID does not exist" containerID="a40acdcd1dd204a8e3273392b221ff5ebe518c72f4fb3d8b53d68302c894c7b2" Oct 29 23:27:17.616053 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:17.615940 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a40acdcd1dd204a8e3273392b221ff5ebe518c72f4fb3d8b53d68302c894c7b2"} err="failed to get container status \"a40acdcd1dd204a8e3273392b221ff5ebe518c72f4fb3d8b53d68302c894c7b2\": rpc error: code = NotFound desc = could not find container \"a40acdcd1dd204a8e3273392b221ff5ebe518c72f4fb3d8b53d68302c894c7b2\": container with ID starting with a40acdcd1dd204a8e3273392b221ff5ebe518c72f4fb3d8b53d68302c894c7b2 not found: ID does not exist" Oct 29 23:27:17.653771 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:17.653741 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-024aa6-b99d57cd7-pslx4"] Oct 29 23:27:17.665918 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:17.665887 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-024aa6-b99d57cd7-pslx4"] Oct 29 23:27:18.289006 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:18.288973 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:27:19.061977 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:27:19.061941 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d2dnn" containerName="extractor" Oct 29 23:27:19.091187 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:19.091161 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-dcacb9-767c887c7b-jrk26" Oct 29 23:27:19.095479 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:19.095455 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-dcacb9-86d7d89559-xk4m6" Oct 29 23:27:19.275506 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:19.275425 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38161411-6a7a-4a65-aacf-e39baf965f8e-kube-api-access-gsqb2\" (UniqueName: \"kubernetes.io/projected/38161411-6a7a-4a65-aacf-e39baf965f8e-kube-api-access-gsqb2\") pod \"38161411-6a7a-4a65-aacf-e39baf965f8e\" (UID: \"38161411-6a7a-4a65-aacf-e39baf965f8e\") " Oct 29 23:27:19.275506 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:19.275474 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/38161411-6a7a-4a65-aacf-e39baf965f8e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38161411-6a7a-4a65-aacf-e39baf965f8e-plugins-config-volume\") pod \"38161411-6a7a-4a65-aacf-e39baf965f8e\" (UID: \"38161411-6a7a-4a65-aacf-e39baf965f8e\") " Oct 29 23:27:19.275707 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:19.275603 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26ab3c63-6e66-4981-b86c-bdf7055db34f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26ab3c63-6e66-4981-b86c-bdf7055db34f-plugins-config-volume\") pod \"26ab3c63-6e66-4981-b86c-bdf7055db34f\" (UID: \"26ab3c63-6e66-4981-b86c-bdf7055db34f\") " Oct 29 23:27:19.275763 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:19.275704 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26ab3c63-6e66-4981-b86c-bdf7055db34f-kube-api-access-sn2sd\" (UniqueName: \"kubernetes.io/projected/26ab3c63-6e66-4981-b86c-bdf7055db34f-kube-api-access-sn2sd\") pod \"26ab3c63-6e66-4981-b86c-bdf7055db34f\" (UID: \"26ab3c63-6e66-4981-b86c-bdf7055db34f\") " Oct 29 23:27:19.275955 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:19.275933 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38161411-6a7a-4a65-aacf-e39baf965f8e-plugins-config-volume" pod "38161411-6a7a-4a65-aacf-e39baf965f8e" (UID: "38161411-6a7a-4a65-aacf-e39baf965f8e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:27:19.276040 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:19.275944 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26ab3c63-6e66-4981-b86c-bdf7055db34f-plugins-config-volume" pod "26ab3c63-6e66-4981-b86c-bdf7055db34f" (UID: "26ab3c63-6e66-4981-b86c-bdf7055db34f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:27:19.276437 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:19.276421 5057 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26ab3c63-6e66-4981-b86c-bdf7055db34f-plugins-config-volume\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:27:19.276498 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:19.276439 5057 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38161411-6a7a-4a65-aacf-e39baf965f8e-plugins-config-volume\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:27:19.277759 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:19.277730 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26ab3c63-6e66-4981-b86c-bdf7055db34f-kube-api-access-sn2sd" pod "26ab3c63-6e66-4981-b86c-bdf7055db34f" (UID: "26ab3c63-6e66-4981-b86c-bdf7055db34f"). InnerVolumeSpecName "kube-api-access-sn2sd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:27:19.277925 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:19.277904 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38161411-6a7a-4a65-aacf-e39baf965f8e-kube-api-access-gsqb2" pod "38161411-6a7a-4a65-aacf-e39baf965f8e" (UID: "38161411-6a7a-4a65-aacf-e39baf965f8e"). InnerVolumeSpecName "kube-api-access-gsqb2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:27:19.330869 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:19.330840 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a5e765c-b65b-4411-92a5-fad9521c9291" path="/var/lib/kubelet/pods/7a5e765c-b65b-4411-92a5-fad9521c9291/volumes" Oct 29 23:27:19.378100 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:19.378075 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sn2sd\" (UniqueName: \"kubernetes.io/projected/26ab3c63-6e66-4981-b86c-bdf7055db34f-kube-api-access-sn2sd\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:27:19.378100 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:19.378101 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gsqb2\" (UniqueName: \"kubernetes.io/projected/38161411-6a7a-4a65-aacf-e39baf965f8e-kube-api-access-gsqb2\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:27:19.640110 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:19.640023 5057 generic.go:356] "Generic (PLEG): container finished" podID="26ab3c63-6e66-4981-b86c-bdf7055db34f" containerID="09baca9166765ceb8dd71e72fb60b7cdeabed1278bb49142c70a5587b32ab390" exitCode=0 Oct 29 23:27:19.640110 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:19.640081 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-dcacb9-767c887c7b-jrk26" Oct 29 23:27:19.640365 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:19.640115 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-dcacb9-767c887c7b-jrk26" event={"ID":"26ab3c63-6e66-4981-b86c-bdf7055db34f","Type":"ContainerDied","Data":"09baca9166765ceb8dd71e72fb60b7cdeabed1278bb49142c70a5587b32ab390"} Oct 29 23:27:19.640365 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:19.640179 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-dcacb9-767c887c7b-jrk26" event={"ID":"26ab3c63-6e66-4981-b86c-bdf7055db34f","Type":"ContainerDied","Data":"625553809c5ee865c0ceb62978a86c5120bdd73dc2a6747105a414afcc143d5d"} Oct 29 23:27:19.640365 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:19.640211 5057 scope.go:122] "RemoveContainer" containerID="09baca9166765ceb8dd71e72fb60b7cdeabed1278bb49142c70a5587b32ab390" Oct 29 23:27:19.642001 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:19.641975 5057 generic.go:356] "Generic (PLEG): container finished" podID="38161411-6a7a-4a65-aacf-e39baf965f8e" containerID="7af091fe73f5cc4c7a442becd64d2f40546de358686d9b023fcd45d2cef5ec3a" exitCode=0 Oct 29 23:27:19.642154 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:19.642015 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-dcacb9-86d7d89559-xk4m6" Oct 29 23:27:19.642154 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:19.642109 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-dcacb9-86d7d89559-xk4m6" event={"ID":"38161411-6a7a-4a65-aacf-e39baf965f8e","Type":"ContainerDied","Data":"7af091fe73f5cc4c7a442becd64d2f40546de358686d9b023fcd45d2cef5ec3a"} Oct 29 23:27:19.642154 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:19.642148 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-dcacb9-86d7d89559-xk4m6" event={"ID":"38161411-6a7a-4a65-aacf-e39baf965f8e","Type":"ContainerDied","Data":"a84e586a6dce7e5e458fe1f7635c81cce8aa562f97e1f1406b2b180fbaa11ff4"} Oct 29 23:27:19.652273 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:19.652252 5057 scope.go:122] "RemoveContainer" containerID="09baca9166765ceb8dd71e72fb60b7cdeabed1278bb49142c70a5587b32ab390" Oct 29 23:27:19.652563 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:27:19.652546 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09baca9166765ceb8dd71e72fb60b7cdeabed1278bb49142c70a5587b32ab390\": container with ID starting with 09baca9166765ceb8dd71e72fb60b7cdeabed1278bb49142c70a5587b32ab390 not found: ID does not exist" containerID="09baca9166765ceb8dd71e72fb60b7cdeabed1278bb49142c70a5587b32ab390" Oct 29 23:27:19.652627 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:19.652575 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09baca9166765ceb8dd71e72fb60b7cdeabed1278bb49142c70a5587b32ab390"} err="failed to get container status \"09baca9166765ceb8dd71e72fb60b7cdeabed1278bb49142c70a5587b32ab390\": rpc error: code = NotFound desc = could not find container \"09baca9166765ceb8dd71e72fb60b7cdeabed1278bb49142c70a5587b32ab390\": container with ID starting with 09baca9166765ceb8dd71e72fb60b7cdeabed1278bb49142c70a5587b32ab390 not found: ID does not exist" Oct 29 23:27:19.652627 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:19.652593 5057 scope.go:122] "RemoveContainer" containerID="7af091fe73f5cc4c7a442becd64d2f40546de358686d9b023fcd45d2cef5ec3a" Oct 29 23:27:19.663951 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:19.663931 5057 scope.go:122] "RemoveContainer" containerID="7af091fe73f5cc4c7a442becd64d2f40546de358686d9b023fcd45d2cef5ec3a" Oct 29 23:27:19.664213 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:27:19.664196 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7af091fe73f5cc4c7a442becd64d2f40546de358686d9b023fcd45d2cef5ec3a\": container with ID starting with 7af091fe73f5cc4c7a442becd64d2f40546de358686d9b023fcd45d2cef5ec3a not found: ID does not exist" containerID="7af091fe73f5cc4c7a442becd64d2f40546de358686d9b023fcd45d2cef5ec3a" Oct 29 23:27:19.664285 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:19.664217 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7af091fe73f5cc4c7a442becd64d2f40546de358686d9b023fcd45d2cef5ec3a"} err="failed to get container status \"7af091fe73f5cc4c7a442becd64d2f40546de358686d9b023fcd45d2cef5ec3a\": rpc error: code = NotFound desc = could not find container \"7af091fe73f5cc4c7a442becd64d2f40546de358686d9b023fcd45d2cef5ec3a\": container with ID starting with 7af091fe73f5cc4c7a442becd64d2f40546de358686d9b023fcd45d2cef5ec3a not found: ID does not exist" Oct 29 23:27:19.695674 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:19.695649 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-dcacb9-767c887c7b-jrk26"] Oct 29 23:27:19.708655 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:19.708628 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-dcacb9-767c887c7b-jrk26"] Oct 29 23:27:19.731325 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:19.731295 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-dcacb9-86d7d89559-xk4m6"] Oct 29 23:27:19.737322 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:19.737301 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-dcacb9-86d7d89559-xk4m6"] Oct 29 23:27:19.883843 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:19.883801 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:27:21.330584 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:21.330546 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26ab3c63-6e66-4981-b86c-bdf7055db34f" path="/var/lib/kubelet/pods/26ab3c63-6e66-4981-b86c-bdf7055db34f/volumes" Oct 29 23:27:21.331015 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:21.330999 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38161411-6a7a-4a65-aacf-e39baf965f8e" path="/var/lib/kubelet/pods/38161411-6a7a-4a65-aacf-e39baf965f8e/volumes" Oct 29 23:27:21.509028 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:21.508991 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:27:21.875031 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:21.874993 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-c880c7-s2nck"] Oct 29 23:27:21.875992 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:21.875965 5057 cpu_manager.go:414] "containerMap: removing container" podUID="d9ed3669-6751-4a9c-8c44-ba9a7df4146a" containerName="revoke-keys" Oct 29 23:27:21.875992 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:21.875990 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9ed3669-6751-4a9c-8c44-ba9a7df4146a" containerName="revoke-keys" Oct 29 23:27:21.876195 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:21.876009 5057 cpu_manager.go:414] "containerMap: removing container" podUID="26ab3c63-6e66-4981-b86c-bdf7055db34f" containerName="payload-processing" Oct 29 23:27:21.876195 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:21.876016 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26ab3c63-6e66-4981-b86c-bdf7055db34f" containerName="payload-processing" Oct 29 23:27:21.876195 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:21.876031 5057 cpu_manager.go:414] "containerMap: removing container" podUID="7a5e765c-b65b-4411-92a5-fad9521c9291" containerName="payload-pre-processing" Oct 29 23:27:21.876195 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:21.876040 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a5e765c-b65b-4411-92a5-fad9521c9291" containerName="payload-pre-processing" Oct 29 23:27:21.876195 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:21.876057 5057 cpu_manager.go:414] "containerMap: removing container" podUID="38161411-6a7a-4a65-aacf-e39baf965f8e" containerName="payload-pre-processing" Oct 29 23:27:21.876195 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:21.876064 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38161411-6a7a-4a65-aacf-e39baf965f8e" containerName="payload-pre-processing" Oct 29 23:27:21.876195 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:21.876086 5057 cpu_manager.go:414] "containerMap: removing container" podUID="e05f5f87-b9fc-4dfa-b61a-ce1f776e146c" containerName="payload-pre-processing" Oct 29 23:27:21.876195 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:21.876094 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e05f5f87-b9fc-4dfa-b61a-ce1f776e146c" containerName="payload-pre-processing" Oct 29 23:27:21.876195 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:21.876139 5057 cpu_manager.go:414] "containerMap: removing container" podUID="0157b7d8-b442-4302-9fc2-4e87e55ad7aa" containerName="payload-processing" Oct 29 23:27:21.876195 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:21.876146 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0157b7d8-b442-4302-9fc2-4e87e55ad7aa" containerName="payload-processing" Oct 29 23:27:21.876793 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:21.876201 5057 cpu_manager.go:414] "containerMap: removing container" podUID="b56f7c05-6607-4652-a92d-a55b5b21fcf0" containerName="revoke-keys" Oct 29 23:27:21.876793 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:21.876209 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b56f7c05-6607-4652-a92d-a55b5b21fcf0" containerName="revoke-keys" Oct 29 23:27:21.876793 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:21.876447 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9ed3669-6751-4a9c-8c44-ba9a7df4146a" containerName="revoke-keys" Oct 29 23:27:21.876793 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:21.876469 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="26ab3c63-6e66-4981-b86c-bdf7055db34f" containerName="payload-processing" Oct 29 23:27:21.876793 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:21.876482 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a5e765c-b65b-4411-92a5-fad9521c9291" containerName="payload-pre-processing" Oct 29 23:27:21.876793 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:21.876494 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="38161411-6a7a-4a65-aacf-e39baf965f8e" containerName="payload-pre-processing" Oct 29 23:27:21.876793 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:21.876504 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="e05f5f87-b9fc-4dfa-b61a-ce1f776e146c" containerName="payload-pre-processing" Oct 29 23:27:21.876793 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:21.876524 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="0157b7d8-b442-4302-9fc2-4e87e55ad7aa" containerName="payload-processing" Oct 29 23:27:21.876793 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:21.876547 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="b56f7c05-6607-4652-a92d-a55b5b21fcf0" containerName="revoke-keys" Oct 29 23:27:21.882257 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:21.882237 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-c880c7-s2nck" Oct 29 23:27:21.899877 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:21.899848 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-c880c7-s2nck"] Oct 29 23:27:21.905009 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:21.904989 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:27:21.905388 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:21.905369 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dmx57\"" Oct 29 23:27:22.021699 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:22.021660 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/451b200e-e1c1-4acc-968c-c79f0e1425de-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-c880c7-s2nck\" (UID: \"451b200e-e1c1-4acc-968c-c79f0e1425de\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-c880c7-s2nck" Oct 29 23:27:22.021877 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:22.021816 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g878f\" (UniqueName: \"kubernetes.io/projected/451b200e-e1c1-4acc-968c-c79f0e1425de-kube-api-access-g878f\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-c880c7-s2nck\" (UID: \"451b200e-e1c1-4acc-968c-c79f0e1425de\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-c880c7-s2nck" Oct 29 23:27:22.021968 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:22.021893 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/451b200e-e1c1-4acc-968c-c79f0e1425de-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-c880c7-s2nck\" (UID: \"451b200e-e1c1-4acc-968c-c79f0e1425de\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-c880c7-s2nck" Oct 29 23:27:22.124134 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:22.124097 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g878f\" (UniqueName: \"kubernetes.io/projected/451b200e-e1c1-4acc-968c-c79f0e1425de-kube-api-access-g878f\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-c880c7-s2nck\" (UID: \"451b200e-e1c1-4acc-968c-c79f0e1425de\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-c880c7-s2nck" Oct 29 23:27:22.124361 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:22.124159 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/451b200e-e1c1-4acc-968c-c79f0e1425de-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-c880c7-s2nck\" (UID: \"451b200e-e1c1-4acc-968c-c79f0e1425de\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-c880c7-s2nck" Oct 29 23:27:22.124545 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:22.124526 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/451b200e-e1c1-4acc-968c-c79f0e1425de-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-c880c7-s2nck\" (UID: \"451b200e-e1c1-4acc-968c-c79f0e1425de\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-c880c7-s2nck" Oct 29 23:27:22.124881 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:22.124856 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/451b200e-e1c1-4acc-968c-c79f0e1425de-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-c880c7-s2nck\" (UID: \"451b200e-e1c1-4acc-968c-c79f0e1425de\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-c880c7-s2nck" Oct 29 23:27:22.126986 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:22.126929 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/451b200e-e1c1-4acc-968c-c79f0e1425de-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-c880c7-s2nck\" (UID: \"451b200e-e1c1-4acc-968c-c79f0e1425de\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-c880c7-s2nck" Oct 29 23:27:22.146698 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:22.146675 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g878f\" (UniqueName: \"kubernetes.io/projected/451b200e-e1c1-4acc-968c-c79f0e1425de-kube-api-access-g878f\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-c880c7-s2nck\" (UID: \"451b200e-e1c1-4acc-968c-c79f0e1425de\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-c880c7-s2nck" Oct 29 23:27:22.208171 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:22.208128 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-c880c7-s2nck" Oct 29 23:27:22.525214 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:22.525183 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-c880c7-s2nck"] Oct 29 23:27:22.525885 ip-10-0-140-74 kubenswrapper[5057]: W1029 23:27:22.525855 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod451b200e_e1c1_4acc_968c_c79f0e1425de.slice/crio-a335a6bf06ff0202b295a82411b21422f5a5270bd5b0d2a489ff80ccb2f24416 WatchSource:0}: Error finding container a335a6bf06ff0202b295a82411b21422f5a5270bd5b0d2a489ff80ccb2f24416: Status 404 returned error can't find the container with id a335a6bf06ff0202b295a82411b21422f5a5270bd5b0d2a489ff80ccb2f24416 Oct 29 23:27:22.672292 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:22.672260 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-c880c7-s2nck" event={"ID":"451b200e-e1c1-4acc-968c-c79f0e1425de","Type":"ContainerStarted","Data":"a335a6bf06ff0202b295a82411b21422f5a5270bd5b0d2a489ff80ccb2f24416"} Oct 29 23:27:22.963553 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:22.963522 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:27:23.596848 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:23.596817 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:27:23.683366 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:23.683299 5057 generic.go:356] "Generic (PLEG): container finished" podID="451b200e-e1c1-4acc-968c-c79f0e1425de" containerID="a362653d5a9b4c999fa6c9862f67171d87c2483245fbf5446675f0caf135a30e" exitCode=0 Oct 29 23:27:23.683532 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:23.683361 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-c880c7-s2nck" event={"ID":"451b200e-e1c1-4acc-968c-c79f0e1425de","Type":"ContainerDied","Data":"a362653d5a9b4c999fa6c9862f67171d87c2483245fbf5446675f0caf135a30e"} Oct 29 23:27:23.708965 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:23.708935 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:27:23.819050 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:23.818967 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:27:24.249241 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:24.249205 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:27:24.869779 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:24.869750 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-c880c7-s2nck" Oct 29 23:27:24.970026 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:24.969986 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/451b200e-e1c1-4acc-968c-c79f0e1425de-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/451b200e-e1c1-4acc-968c-c79f0e1425de-maas-api-xks-ca\") pod \"451b200e-e1c1-4acc-968c-c79f0e1425de\" (UID: \"451b200e-e1c1-4acc-968c-c79f0e1425de\") " Oct 29 23:27:24.970244 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:24.970051 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/451b200e-e1c1-4acc-968c-c79f0e1425de-kube-api-access-g878f\" (UniqueName: \"kubernetes.io/projected/451b200e-e1c1-4acc-968c-c79f0e1425de-kube-api-access-g878f\") pod \"451b200e-e1c1-4acc-968c-c79f0e1425de\" (UID: \"451b200e-e1c1-4acc-968c-c79f0e1425de\") " Oct 29 23:27:24.970244 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:24.970114 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/451b200e-e1c1-4acc-968c-c79f0e1425de-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/451b200e-e1c1-4acc-968c-c79f0e1425de-maas-api-openshift-ca\") pod \"451b200e-e1c1-4acc-968c-c79f0e1425de\" (UID: \"451b200e-e1c1-4acc-968c-c79f0e1425de\") " Oct 29 23:27:24.971486 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:24.971451 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/451b200e-e1c1-4acc-968c-c79f0e1425de-maas-api-openshift-ca" pod "451b200e-e1c1-4acc-968c-c79f0e1425de" (UID: "451b200e-e1c1-4acc-968c-c79f0e1425de"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:27:24.973078 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:24.973047 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/451b200e-e1c1-4acc-968c-c79f0e1425de-maas-api-xks-ca" pod "451b200e-e1c1-4acc-968c-c79f0e1425de" (UID: "451b200e-e1c1-4acc-968c-c79f0e1425de"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:27:24.978441 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:24.978411 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/451b200e-e1c1-4acc-968c-c79f0e1425de-kube-api-access-g878f" pod "451b200e-e1c1-4acc-968c-c79f0e1425de" (UID: "451b200e-e1c1-4acc-968c-c79f0e1425de"). InnerVolumeSpecName "kube-api-access-g878f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:27:25.073028 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:25.072998 5057 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/451b200e-e1c1-4acc-968c-c79f0e1425de-maas-api-xks-ca\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:27:25.073028 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:25.073028 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g878f\" (UniqueName: \"kubernetes.io/projected/451b200e-e1c1-4acc-968c-c79f0e1425de-kube-api-access-g878f\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:27:25.073190 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:25.073040 5057 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/451b200e-e1c1-4acc-968c-c79f0e1425de-maas-api-openshift-ca\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:27:25.258390 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:25.258280 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:27:25.740902 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:25.740865 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-c880c7-s2nck" event={"ID":"451b200e-e1c1-4acc-968c-c79f0e1425de","Type":"ContainerDied","Data":"a335a6bf06ff0202b295a82411b21422f5a5270bd5b0d2a489ff80ccb2f24416"} Oct 29 23:27:25.741104 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:25.740926 5057 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a335a6bf06ff0202b295a82411b21422f5a5270bd5b0d2a489ff80ccb2f24416" Oct 29 23:27:25.741104 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:25.740997 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-c880c7-s2nck" Oct 29 23:27:26.085259 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:26.085177 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:27:26.227082 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:26.227042 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:27:26.265780 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:26.265747 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-024aa6-wzpcp"] Oct 29 23:27:26.270426 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:26.270402 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-024aa6-wzpcp"] Oct 29 23:27:26.340442 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:26.340347 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:27:26.750723 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:26.750690 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:27:27.332301 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:27.332263 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec6f58bd-af5d-40f6-a333-d355d71fe880" path="/var/lib/kubelet/pods/ec6f58bd-af5d-40f6-a333-d355d71fe880/volumes" Oct 29 23:27:27.993448 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:27.993412 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:27:29.616484 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:29.616443 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:27:30.118822 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:30.118788 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dcc10992-4pss8"] Oct 29 23:27:30.119799 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:30.119780 5057 cpu_manager.go:414] "containerMap: removing container" podUID="451b200e-e1c1-4acc-968c-c79f0e1425de" containerName="revoke-keys" Oct 29 23:27:30.119880 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:30.119800 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="451b200e-e1c1-4acc-968c-c79f0e1425de" containerName="revoke-keys" Oct 29 23:27:30.119937 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:30.119927 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="451b200e-e1c1-4acc-968c-c79f0e1425de" containerName="revoke-keys" Oct 29 23:27:30.125093 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:30.125074 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcc10992-4pss8" Oct 29 23:27:30.142218 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:30.142190 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dcc10992-4pss8"] Oct 29 23:27:30.147177 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:30.147157 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:27:30.147288 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:30.147255 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dmx57\"" Oct 29 23:27:30.250382 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:30.250344 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5a9fd37-8a4a-4703-9114-ca13cac40108-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dcc10992-4pss8\" (UID: \"a5a9fd37-8a4a-4703-9114-ca13cac40108\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcc10992-4pss8" Oct 29 23:27:30.250783 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:30.250754 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5wr4\" (UniqueName: \"kubernetes.io/projected/a5a9fd37-8a4a-4703-9114-ca13cac40108-kube-api-access-h5wr4\") pod \"maas-api-revoke-sub-dcc10992-4pss8\" (UID: \"a5a9fd37-8a4a-4703-9114-ca13cac40108\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcc10992-4pss8" Oct 29 23:27:30.250873 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:30.250802 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5a9fd37-8a4a-4703-9114-ca13cac40108-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dcc10992-4pss8\" (UID: \"a5a9fd37-8a4a-4703-9114-ca13cac40108\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcc10992-4pss8" Oct 29 23:27:30.352469 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:30.352433 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5wr4\" (UniqueName: \"kubernetes.io/projected/a5a9fd37-8a4a-4703-9114-ca13cac40108-kube-api-access-h5wr4\") pod \"maas-api-revoke-sub-dcc10992-4pss8\" (UID: \"a5a9fd37-8a4a-4703-9114-ca13cac40108\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcc10992-4pss8" Oct 29 23:27:30.352469 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:30.352472 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5a9fd37-8a4a-4703-9114-ca13cac40108-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dcc10992-4pss8\" (UID: \"a5a9fd37-8a4a-4703-9114-ca13cac40108\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcc10992-4pss8" Oct 29 23:27:30.352686 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:30.352516 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5a9fd37-8a4a-4703-9114-ca13cac40108-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dcc10992-4pss8\" (UID: \"a5a9fd37-8a4a-4703-9114-ca13cac40108\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcc10992-4pss8" Oct 29 23:27:30.353286 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:30.353260 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5a9fd37-8a4a-4703-9114-ca13cac40108-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dcc10992-4pss8\" (UID: \"a5a9fd37-8a4a-4703-9114-ca13cac40108\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcc10992-4pss8" Oct 29 23:27:30.354899 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:30.354877 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5a9fd37-8a4a-4703-9114-ca13cac40108-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dcc10992-4pss8\" (UID: \"a5a9fd37-8a4a-4703-9114-ca13cac40108\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcc10992-4pss8" Oct 29 23:27:30.377262 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:30.377201 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5wr4\" (UniqueName: \"kubernetes.io/projected/a5a9fd37-8a4a-4703-9114-ca13cac40108-kube-api-access-h5wr4\") pod \"maas-api-revoke-sub-dcc10992-4pss8\" (UID: \"a5a9fd37-8a4a-4703-9114-ca13cac40108\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcc10992-4pss8" Oct 29 23:27:30.440711 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:30.440683 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcc10992-4pss8" Oct 29 23:27:30.757824 ip-10-0-140-74 kubenswrapper[5057]: W1029 23:27:30.757794 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5a9fd37_8a4a_4703_9114_ca13cac40108.slice/crio-5837602bcfeb1756b7df340fbbb9d504f3026dc398e55e526f8950748a2dbad4 WatchSource:0}: Error finding container 5837602bcfeb1756b7df340fbbb9d504f3026dc398e55e526f8950748a2dbad4: Status 404 returned error can't find the container with id 5837602bcfeb1756b7df340fbbb9d504f3026dc398e55e526f8950748a2dbad4 Oct 29 23:27:30.760391 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:30.760363 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dcc10992-4pss8"] Oct 29 23:27:30.788660 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:30.788637 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcc10992-4pss8" event={"ID":"a5a9fd37-8a4a-4703-9114-ca13cac40108","Type":"ContainerStarted","Data":"5837602bcfeb1756b7df340fbbb9d504f3026dc398e55e526f8950748a2dbad4"} Oct 29 23:27:31.042643 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:31.042554 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:27:31.160428 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:31.160381 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:27:31.290013 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:31.289535 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:27:31.404384 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:31.399553 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-dcacb9-openshift-default-54b777975-58gd9"] Oct 29 23:27:31.404384 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:31.399792 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-dcacb9-openshift-default-54b777975-58gd9" podUID="3c8c58af-2d80-4b3e-901f-a0a65338b5d7" containerName="istio-proxy" containerID="cri-o://7b92baa13c881e9839e072042cbcb7a8ca8d9c87ac1cc91dccad45334fe527d5" gracePeriod=30 Oct 29 23:27:31.416387 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:31.416307 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:27:31.801296 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:31.801256 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcc10992-4pss8" event={"ID":"a5a9fd37-8a4a-4703-9114-ca13cac40108","Type":"ContainerStarted","Data":"7aa13ef41f18e9c3406f2adbcde5d172261970c2edbe179c1b8191531624bcb2"} Oct 29 23:27:31.846126 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:31.846077 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcc10992-4pss8" podStartSLOduration=1.8460631680000001 podStartE2EDuration="1.846063168s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:27:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:27:31.840854086 +0000 UTC m=+3232.624067511" watchObservedRunningTime="2026-10-29 23:27:31.846063168 +0000 UTC m=+3232.629276591" Oct 29 23:27:32.321102 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:32.321069 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-c880c7-5d864f4568-xqcv8"] Oct 29 23:27:32.366882 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:32.366839 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-c880c7-58cdf8ddd7-fj789"] Oct 29 23:27:32.497243 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:32.497161 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:27:32.728587 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:32.728548 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:27:32.813850 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:32.813811 5057 generic.go:356] "Generic (PLEG): container finished" podID="a5a9fd37-8a4a-4703-9114-ca13cac40108" containerID="7aa13ef41f18e9c3406f2adbcde5d172261970c2edbe179c1b8191531624bcb2" exitCode=0 Oct 29 23:27:32.814289 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:32.813906 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcc10992-4pss8" event={"ID":"a5a9fd37-8a4a-4703-9114-ca13cac40108","Type":"ContainerDied","Data":"7aa13ef41f18e9c3406f2adbcde5d172261970c2edbe179c1b8191531624bcb2"} Oct 29 23:27:33.033634 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:33.033548 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:27:34.011934 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:34.011900 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcc10992-4pss8" Oct 29 23:27:34.140378 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:34.140259 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a5a9fd37-8a4a-4703-9114-ca13cac40108-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5a9fd37-8a4a-4703-9114-ca13cac40108-maas-api-xks-ca\") pod \"a5a9fd37-8a4a-4703-9114-ca13cac40108\" (UID: \"a5a9fd37-8a4a-4703-9114-ca13cac40108\") " Oct 29 23:27:34.140566 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:34.140411 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5a9fd37-8a4a-4703-9114-ca13cac40108-kube-api-access-h5wr4\" (UniqueName: \"kubernetes.io/projected/a5a9fd37-8a4a-4703-9114-ca13cac40108-kube-api-access-h5wr4\") pod \"a5a9fd37-8a4a-4703-9114-ca13cac40108\" (UID: \"a5a9fd37-8a4a-4703-9114-ca13cac40108\") " Oct 29 23:27:34.140566 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:34.140475 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5a9fd37-8a4a-4703-9114-ca13cac40108-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5a9fd37-8a4a-4703-9114-ca13cac40108-maas-api-openshift-ca\") pod \"a5a9fd37-8a4a-4703-9114-ca13cac40108\" (UID: \"a5a9fd37-8a4a-4703-9114-ca13cac40108\") " Oct 29 23:27:34.140877 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:34.140849 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5a9fd37-8a4a-4703-9114-ca13cac40108-maas-api-openshift-ca" pod "a5a9fd37-8a4a-4703-9114-ca13cac40108" (UID: "a5a9fd37-8a4a-4703-9114-ca13cac40108"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:27:34.141051 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:34.141022 5057 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5a9fd37-8a4a-4703-9114-ca13cac40108-maas-api-openshift-ca\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:27:34.142413 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:34.142393 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5a9fd37-8a4a-4703-9114-ca13cac40108-maas-api-xks-ca" pod "a5a9fd37-8a4a-4703-9114-ca13cac40108" (UID: "a5a9fd37-8a4a-4703-9114-ca13cac40108"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:27:34.142593 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:34.142577 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5a9fd37-8a4a-4703-9114-ca13cac40108-kube-api-access-h5wr4" pod "a5a9fd37-8a4a-4703-9114-ca13cac40108" (UID: "a5a9fd37-8a4a-4703-9114-ca13cac40108"). InnerVolumeSpecName "kube-api-access-h5wr4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:27:34.228688 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:34.228634 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:27:34.242718 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:34.242692 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h5wr4\" (UniqueName: \"kubernetes.io/projected/a5a9fd37-8a4a-4703-9114-ca13cac40108-kube-api-access-h5wr4\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:27:34.242718 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:34.242718 5057 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5a9fd37-8a4a-4703-9114-ca13cac40108-maas-api-xks-ca\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:27:34.352389 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:34.352351 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:27:34.616051 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:34.616012 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:27:34.836098 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:34.836068 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcc10992-4pss8" Oct 29 23:27:34.836270 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:34.836099 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcc10992-4pss8" event={"ID":"a5a9fd37-8a4a-4703-9114-ca13cac40108","Type":"ContainerDied","Data":"5837602bcfeb1756b7df340fbbb9d504f3026dc398e55e526f8950748a2dbad4"} Oct 29 23:27:34.836270 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:34.836150 5057 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5837602bcfeb1756b7df340fbbb9d504f3026dc398e55e526f8950748a2dbad4" Oct 29 23:27:34.910592 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:34.910497 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:27:36.117908 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:36.117871 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:27:36.230365 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:36.230299 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:27:36.627081 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:36.626622 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:27:36.696580 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:36.696499 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w2-b0c3c3/rate-test-model-b0c3c3-kserve-5b45fbb77-6jzx9"] Oct 29 23:27:36.728581 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:36.728547 5057 cpu_manager.go:414] "containerMap: removing container" podUID="a5a9fd37-8a4a-4703-9114-ca13cac40108" containerName="revoke-keys" Oct 29 23:27:36.728581 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:36.728581 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5a9fd37-8a4a-4703-9114-ca13cac40108" containerName="revoke-keys" Oct 29 23:27:36.728906 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:36.728887 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5a9fd37-8a4a-4703-9114-ca13cac40108" containerName="revoke-keys" Oct 29 23:27:36.734211 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:36.734188 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-b0c3c3/rate-test-model-b0c3c3-kserve-5b45fbb77-6jzx9"] Oct 29 23:27:36.734396 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:36.734270 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-b0c3c3/rate-test-model-b0c3c3-kserve-5b45fbb77-6jzx9" Oct 29 23:27:36.744745 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:36.744723 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:27:36.748681 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:36.748649 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-b0c3c3\"/\"openshift-service-ca.crt\"" Oct 29 23:27:36.760385 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:36.760360 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-b0c3c3\"/\"default-dockercfg-ksrdt\"" Oct 29 23:27:36.760502 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:36.760396 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-b0c3c3\"/\"kube-root-ca.crt\"" Oct 29 23:27:36.760502 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:36.760445 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-b0c3c3\"/\"rate-test-model-b0c3c3-kserve-self-signed-certs\"" Oct 29 23:27:36.766526 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:36.766506 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-dcacb9-openshift-default-54b777975-58gd9" Oct 29 23:27:36.823531 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:36.823480 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/84dd8e63-57ac-43db-9c6e-63725aa2e6a5-dshm\") pod \"rate-test-model-b0c3c3-kserve-5b45fbb77-6jzx9\" (UID: \"84dd8e63-57ac-43db-9c6e-63725aa2e6a5\") " pod="ai-tenant-e2e-shared-a-w2-b0c3c3/rate-test-model-b0c3c3-kserve-5b45fbb77-6jzx9" Oct 29 23:27:36.823726 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:36.823659 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzw5m\" (UniqueName: \"kubernetes.io/projected/84dd8e63-57ac-43db-9c6e-63725aa2e6a5-kube-api-access-gzw5m\") pod \"rate-test-model-b0c3c3-kserve-5b45fbb77-6jzx9\" (UID: \"84dd8e63-57ac-43db-9c6e-63725aa2e6a5\") " pod="ai-tenant-e2e-shared-a-w2-b0c3c3/rate-test-model-b0c3c3-kserve-5b45fbb77-6jzx9" Oct 29 23:27:36.823802 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:36.823727 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/84dd8e63-57ac-43db-9c6e-63725aa2e6a5-home\") pod \"rate-test-model-b0c3c3-kserve-5b45fbb77-6jzx9\" (UID: \"84dd8e63-57ac-43db-9c6e-63725aa2e6a5\") " pod="ai-tenant-e2e-shared-a-w2-b0c3c3/rate-test-model-b0c3c3-kserve-5b45fbb77-6jzx9" Oct 29 23:27:36.823802 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:36.823781 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/84dd8e63-57ac-43db-9c6e-63725aa2e6a5-tmp-dir\") pod \"rate-test-model-b0c3c3-kserve-5b45fbb77-6jzx9\" (UID: \"84dd8e63-57ac-43db-9c6e-63725aa2e6a5\") " pod="ai-tenant-e2e-shared-a-w2-b0c3c3/rate-test-model-b0c3c3-kserve-5b45fbb77-6jzx9" Oct 29 23:27:36.823921 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:36.823810 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/84dd8e63-57ac-43db-9c6e-63725aa2e6a5-tls-certs\") pod \"rate-test-model-b0c3c3-kserve-5b45fbb77-6jzx9\" (UID: \"84dd8e63-57ac-43db-9c6e-63725aa2e6a5\") " pod="ai-tenant-e2e-shared-a-w2-b0c3c3/rate-test-model-b0c3c3-kserve-5b45fbb77-6jzx9" Oct 29 23:27:36.823921 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:36.823886 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/84dd8e63-57ac-43db-9c6e-63725aa2e6a5-model-cache\") pod \"rate-test-model-b0c3c3-kserve-5b45fbb77-6jzx9\" (UID: \"84dd8e63-57ac-43db-9c6e-63725aa2e6a5\") " pod="ai-tenant-e2e-shared-a-w2-b0c3c3/rate-test-model-b0c3c3-kserve-5b45fbb77-6jzx9" Oct 29 23:27:36.862749 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:36.862715 5057 generic.go:356] "Generic (PLEG): container finished" podID="3c8c58af-2d80-4b3e-901f-a0a65338b5d7" containerID="7b92baa13c881e9839e072042cbcb7a8ca8d9c87ac1cc91dccad45334fe527d5" exitCode=0 Oct 29 23:27:36.862896 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:36.862773 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-dcacb9-openshift-default-54b777975-58gd9" Oct 29 23:27:36.862896 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:36.862801 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-dcacb9-openshift-default-54b777975-58gd9" event={"ID":"3c8c58af-2d80-4b3e-901f-a0a65338b5d7","Type":"ContainerDied","Data":"7b92baa13c881e9839e072042cbcb7a8ca8d9c87ac1cc91dccad45334fe527d5"} Oct 29 23:27:36.862896 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:36.862856 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-dcacb9-openshift-default-54b777975-58gd9" event={"ID":"3c8c58af-2d80-4b3e-901f-a0a65338b5d7","Type":"ContainerDied","Data":"87c7f93616689bf37fb669f4724805448037cac4b53976d90f2dc8ea1b3a2c46"} Oct 29 23:27:36.862896 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:36.862879 5057 scope.go:122] "RemoveContainer" containerID="7b92baa13c881e9839e072042cbcb7a8ca8d9c87ac1cc91dccad45334fe527d5" Oct 29 23:27:36.880481 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:36.880454 5057 scope.go:122] "RemoveContainer" containerID="7b92baa13c881e9839e072042cbcb7a8ca8d9c87ac1cc91dccad45334fe527d5" Oct 29 23:27:36.880822 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:27:36.880789 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b92baa13c881e9839e072042cbcb7a8ca8d9c87ac1cc91dccad45334fe527d5\": container with ID starting with 7b92baa13c881e9839e072042cbcb7a8ca8d9c87ac1cc91dccad45334fe527d5 not found: ID does not exist" containerID="7b92baa13c881e9839e072042cbcb7a8ca8d9c87ac1cc91dccad45334fe527d5" Oct 29 23:27:36.880932 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:36.880826 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b92baa13c881e9839e072042cbcb7a8ca8d9c87ac1cc91dccad45334fe527d5"} err="failed to get container status \"7b92baa13c881e9839e072042cbcb7a8ca8d9c87ac1cc91dccad45334fe527d5\": rpc error: code = NotFound desc = could not find container \"7b92baa13c881e9839e072042cbcb7a8ca8d9c87ac1cc91dccad45334fe527d5\": container with ID starting with 7b92baa13c881e9839e072042cbcb7a8ca8d9c87ac1cc91dccad45334fe527d5 not found: ID does not exist" Oct 29 23:27:36.965940 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:36.965897 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c8c58af-2d80-4b3e-901f-a0a65338b5d7-istio-token\" (UniqueName: \"kubernetes.io/projected/3c8c58af-2d80-4b3e-901f-a0a65338b5d7-istio-token\") pod \"3c8c58af-2d80-4b3e-901f-a0a65338b5d7\" (UID: \"3c8c58af-2d80-4b3e-901f-a0a65338b5d7\") " Oct 29 23:27:36.966163 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:36.965975 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c8c58af-2d80-4b3e-901f-a0a65338b5d7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c8c58af-2d80-4b3e-901f-a0a65338b5d7-istio-data\") pod \"3c8c58af-2d80-4b3e-901f-a0a65338b5d7\" (UID: \"3c8c58af-2d80-4b3e-901f-a0a65338b5d7\") " Oct 29 23:27:36.966163 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:36.966025 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c8c58af-2d80-4b3e-901f-a0a65338b5d7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c8c58af-2d80-4b3e-901f-a0a65338b5d7-workload-socket\") pod \"3c8c58af-2d80-4b3e-901f-a0a65338b5d7\" (UID: \"3c8c58af-2d80-4b3e-901f-a0a65338b5d7\") " Oct 29 23:27:36.966294 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:36.966199 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3c8c58af-2d80-4b3e-901f-a0a65338b5d7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c8c58af-2d80-4b3e-901f-a0a65338b5d7-istio-podinfo\") pod \"3c8c58af-2d80-4b3e-901f-a0a65338b5d7\" (UID: \"3c8c58af-2d80-4b3e-901f-a0a65338b5d7\") " Oct 29 23:27:36.966294 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:36.966243 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c8c58af-2d80-4b3e-901f-a0a65338b5d7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c8c58af-2d80-4b3e-901f-a0a65338b5d7-istiod-ca-cert\") pod \"3c8c58af-2d80-4b3e-901f-a0a65338b5d7\" (UID: \"3c8c58af-2d80-4b3e-901f-a0a65338b5d7\") " Oct 29 23:27:36.966437 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:36.966416 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c8c58af-2d80-4b3e-901f-a0a65338b5d7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c8c58af-2d80-4b3e-901f-a0a65338b5d7-workload-certs\") pod \"3c8c58af-2d80-4b3e-901f-a0a65338b5d7\" (UID: \"3c8c58af-2d80-4b3e-901f-a0a65338b5d7\") " Oct 29 23:27:36.966498 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:36.966454 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c8c58af-2d80-4b3e-901f-a0a65338b5d7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c8c58af-2d80-4b3e-901f-a0a65338b5d7-istio-envoy\") pod \"3c8c58af-2d80-4b3e-901f-a0a65338b5d7\" (UID: \"3c8c58af-2d80-4b3e-901f-a0a65338b5d7\") " Oct 29 23:27:36.966567 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:36.966550 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c8c58af-2d80-4b3e-901f-a0a65338b5d7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c8c58af-2d80-4b3e-901f-a0a65338b5d7-credential-socket\") pod \"3c8c58af-2d80-4b3e-901f-a0a65338b5d7\" (UID: \"3c8c58af-2d80-4b3e-901f-a0a65338b5d7\") " Oct 29 23:27:36.966631 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:36.966593 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c8c58af-2d80-4b3e-901f-a0a65338b5d7-kube-api-access-4z9hn\" (UniqueName: \"kubernetes.io/projected/3c8c58af-2d80-4b3e-901f-a0a65338b5d7-kube-api-access-4z9hn\") pod \"3c8c58af-2d80-4b3e-901f-a0a65338b5d7\" (UID: \"3c8c58af-2d80-4b3e-901f-a0a65338b5d7\") " Oct 29 23:27:36.967202 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:36.967176 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/84dd8e63-57ac-43db-9c6e-63725aa2e6a5-home\") pod \"rate-test-model-b0c3c3-kserve-5b45fbb77-6jzx9\" (UID: \"84dd8e63-57ac-43db-9c6e-63725aa2e6a5\") " pod="ai-tenant-e2e-shared-a-w2-b0c3c3/rate-test-model-b0c3c3-kserve-5b45fbb77-6jzx9" Oct 29 23:27:36.967280 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:36.967264 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/84dd8e63-57ac-43db-9c6e-63725aa2e6a5-tmp-dir\") pod \"rate-test-model-b0c3c3-kserve-5b45fbb77-6jzx9\" (UID: \"84dd8e63-57ac-43db-9c6e-63725aa2e6a5\") " pod="ai-tenant-e2e-shared-a-w2-b0c3c3/rate-test-model-b0c3c3-kserve-5b45fbb77-6jzx9" Oct 29 23:27:36.967389 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:36.967300 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/84dd8e63-57ac-43db-9c6e-63725aa2e6a5-tls-certs\") pod \"rate-test-model-b0c3c3-kserve-5b45fbb77-6jzx9\" (UID: \"84dd8e63-57ac-43db-9c6e-63725aa2e6a5\") " pod="ai-tenant-e2e-shared-a-w2-b0c3c3/rate-test-model-b0c3c3-kserve-5b45fbb77-6jzx9" Oct 29 23:27:36.967458 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:36.967406 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/84dd8e63-57ac-43db-9c6e-63725aa2e6a5-model-cache\") pod \"rate-test-model-b0c3c3-kserve-5b45fbb77-6jzx9\" (UID: \"84dd8e63-57ac-43db-9c6e-63725aa2e6a5\") " pod="ai-tenant-e2e-shared-a-w2-b0c3c3/rate-test-model-b0c3c3-kserve-5b45fbb77-6jzx9" Oct 29 23:27:36.967620 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:36.967600 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/84dd8e63-57ac-43db-9c6e-63725aa2e6a5-dshm\") pod \"rate-test-model-b0c3c3-kserve-5b45fbb77-6jzx9\" (UID: \"84dd8e63-57ac-43db-9c6e-63725aa2e6a5\") " pod="ai-tenant-e2e-shared-a-w2-b0c3c3/rate-test-model-b0c3c3-kserve-5b45fbb77-6jzx9" Oct 29 23:27:36.967888 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:36.967868 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzw5m\" (UniqueName: \"kubernetes.io/projected/84dd8e63-57ac-43db-9c6e-63725aa2e6a5-kube-api-access-gzw5m\") pod \"rate-test-model-b0c3c3-kserve-5b45fbb77-6jzx9\" (UID: \"84dd8e63-57ac-43db-9c6e-63725aa2e6a5\") " pod="ai-tenant-e2e-shared-a-w2-b0c3c3/rate-test-model-b0c3c3-kserve-5b45fbb77-6jzx9" Oct 29 23:27:36.973493 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:36.973461 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c8c58af-2d80-4b3e-901f-a0a65338b5d7-credential-socket" pod "3c8c58af-2d80-4b3e-901f-a0a65338b5d7" (UID: "3c8c58af-2d80-4b3e-901f-a0a65338b5d7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:27:36.973669 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:36.973576 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c8c58af-2d80-4b3e-901f-a0a65338b5d7-workload-socket" pod "3c8c58af-2d80-4b3e-901f-a0a65338b5d7" (UID: "3c8c58af-2d80-4b3e-901f-a0a65338b5d7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:27:36.973740 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:36.973711 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c8c58af-2d80-4b3e-901f-a0a65338b5d7-workload-certs" pod "3c8c58af-2d80-4b3e-901f-a0a65338b5d7" (UID: "3c8c58af-2d80-4b3e-901f-a0a65338b5d7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:27:36.977628 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:36.977552 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c8c58af-2d80-4b3e-901f-a0a65338b5d7-istio-data" pod "3c8c58af-2d80-4b3e-901f-a0a65338b5d7" (UID: "3c8c58af-2d80-4b3e-901f-a0a65338b5d7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:27:36.979250 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:36.979227 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/84dd8e63-57ac-43db-9c6e-63725aa2e6a5-home\") pod \"rate-test-model-b0c3c3-kserve-5b45fbb77-6jzx9\" (UID: \"84dd8e63-57ac-43db-9c6e-63725aa2e6a5\") " pod="ai-tenant-e2e-shared-a-w2-b0c3c3/rate-test-model-b0c3c3-kserve-5b45fbb77-6jzx9" Oct 29 23:27:36.979892 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:36.979873 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/84dd8e63-57ac-43db-9c6e-63725aa2e6a5-tmp-dir\") pod \"rate-test-model-b0c3c3-kserve-5b45fbb77-6jzx9\" (UID: \"84dd8e63-57ac-43db-9c6e-63725aa2e6a5\") " pod="ai-tenant-e2e-shared-a-w2-b0c3c3/rate-test-model-b0c3c3-kserve-5b45fbb77-6jzx9" Oct 29 23:27:36.981483 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:36.981301 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/84dd8e63-57ac-43db-9c6e-63725aa2e6a5-model-cache\") pod \"rate-test-model-b0c3c3-kserve-5b45fbb77-6jzx9\" (UID: \"84dd8e63-57ac-43db-9c6e-63725aa2e6a5\") " pod="ai-tenant-e2e-shared-a-w2-b0c3c3/rate-test-model-b0c3c3-kserve-5b45fbb77-6jzx9" Oct 29 23:27:36.984481 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:36.981752 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c8c58af-2d80-4b3e-901f-a0a65338b5d7-istiod-ca-cert" pod "3c8c58af-2d80-4b3e-901f-a0a65338b5d7" (UID: "3c8c58af-2d80-4b3e-901f-a0a65338b5d7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:27:36.984481 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:36.982739 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c8c58af-2d80-4b3e-901f-a0a65338b5d7-istio-token" pod "3c8c58af-2d80-4b3e-901f-a0a65338b5d7" (UID: "3c8c58af-2d80-4b3e-901f-a0a65338b5d7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:27:36.984651 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:36.984528 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c8c58af-2d80-4b3e-901f-a0a65338b5d7-istio-envoy" pod "3c8c58af-2d80-4b3e-901f-a0a65338b5d7" (UID: "3c8c58af-2d80-4b3e-901f-a0a65338b5d7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:27:36.984651 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:36.984564 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c8c58af-2d80-4b3e-901f-a0a65338b5d7-kube-api-access-4z9hn" pod "3c8c58af-2d80-4b3e-901f-a0a65338b5d7" (UID: "3c8c58af-2d80-4b3e-901f-a0a65338b5d7"). InnerVolumeSpecName "kube-api-access-4z9hn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:27:36.991837 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:36.991273 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/84dd8e63-57ac-43db-9c6e-63725aa2e6a5-dshm\") pod \"rate-test-model-b0c3c3-kserve-5b45fbb77-6jzx9\" (UID: \"84dd8e63-57ac-43db-9c6e-63725aa2e6a5\") " pod="ai-tenant-e2e-shared-a-w2-b0c3c3/rate-test-model-b0c3c3-kserve-5b45fbb77-6jzx9" Oct 29 23:27:36.991837 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:36.991403 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3c8c58af-2d80-4b3e-901f-a0a65338b5d7-istio-podinfo" pod "3c8c58af-2d80-4b3e-901f-a0a65338b5d7" (UID: "3c8c58af-2d80-4b3e-901f-a0a65338b5d7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:27:36.991837 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:36.991444 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/84dd8e63-57ac-43db-9c6e-63725aa2e6a5-tls-certs\") pod \"rate-test-model-b0c3c3-kserve-5b45fbb77-6jzx9\" (UID: \"84dd8e63-57ac-43db-9c6e-63725aa2e6a5\") " pod="ai-tenant-e2e-shared-a-w2-b0c3c3/rate-test-model-b0c3c3-kserve-5b45fbb77-6jzx9" Oct 29 23:27:36.991837 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:36.991795 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzw5m\" (UniqueName: \"kubernetes.io/projected/84dd8e63-57ac-43db-9c6e-63725aa2e6a5-kube-api-access-gzw5m\") pod \"rate-test-model-b0c3c3-kserve-5b45fbb77-6jzx9\" (UID: \"84dd8e63-57ac-43db-9c6e-63725aa2e6a5\") " pod="ai-tenant-e2e-shared-a-w2-b0c3c3/rate-test-model-b0c3c3-kserve-5b45fbb77-6jzx9" Oct 29 23:27:37.070012 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:37.069975 5057 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c8c58af-2d80-4b3e-901f-a0a65338b5d7-credential-socket\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:27:37.070012 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:37.070005 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4z9hn\" (UniqueName: \"kubernetes.io/projected/3c8c58af-2d80-4b3e-901f-a0a65338b5d7-kube-api-access-4z9hn\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:27:37.070012 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:37.070016 5057 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c8c58af-2d80-4b3e-901f-a0a65338b5d7-istio-token\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:27:37.070263 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:37.070024 5057 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c8c58af-2d80-4b3e-901f-a0a65338b5d7-istio-data\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:27:37.070263 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:37.070032 5057 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c8c58af-2d80-4b3e-901f-a0a65338b5d7-workload-socket\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:27:37.070263 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:37.070040 5057 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c8c58af-2d80-4b3e-901f-a0a65338b5d7-istio-podinfo\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:27:37.070263 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:37.070049 5057 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c8c58af-2d80-4b3e-901f-a0a65338b5d7-istiod-ca-cert\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:27:37.070263 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:37.070057 5057 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c8c58af-2d80-4b3e-901f-a0a65338b5d7-workload-certs\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:27:37.070263 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:37.070065 5057 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c8c58af-2d80-4b3e-901f-a0a65338b5d7-istio-envoy\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:27:37.087418 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:37.087390 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-b0c3c3/rate-test-model-b0c3c3-kserve-5b45fbb77-6jzx9" Oct 29 23:27:37.225043 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:37.224967 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-dcacb9-openshift-default-54b777975-58gd9"] Oct 29 23:27:37.230193 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:37.230152 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-dcacb9-openshift-default-54b777975-58gd9"] Oct 29 23:27:37.321454 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:37.321400 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-c880c7-5d864f4568-xqcv8" podUID="7b882cd5-e248-46d5-83d5-cdc7f3db72ba" containerName="payload-processing" containerID="cri-o://83d27099d1380f07d976647486fafad63b526484eb1060021d0bfdef3fed710f" gracePeriod=25 Oct 29 23:27:37.336051 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:37.336018 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c8c58af-2d80-4b3e-901f-a0a65338b5d7" path="/var/lib/kubelet/pods/3c8c58af-2d80-4b3e-901f-a0a65338b5d7/volumes" Oct 29 23:27:37.368550 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:37.368510 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-c880c7-58cdf8ddd7-fj789" podUID="babbd117-8898-4d39-9f04-e4d165379a0a" containerName="payload-pre-processing" containerID="cri-o://27cad8308fa7ddaf60e17e1650ac4b36bf82ec7345002409746a03c202cda7a4" gracePeriod=25 Oct 29 23:27:37.404692 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:37.404665 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-b0c3c3/rate-test-model-b0c3c3-kserve-5b45fbb77-6jzx9"] Oct 29 23:27:37.433385 ip-10-0-140-74 kubenswrapper[5057]: W1029 23:27:37.423227 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod84dd8e63_57ac_43db_9c6e_63725aa2e6a5.slice/crio-ed042a51cd472ea613b240a70f1b32137b65950dbe694f3d8677881e6b0fbdd5 WatchSource:0}: Error finding container ed042a51cd472ea613b240a70f1b32137b65950dbe694f3d8677881e6b0fbdd5: Status 404 returned error can't find the container with id ed042a51cd472ea613b240a70f1b32137b65950dbe694f3d8677881e6b0fbdd5 Oct 29 23:27:37.706369 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:37.704390 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-c880c7-5d864f4568-xqcv8" Oct 29 23:27:37.748401 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:37.746499 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-c880c7-58cdf8ddd7-fj789" Oct 29 23:27:37.784994 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:37.784951 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/babbd117-8898-4d39-9f04-e4d165379a0a-kube-api-access-sm9bg\" (UniqueName: \"kubernetes.io/projected/babbd117-8898-4d39-9f04-e4d165379a0a-kube-api-access-sm9bg\") pod \"babbd117-8898-4d39-9f04-e4d165379a0a\" (UID: \"babbd117-8898-4d39-9f04-e4d165379a0a\") " Oct 29 23:27:37.785173 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:37.785067 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/babbd117-8898-4d39-9f04-e4d165379a0a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/babbd117-8898-4d39-9f04-e4d165379a0a-plugins-config-volume\") pod \"babbd117-8898-4d39-9f04-e4d165379a0a\" (UID: \"babbd117-8898-4d39-9f04-e4d165379a0a\") " Oct 29 23:27:37.785173 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:37.785144 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b882cd5-e248-46d5-83d5-cdc7f3db72ba-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b882cd5-e248-46d5-83d5-cdc7f3db72ba-plugins-config-volume\") pod \"7b882cd5-e248-46d5-83d5-cdc7f3db72ba\" (UID: \"7b882cd5-e248-46d5-83d5-cdc7f3db72ba\") " Oct 29 23:27:37.785312 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:37.785185 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b882cd5-e248-46d5-83d5-cdc7f3db72ba-kube-api-access-fgvsh\" (UniqueName: \"kubernetes.io/projected/7b882cd5-e248-46d5-83d5-cdc7f3db72ba-kube-api-access-fgvsh\") pod \"7b882cd5-e248-46d5-83d5-cdc7f3db72ba\" (UID: \"7b882cd5-e248-46d5-83d5-cdc7f3db72ba\") " Oct 29 23:27:37.786729 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:37.786704 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b882cd5-e248-46d5-83d5-cdc7f3db72ba-plugins-config-volume" pod "7b882cd5-e248-46d5-83d5-cdc7f3db72ba" (UID: "7b882cd5-e248-46d5-83d5-cdc7f3db72ba"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:27:37.786948 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:37.786910 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/babbd117-8898-4d39-9f04-e4d165379a0a-plugins-config-volume" pod "babbd117-8898-4d39-9f04-e4d165379a0a" (UID: "babbd117-8898-4d39-9f04-e4d165379a0a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:27:37.809368 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:37.806523 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b882cd5-e248-46d5-83d5-cdc7f3db72ba-kube-api-access-fgvsh" pod "7b882cd5-e248-46d5-83d5-cdc7f3db72ba" (UID: "7b882cd5-e248-46d5-83d5-cdc7f3db72ba"). InnerVolumeSpecName "kube-api-access-fgvsh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:27:37.809368 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:37.807482 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/babbd117-8898-4d39-9f04-e4d165379a0a-kube-api-access-sm9bg" pod "babbd117-8898-4d39-9f04-e4d165379a0a" (UID: "babbd117-8898-4d39-9f04-e4d165379a0a"). InnerVolumeSpecName "kube-api-access-sm9bg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:27:37.874187 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:37.874154 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-b0c3c3/rate-test-model-b0c3c3-kserve-5b45fbb77-6jzx9" event={"ID":"84dd8e63-57ac-43db-9c6e-63725aa2e6a5","Type":"ContainerStarted","Data":"b26ccc84f4f7356de821f8e20eb1b99344d31b4e48b3d350f083584ac3fa54fd"} Oct 29 23:27:37.874403 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:37.874197 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-b0c3c3/rate-test-model-b0c3c3-kserve-5b45fbb77-6jzx9" event={"ID":"84dd8e63-57ac-43db-9c6e-63725aa2e6a5","Type":"ContainerStarted","Data":"ed042a51cd472ea613b240a70f1b32137b65950dbe694f3d8677881e6b0fbdd5"} Oct 29 23:27:37.875633 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:37.875603 5057 generic.go:356] "Generic (PLEG): container finished" podID="babbd117-8898-4d39-9f04-e4d165379a0a" containerID="27cad8308fa7ddaf60e17e1650ac4b36bf82ec7345002409746a03c202cda7a4" exitCode=0 Oct 29 23:27:37.875764 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:37.875649 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-c880c7-58cdf8ddd7-fj789" Oct 29 23:27:37.875764 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:37.875684 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-c880c7-58cdf8ddd7-fj789" event={"ID":"babbd117-8898-4d39-9f04-e4d165379a0a","Type":"ContainerDied","Data":"27cad8308fa7ddaf60e17e1650ac4b36bf82ec7345002409746a03c202cda7a4"} Oct 29 23:27:37.875764 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:37.875725 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-c880c7-58cdf8ddd7-fj789" event={"ID":"babbd117-8898-4d39-9f04-e4d165379a0a","Type":"ContainerDied","Data":"8990e4b14d44e2cbd1a6fceffd122d53c63271a286a0f5b91a06ac4cc5deb87a"} Oct 29 23:27:37.875764 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:37.875748 5057 scope.go:122] "RemoveContainer" containerID="27cad8308fa7ddaf60e17e1650ac4b36bf82ec7345002409746a03c202cda7a4" Oct 29 23:27:37.877530 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:37.877504 5057 generic.go:356] "Generic (PLEG): container finished" podID="7b882cd5-e248-46d5-83d5-cdc7f3db72ba" containerID="83d27099d1380f07d976647486fafad63b526484eb1060021d0bfdef3fed710f" exitCode=0 Oct 29 23:27:37.877662 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:37.877556 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-c880c7-5d864f4568-xqcv8" Oct 29 23:27:37.877662 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:37.877587 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-c880c7-5d864f4568-xqcv8" event={"ID":"7b882cd5-e248-46d5-83d5-cdc7f3db72ba","Type":"ContainerDied","Data":"83d27099d1380f07d976647486fafad63b526484eb1060021d0bfdef3fed710f"} Oct 29 23:27:37.877662 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:37.877627 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-c880c7-5d864f4568-xqcv8" event={"ID":"7b882cd5-e248-46d5-83d5-cdc7f3db72ba","Type":"ContainerDied","Data":"88b0efcc58d58193a8c904bbd4263a228dfe97be234b9b67987d8e7f46ac4166"} Oct 29 23:27:37.888364 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:37.888322 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sm9bg\" (UniqueName: \"kubernetes.io/projected/babbd117-8898-4d39-9f04-e4d165379a0a-kube-api-access-sm9bg\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:27:37.888460 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:37.888369 5057 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/babbd117-8898-4d39-9f04-e4d165379a0a-plugins-config-volume\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:27:37.888460 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:37.888385 5057 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b882cd5-e248-46d5-83d5-cdc7f3db72ba-plugins-config-volume\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:27:37.888460 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:37.888399 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fgvsh\" (UniqueName: \"kubernetes.io/projected/7b882cd5-e248-46d5-83d5-cdc7f3db72ba-kube-api-access-fgvsh\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:27:37.890064 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:37.889946 5057 scope.go:122] "RemoveContainer" containerID="27cad8308fa7ddaf60e17e1650ac4b36bf82ec7345002409746a03c202cda7a4" Oct 29 23:27:37.890381 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:27:37.890303 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27cad8308fa7ddaf60e17e1650ac4b36bf82ec7345002409746a03c202cda7a4\": container with ID starting with 27cad8308fa7ddaf60e17e1650ac4b36bf82ec7345002409746a03c202cda7a4 not found: ID does not exist" containerID="27cad8308fa7ddaf60e17e1650ac4b36bf82ec7345002409746a03c202cda7a4" Oct 29 23:27:37.890472 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:37.890377 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27cad8308fa7ddaf60e17e1650ac4b36bf82ec7345002409746a03c202cda7a4"} err="failed to get container status \"27cad8308fa7ddaf60e17e1650ac4b36bf82ec7345002409746a03c202cda7a4\": rpc error: code = NotFound desc = could not find container \"27cad8308fa7ddaf60e17e1650ac4b36bf82ec7345002409746a03c202cda7a4\": container with ID starting with 27cad8308fa7ddaf60e17e1650ac4b36bf82ec7345002409746a03c202cda7a4 not found: ID does not exist" Oct 29 23:27:37.890472 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:37.890404 5057 scope.go:122] "RemoveContainer" containerID="83d27099d1380f07d976647486fafad63b526484eb1060021d0bfdef3fed710f" Oct 29 23:27:37.903468 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:37.903442 5057 scope.go:122] "RemoveContainer" containerID="83d27099d1380f07d976647486fafad63b526484eb1060021d0bfdef3fed710f" Oct 29 23:27:37.903787 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:27:37.903765 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83d27099d1380f07d976647486fafad63b526484eb1060021d0bfdef3fed710f\": container with ID starting with 83d27099d1380f07d976647486fafad63b526484eb1060021d0bfdef3fed710f not found: ID does not exist" containerID="83d27099d1380f07d976647486fafad63b526484eb1060021d0bfdef3fed710f" Oct 29 23:27:37.903859 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:37.903793 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83d27099d1380f07d976647486fafad63b526484eb1060021d0bfdef3fed710f"} err="failed to get container status \"83d27099d1380f07d976647486fafad63b526484eb1060021d0bfdef3fed710f\": rpc error: code = NotFound desc = could not find container \"83d27099d1380f07d976647486fafad63b526484eb1060021d0bfdef3fed710f\": container with ID starting with 83d27099d1380f07d976647486fafad63b526484eb1060021d0bfdef3fed710f not found: ID does not exist" Oct 29 23:27:37.921789 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:37.921741 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w2-b0c3c3/rate-test-model-b0c3c3-kserve-5b45fbb77-6jzx9" podStartSLOduration=1.921724199 podStartE2EDuration="1.921724199s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:27:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:27:37.91448096 +0000 UTC m=+3238.697694387" watchObservedRunningTime="2026-10-29 23:27:37.921724199 +0000 UTC m=+3238.704937625" Oct 29 23:27:37.953601 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:37.953572 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-c880c7-5d864f4568-xqcv8"] Oct 29 23:27:37.958590 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:37.958559 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-c880c7-5d864f4568-xqcv8"] Oct 29 23:27:37.992218 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:37.992141 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-c880c7-58cdf8ddd7-fj789"] Oct 29 23:27:37.996704 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:37.996679 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-c880c7-58cdf8ddd7-fj789"] Oct 29 23:27:38.088020 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:38.087987 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w2-b0c3c3/rate-test-model-b0c3c3-kserve-5b45fbb77-6jzx9" Oct 29 23:27:38.217840 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:38.217803 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:27:39.331285 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:39.331252 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b882cd5-e248-46d5-83d5-cdc7f3db72ba" path="/var/lib/kubelet/pods/7b882cd5-e248-46d5-83d5-cdc7f3db72ba/volumes" Oct 29 23:27:39.331740 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:39.331723 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="babbd117-8898-4d39-9f04-e4d165379a0a" path="/var/lib/kubelet/pods/babbd117-8898-4d39-9f04-e4d165379a0a/volumes" Oct 29 23:27:39.872236 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:39.872204 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:27:40.415981 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:40.415923 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:27:41.466385 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:41.466322 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:27:42.485629 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:42.485594 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-cc6328-6fd658c9b8-vf7tb"] Oct 29 23:27:42.523071 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:42.523038 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-cc6328-64879d8c8c-62gfk"] Oct 29 23:27:42.638566 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:42.638499 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:27:42.760579 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:42.760498 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:27:43.251886 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:43.246608 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:27:44.020637 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:44.020606 5057 scope.go:122] "RemoveContainer" containerID="d66683bb06b93b6757593cd6e707ffd3602aeb34fcbc66e6876ad26bea6d04a0" Oct 29 23:27:44.606878 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:44.606839 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:27:45.275919 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:45.275890 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:27:45.477095 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:45.477057 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:27:45.575109 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:45.575038 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-32717a-jsm8v"] Oct 29 23:27:45.584935 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:45.584905 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-32717a-jsm8v"] Oct 29 23:27:45.593991 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:45.593962 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:27:45.948483 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:45.948453 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:27:46.080147 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:46.080110 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:27:46.082390 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:46.082362 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-3e5b21-7whs8"] Oct 29 23:27:46.093421 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:46.093390 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-3e5b21-7whs8"] Oct 29 23:27:46.190216 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:46.190179 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:27:46.581444 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:46.581409 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:27:46.951233 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:46.951201 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:27:47.078969 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:47.078933 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:27:47.087904 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:47.087876 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w2-b0c3c3/rate-test-model-b0c3c3-kserve-5b45fbb77-6jzx9" Oct 29 23:27:47.101017 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:47.100995 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w2-b0c3c3/rate-test-model-b0c3c3-kserve-5b45fbb77-6jzx9" Oct 29 23:27:47.332858 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:47.332784 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b56f7c05-6607-4652-a92d-a55b5b21fcf0" path="/var/lib/kubelet/pods/b56f7c05-6607-4652-a92d-a55b5b21fcf0/volumes" Oct 29 23:27:47.333260 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:47.333245 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d9ed3669-6751-4a9c-8c44-ba9a7df4146a" path="/var/lib/kubelet/pods/d9ed3669-6751-4a9c-8c44-ba9a7df4146a/volumes" Oct 29 23:27:47.486403 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:47.486359 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-cc6328-6fd658c9b8-vf7tb" podUID="fb9f22eb-eef1-4b1d-8fec-3fa6a2555222" containerName="payload-processing" containerID="cri-o://c5ba1c3001ad84d2ecd235d7945e11feaad470f254b82f776d97a94a40408b4c" gracePeriod=25 Oct 29 23:27:47.523664 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:47.523611 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cc6328-64879d8c8c-62gfk" podUID="3eb0219c-cf7b-4f66-b484-a78a2a5514a1" containerName="payload-pre-processing" containerID="cri-o://adc446a0e96784a34afcefe74ac6ed13c71698783c9d1dd2d33149600489864d" gracePeriod=25 Oct 29 23:27:47.721903 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:47.721716 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:27:47.819414 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:47.819387 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-cc6328-6fd658c9b8-vf7tb" Oct 29 23:27:47.884439 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:47.884325 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb9f22eb-eef1-4b1d-8fec-3fa6a2555222-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb9f22eb-eef1-4b1d-8fec-3fa6a2555222-plugins-config-volume\") pod \"fb9f22eb-eef1-4b1d-8fec-3fa6a2555222\" (UID: \"fb9f22eb-eef1-4b1d-8fec-3fa6a2555222\") " Oct 29 23:27:47.884600 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:47.884518 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb9f22eb-eef1-4b1d-8fec-3fa6a2555222-kube-api-access-45shh\" (UniqueName: \"kubernetes.io/projected/fb9f22eb-eef1-4b1d-8fec-3fa6a2555222-kube-api-access-45shh\") pod \"fb9f22eb-eef1-4b1d-8fec-3fa6a2555222\" (UID: \"fb9f22eb-eef1-4b1d-8fec-3fa6a2555222\") " Oct 29 23:27:47.884764 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:47.884741 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb9f22eb-eef1-4b1d-8fec-3fa6a2555222-plugins-config-volume" pod "fb9f22eb-eef1-4b1d-8fec-3fa6a2555222" (UID: "fb9f22eb-eef1-4b1d-8fec-3fa6a2555222"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:27:47.885209 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:47.885191 5057 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb9f22eb-eef1-4b1d-8fec-3fa6a2555222-plugins-config-volume\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:27:47.886682 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:47.886657 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb9f22eb-eef1-4b1d-8fec-3fa6a2555222-kube-api-access-45shh" pod "fb9f22eb-eef1-4b1d-8fec-3fa6a2555222" (UID: "fb9f22eb-eef1-4b1d-8fec-3fa6a2555222"). InnerVolumeSpecName "kube-api-access-45shh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:27:47.986719 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:47.986677 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-45shh\" (UniqueName: \"kubernetes.io/projected/fb9f22eb-eef1-4b1d-8fec-3fa6a2555222-kube-api-access-45shh\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:27:47.992391 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:47.992194 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:27:47.992643 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:47.992617 5057 generic.go:356] "Generic (PLEG): container finished" podID="fb9f22eb-eef1-4b1d-8fec-3fa6a2555222" containerID="c5ba1c3001ad84d2ecd235d7945e11feaad470f254b82f776d97a94a40408b4c" exitCode=0 Oct 29 23:27:47.992720 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:47.992662 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-cc6328-6fd658c9b8-vf7tb" Oct 29 23:27:47.992720 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:47.992704 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-cc6328-6fd658c9b8-vf7tb" event={"ID":"fb9f22eb-eef1-4b1d-8fec-3fa6a2555222","Type":"ContainerDied","Data":"c5ba1c3001ad84d2ecd235d7945e11feaad470f254b82f776d97a94a40408b4c"} Oct 29 23:27:47.992829 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:47.992759 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-cc6328-6fd658c9b8-vf7tb" event={"ID":"fb9f22eb-eef1-4b1d-8fec-3fa6a2555222","Type":"ContainerDied","Data":"f464a7236ed44aa03557c37afaedfed1970188a4098754b95fe5f52226cba0c2"} Oct 29 23:27:47.992829 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:47.992786 5057 scope.go:122] "RemoveContainer" containerID="c5ba1c3001ad84d2ecd235d7945e11feaad470f254b82f776d97a94a40408b4c" Oct 29 23:27:48.006179 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:48.006158 5057 scope.go:122] "RemoveContainer" containerID="c5ba1c3001ad84d2ecd235d7945e11feaad470f254b82f776d97a94a40408b4c" Oct 29 23:27:48.007117 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:27:48.007085 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5ba1c3001ad84d2ecd235d7945e11feaad470f254b82f776d97a94a40408b4c\": container with ID starting with c5ba1c3001ad84d2ecd235d7945e11feaad470f254b82f776d97a94a40408b4c not found: ID does not exist" containerID="c5ba1c3001ad84d2ecd235d7945e11feaad470f254b82f776d97a94a40408b4c" Oct 29 23:27:48.007244 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:48.007147 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5ba1c3001ad84d2ecd235d7945e11feaad470f254b82f776d97a94a40408b4c"} err="failed to get container status \"c5ba1c3001ad84d2ecd235d7945e11feaad470f254b82f776d97a94a40408b4c\": rpc error: code = NotFound desc = could not find container \"c5ba1c3001ad84d2ecd235d7945e11feaad470f254b82f776d97a94a40408b4c\": container with ID starting with c5ba1c3001ad84d2ecd235d7945e11feaad470f254b82f776d97a94a40408b4c not found: ID does not exist" Oct 29 23:27:48.010282 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:48.010260 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w2-b0c3c3/rate-test-model-b0c3c3-kserve-5b45fbb77-6jzx9" Oct 29 23:27:48.066137 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:48.066102 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-cc6328-6fd658c9b8-vf7tb"] Oct 29 23:27:48.071048 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:48.071018 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-cc6328-6fd658c9b8-vf7tb"] Oct 29 23:27:48.126617 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:48.126587 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:27:49.030529 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:27:49.030485 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d2dnn" containerName="extractor" Oct 29 23:27:49.330885 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:49.330798 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb9f22eb-eef1-4b1d-8fec-3fa6a2555222" path="/var/lib/kubelet/pods/fb9f22eb-eef1-4b1d-8fec-3fa6a2555222/volumes" Oct 29 23:27:49.712178 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:49.712146 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:27:50.170972 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:50.170888 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cc6328-64879d8c8c-62gfk" podUID="3eb0219c-cf7b-4f66-b484-a78a2a5514a1" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:27:51.227717 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:51.225989 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v2tfm"] Oct 29 23:27:51.625795 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:51.625698 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:27:52.209890 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:52.209849 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:27:53.493354 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:53.492872 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:27:54.260878 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:54.260842 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:27:54.388801 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:54.388766 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:27:54.516528 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:54.516411 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:27:54.630846 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:54.630808 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:27:54.767380 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:54.767238 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-nxkfc"] Oct 29 23:27:54.767605 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:54.767572 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-nxkfc" podUID="e52f656e-cf07-4fa7-aed0-21d17554a880" containerName="istio-proxy" containerID="cri-o://0249b628f89b9ada1a6072e7ebce456108645a13908d336075140d3d8bf44761" gracePeriod=30 Oct 29 23:27:54.823515 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:54.823412 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:27:55.341393 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:55.341361 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-755453-cnmm7"] Oct 29 23:27:55.342212 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:55.342197 5057 cpu_manager.go:414] "containerMap: removing container" podUID="3c8c58af-2d80-4b3e-901f-a0a65338b5d7" containerName="istio-proxy" Oct 29 23:27:55.342272 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:55.342213 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c8c58af-2d80-4b3e-901f-a0a65338b5d7" containerName="istio-proxy" Oct 29 23:27:55.342272 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:55.342228 5057 cpu_manager.go:414] "containerMap: removing container" podUID="fb9f22eb-eef1-4b1d-8fec-3fa6a2555222" containerName="payload-processing" Oct 29 23:27:55.342272 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:55.342233 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb9f22eb-eef1-4b1d-8fec-3fa6a2555222" containerName="payload-processing" Oct 29 23:27:55.342272 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:55.342249 5057 cpu_manager.go:414] "containerMap: removing container" podUID="babbd117-8898-4d39-9f04-e4d165379a0a" containerName="payload-pre-processing" Oct 29 23:27:55.342272 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:55.342254 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="babbd117-8898-4d39-9f04-e4d165379a0a" containerName="payload-pre-processing" Oct 29 23:27:55.342442 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:55.342275 5057 cpu_manager.go:414] "containerMap: removing container" podUID="7b882cd5-e248-46d5-83d5-cdc7f3db72ba" containerName="payload-processing" Oct 29 23:27:55.342442 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:55.342281 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b882cd5-e248-46d5-83d5-cdc7f3db72ba" containerName="payload-processing" Oct 29 23:27:55.342442 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:55.342418 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c8c58af-2d80-4b3e-901f-a0a65338b5d7" containerName="istio-proxy" Oct 29 23:27:55.342442 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:55.342428 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb9f22eb-eef1-4b1d-8fec-3fa6a2555222" containerName="payload-processing" Oct 29 23:27:55.342442 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:55.342437 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="babbd117-8898-4d39-9f04-e4d165379a0a" containerName="payload-pre-processing" Oct 29 23:27:55.342612 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:55.342446 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b882cd5-e248-46d5-83d5-cdc7f3db72ba" containerName="payload-processing" Oct 29 23:27:55.348085 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:55.348066 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-755453-cnmm7" Oct 29 23:27:55.369090 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:55.369063 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-755453-cnmm7"] Oct 29 23:27:55.419382 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:55.411900 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dmx57\"" Oct 29 23:27:55.419382 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:55.411971 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:27:55.453247 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:55.453206 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7be7f1f-537b-4b1f-abd6-97727768c84c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-755453-cnmm7\" (UID: \"f7be7f1f-537b-4b1f-abd6-97727768c84c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-755453-cnmm7" Oct 29 23:27:55.453465 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:55.453382 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8sm9m\" (UniqueName: \"kubernetes.io/projected/f7be7f1f-537b-4b1f-abd6-97727768c84c-kube-api-access-8sm9m\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-755453-cnmm7\" (UID: \"f7be7f1f-537b-4b1f-abd6-97727768c84c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-755453-cnmm7" Oct 29 23:27:55.453538 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:55.453517 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7be7f1f-537b-4b1f-abd6-97727768c84c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-755453-cnmm7\" (UID: \"f7be7f1f-537b-4b1f-abd6-97727768c84c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-755453-cnmm7" Oct 29 23:27:55.508746 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:55.508707 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:27:55.555162 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:55.555123 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7be7f1f-537b-4b1f-abd6-97727768c84c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-755453-cnmm7\" (UID: \"f7be7f1f-537b-4b1f-abd6-97727768c84c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-755453-cnmm7" Oct 29 23:27:55.555668 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:55.555385 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7be7f1f-537b-4b1f-abd6-97727768c84c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-755453-cnmm7\" (UID: \"f7be7f1f-537b-4b1f-abd6-97727768c84c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-755453-cnmm7" Oct 29 23:27:55.555668 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:55.555505 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8sm9m\" (UniqueName: \"kubernetes.io/projected/f7be7f1f-537b-4b1f-abd6-97727768c84c-kube-api-access-8sm9m\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-755453-cnmm7\" (UID: \"f7be7f1f-537b-4b1f-abd6-97727768c84c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-755453-cnmm7" Oct 29 23:27:55.556375 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:55.556303 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7be7f1f-537b-4b1f-abd6-97727768c84c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-755453-cnmm7\" (UID: \"f7be7f1f-537b-4b1f-abd6-97727768c84c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-755453-cnmm7" Oct 29 23:27:55.565948 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:55.565920 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7be7f1f-537b-4b1f-abd6-97727768c84c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-755453-cnmm7\" (UID: \"f7be7f1f-537b-4b1f-abd6-97727768c84c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-755453-cnmm7" Oct 29 23:27:55.583355 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:55.580453 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8sm9m\" (UniqueName: \"kubernetes.io/projected/f7be7f1f-537b-4b1f-abd6-97727768c84c-kube-api-access-8sm9m\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-755453-cnmm7\" (UID: \"f7be7f1f-537b-4b1f-abd6-97727768c84c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-755453-cnmm7" Oct 29 23:27:55.629126 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:55.629022 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:27:55.663715 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:55.663687 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-755453-cnmm7" Oct 29 23:27:56.019083 ip-10-0-140-74 kubenswrapper[5057]: W1029 23:27:56.019056 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7be7f1f_537b_4b1f_abd6_97727768c84c.slice/crio-8fed21f7558b62d3aae96a52945ae0fe42a068d9406a08317500828337af947c WatchSource:0}: Error finding container 8fed21f7558b62d3aae96a52945ae0fe42a068d9406a08317500828337af947c: Status 404 returned error can't find the container with id 8fed21f7558b62d3aae96a52945ae0fe42a068d9406a08317500828337af947c Oct 29 23:27:56.020143 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:56.020116 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-755453-cnmm7"] Oct 29 23:27:56.128097 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:56.128061 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-755453-cnmm7" event={"ID":"f7be7f1f-537b-4b1f-abd6-97727768c84c","Type":"ContainerStarted","Data":"8fed21f7558b62d3aae96a52945ae0fe42a068d9406a08317500828337af947c"} Oct 29 23:27:57.055513 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:57.055481 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:27:57.139034 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:57.139000 5057 generic.go:356] "Generic (PLEG): container finished" podID="f7be7f1f-537b-4b1f-abd6-97727768c84c" containerID="6bb78445931729089d2b4c79d44dd7df626154a2304a24581aaa5db386360beb" exitCode=0 Oct 29 23:27:57.139167 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:57.139033 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-755453-cnmm7" event={"ID":"f7be7f1f-537b-4b1f-abd6-97727768c84c","Type":"ContainerDied","Data":"6bb78445931729089d2b4c79d44dd7df626154a2304a24581aaa5db386360beb"} Oct 29 23:27:58.417434 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:58.416970 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-755453-cnmm7" Oct 29 23:27:58.435071 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:58.435042 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f7be7f1f-537b-4b1f-abd6-97727768c84c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7be7f1f-537b-4b1f-abd6-97727768c84c-maas-api-xks-ca\") pod \"f7be7f1f-537b-4b1f-abd6-97727768c84c\" (UID: \"f7be7f1f-537b-4b1f-abd6-97727768c84c\") " Oct 29 23:27:58.435224 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:58.435089 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7be7f1f-537b-4b1f-abd6-97727768c84c-kube-api-access-8sm9m\" (UniqueName: \"kubernetes.io/projected/f7be7f1f-537b-4b1f-abd6-97727768c84c-kube-api-access-8sm9m\") pod \"f7be7f1f-537b-4b1f-abd6-97727768c84c\" (UID: \"f7be7f1f-537b-4b1f-abd6-97727768c84c\") " Oct 29 23:27:58.435311 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:58.435298 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f7be7f1f-537b-4b1f-abd6-97727768c84c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7be7f1f-537b-4b1f-abd6-97727768c84c-maas-api-openshift-ca\") pod \"f7be7f1f-537b-4b1f-abd6-97727768c84c\" (UID: \"f7be7f1f-537b-4b1f-abd6-97727768c84c\") " Oct 29 23:27:58.435659 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:58.435626 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7be7f1f-537b-4b1f-abd6-97727768c84c-maas-api-openshift-ca" pod "f7be7f1f-537b-4b1f-abd6-97727768c84c" (UID: "f7be7f1f-537b-4b1f-abd6-97727768c84c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:27:58.435973 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:58.435951 5057 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7be7f1f-537b-4b1f-abd6-97727768c84c-maas-api-openshift-ca\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:27:58.437191 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:58.437163 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7be7f1f-537b-4b1f-abd6-97727768c84c-kube-api-access-8sm9m" pod "f7be7f1f-537b-4b1f-abd6-97727768c84c" (UID: "f7be7f1f-537b-4b1f-abd6-97727768c84c"). InnerVolumeSpecName "kube-api-access-8sm9m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:27:58.437285 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:58.437222 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7be7f1f-537b-4b1f-abd6-97727768c84c-maas-api-xks-ca" pod "f7be7f1f-537b-4b1f-abd6-97727768c84c" (UID: "f7be7f1f-537b-4b1f-abd6-97727768c84c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:27:58.537639 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:58.537600 5057 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7be7f1f-537b-4b1f-abd6-97727768c84c-maas-api-xks-ca\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:27:58.537639 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:58.537633 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8sm9m\" (UniqueName: \"kubernetes.io/projected/f7be7f1f-537b-4b1f-abd6-97727768c84c-kube-api-access-8sm9m\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:27:58.826405 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:58.826366 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:27:59.159387 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:59.159295 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-755453-cnmm7" Oct 29 23:27:59.159387 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:59.159318 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-755453-cnmm7" event={"ID":"f7be7f1f-537b-4b1f-abd6-97727768c84c","Type":"ContainerDied","Data":"8fed21f7558b62d3aae96a52945ae0fe42a068d9406a08317500828337af947c"} Oct 29 23:27:59.159576 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:27:59.159397 5057 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8fed21f7558b62d3aae96a52945ae0fe42a068d9406a08317500828337af947c" Oct 29 23:28:00.107949 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:00.107918 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-nxkfc" Oct 29 23:28:00.171242 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:00.171206 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e52f656e-cf07-4fa7-aed0-21d17554a880-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e52f656e-cf07-4fa7-aed0-21d17554a880-istio-data\") pod \"e52f656e-cf07-4fa7-aed0-21d17554a880\" (UID: \"e52f656e-cf07-4fa7-aed0-21d17554a880\") " Oct 29 23:28:00.171476 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:00.171285 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e52f656e-cf07-4fa7-aed0-21d17554a880-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e52f656e-cf07-4fa7-aed0-21d17554a880-workload-socket\") pod \"e52f656e-cf07-4fa7-aed0-21d17554a880\" (UID: \"e52f656e-cf07-4fa7-aed0-21d17554a880\") " Oct 29 23:28:00.171476 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:00.171413 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e52f656e-cf07-4fa7-aed0-21d17554a880-istio-token\" (UniqueName: \"kubernetes.io/projected/e52f656e-cf07-4fa7-aed0-21d17554a880-istio-token\") pod \"e52f656e-cf07-4fa7-aed0-21d17554a880\" (UID: \"e52f656e-cf07-4fa7-aed0-21d17554a880\") " Oct 29 23:28:00.171476 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:00.171445 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e52f656e-cf07-4fa7-aed0-21d17554a880-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e52f656e-cf07-4fa7-aed0-21d17554a880-istio-envoy\") pod \"e52f656e-cf07-4fa7-aed0-21d17554a880\" (UID: \"e52f656e-cf07-4fa7-aed0-21d17554a880\") " Oct 29 23:28:00.171476 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:00.171470 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e52f656e-cf07-4fa7-aed0-21d17554a880-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e52f656e-cf07-4fa7-aed0-21d17554a880-istiod-ca-cert\") pod \"e52f656e-cf07-4fa7-aed0-21d17554a880\" (UID: \"e52f656e-cf07-4fa7-aed0-21d17554a880\") " Oct 29 23:28:00.171825 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:00.171521 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e52f656e-cf07-4fa7-aed0-21d17554a880-kube-api-access-msbbt\" (UniqueName: \"kubernetes.io/projected/e52f656e-cf07-4fa7-aed0-21d17554a880-kube-api-access-msbbt\") pod \"e52f656e-cf07-4fa7-aed0-21d17554a880\" (UID: \"e52f656e-cf07-4fa7-aed0-21d17554a880\") " Oct 29 23:28:00.171825 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:00.171557 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e52f656e-cf07-4fa7-aed0-21d17554a880-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e52f656e-cf07-4fa7-aed0-21d17554a880-credential-socket\") pod \"e52f656e-cf07-4fa7-aed0-21d17554a880\" (UID: \"e52f656e-cf07-4fa7-aed0-21d17554a880\") " Oct 29 23:28:00.171825 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:00.171733 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e52f656e-cf07-4fa7-aed0-21d17554a880-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e52f656e-cf07-4fa7-aed0-21d17554a880-workload-certs\") pod \"e52f656e-cf07-4fa7-aed0-21d17554a880\" (UID: \"e52f656e-cf07-4fa7-aed0-21d17554a880\") " Oct 29 23:28:00.171988 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:00.171890 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e52f656e-cf07-4fa7-aed0-21d17554a880-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e52f656e-cf07-4fa7-aed0-21d17554a880-istio-podinfo\") pod \"e52f656e-cf07-4fa7-aed0-21d17554a880\" (UID: \"e52f656e-cf07-4fa7-aed0-21d17554a880\") " Oct 29 23:28:00.176309 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:00.174867 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e52f656e-cf07-4fa7-aed0-21d17554a880-workload-socket" pod "e52f656e-cf07-4fa7-aed0-21d17554a880" (UID: "e52f656e-cf07-4fa7-aed0-21d17554a880"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:28:00.176309 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:00.174968 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cc6328-64879d8c8c-62gfk" podUID="3eb0219c-cf7b-4f66-b484-a78a2a5514a1" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:28:00.176309 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:00.175463 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e52f656e-cf07-4fa7-aed0-21d17554a880-istio-data" pod "e52f656e-cf07-4fa7-aed0-21d17554a880" (UID: "e52f656e-cf07-4fa7-aed0-21d17554a880"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:28:00.176309 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:00.175569 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e52f656e-cf07-4fa7-aed0-21d17554a880-workload-certs" pod "e52f656e-cf07-4fa7-aed0-21d17554a880" (UID: "e52f656e-cf07-4fa7-aed0-21d17554a880"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:28:00.176309 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:00.175766 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e52f656e-cf07-4fa7-aed0-21d17554a880-credential-socket" pod "e52f656e-cf07-4fa7-aed0-21d17554a880" (UID: "e52f656e-cf07-4fa7-aed0-21d17554a880"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:28:00.176309 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:00.175878 5057 generic.go:356] "Generic (PLEG): container finished" podID="e52f656e-cf07-4fa7-aed0-21d17554a880" containerID="0249b628f89b9ada1a6072e7ebce456108645a13908d336075140d3d8bf44761" exitCode=0 Oct 29 23:28:00.176309 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:00.175930 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e52f656e-cf07-4fa7-aed0-21d17554a880-istiod-ca-cert" pod "e52f656e-cf07-4fa7-aed0-21d17554a880" (UID: "e52f656e-cf07-4fa7-aed0-21d17554a880"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:28:00.176309 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:00.176017 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-nxkfc" Oct 29 23:28:00.176309 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:00.176060 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-nxkfc" event={"ID":"e52f656e-cf07-4fa7-aed0-21d17554a880","Type":"ContainerDied","Data":"0249b628f89b9ada1a6072e7ebce456108645a13908d336075140d3d8bf44761"} Oct 29 23:28:00.176309 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:00.176139 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-nxkfc" event={"ID":"e52f656e-cf07-4fa7-aed0-21d17554a880","Type":"ContainerDied","Data":"e0810f120c9eac168f9fe8637f5f2098ce789370cd560b103422383a4edf4b2b"} Oct 29 23:28:00.176309 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:00.176167 5057 scope.go:122] "RemoveContainer" containerID="0249b628f89b9ada1a6072e7ebce456108645a13908d336075140d3d8bf44761" Oct 29 23:28:00.182576 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:00.182537 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e52f656e-cf07-4fa7-aed0-21d17554a880-istio-podinfo" pod "e52f656e-cf07-4fa7-aed0-21d17554a880" (UID: "e52f656e-cf07-4fa7-aed0-21d17554a880"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:28:00.190918 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:00.190878 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e52f656e-cf07-4fa7-aed0-21d17554a880-kube-api-access-msbbt" pod "e52f656e-cf07-4fa7-aed0-21d17554a880" (UID: "e52f656e-cf07-4fa7-aed0-21d17554a880"). InnerVolumeSpecName "kube-api-access-msbbt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:28:00.191495 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:00.191460 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e52f656e-cf07-4fa7-aed0-21d17554a880-istio-token" pod "e52f656e-cf07-4fa7-aed0-21d17554a880" (UID: "e52f656e-cf07-4fa7-aed0-21d17554a880"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:28:00.206280 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:00.206233 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e52f656e-cf07-4fa7-aed0-21d17554a880-istio-envoy" pod "e52f656e-cf07-4fa7-aed0-21d17554a880" (UID: "e52f656e-cf07-4fa7-aed0-21d17554a880"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:28:00.232315 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:00.232172 5057 scope.go:122] "RemoveContainer" containerID="0249b628f89b9ada1a6072e7ebce456108645a13908d336075140d3d8bf44761" Oct 29 23:28:00.232756 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:28:00.232723 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0249b628f89b9ada1a6072e7ebce456108645a13908d336075140d3d8bf44761\": container with ID starting with 0249b628f89b9ada1a6072e7ebce456108645a13908d336075140d3d8bf44761 not found: ID does not exist" containerID="0249b628f89b9ada1a6072e7ebce456108645a13908d336075140d3d8bf44761" Oct 29 23:28:00.232845 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:00.232771 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0249b628f89b9ada1a6072e7ebce456108645a13908d336075140d3d8bf44761"} err="failed to get container status \"0249b628f89b9ada1a6072e7ebce456108645a13908d336075140d3d8bf44761\": rpc error: code = NotFound desc = could not find container \"0249b628f89b9ada1a6072e7ebce456108645a13908d336075140d3d8bf44761\": container with ID starting with 0249b628f89b9ada1a6072e7ebce456108645a13908d336075140d3d8bf44761 not found: ID does not exist" Oct 29 23:28:00.274750 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:00.274712 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-msbbt\" (UniqueName: \"kubernetes.io/projected/e52f656e-cf07-4fa7-aed0-21d17554a880-kube-api-access-msbbt\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:28:00.274750 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:00.274747 5057 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e52f656e-cf07-4fa7-aed0-21d17554a880-credential-socket\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:28:00.274966 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:00.274761 5057 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e52f656e-cf07-4fa7-aed0-21d17554a880-workload-certs\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:28:00.274966 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:00.274774 5057 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e52f656e-cf07-4fa7-aed0-21d17554a880-istio-podinfo\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:28:00.274966 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:00.274787 5057 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e52f656e-cf07-4fa7-aed0-21d17554a880-istio-data\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:28:00.274966 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:00.274799 5057 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e52f656e-cf07-4fa7-aed0-21d17554a880-workload-socket\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:28:00.274966 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:00.274810 5057 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e52f656e-cf07-4fa7-aed0-21d17554a880-istio-token\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:28:00.274966 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:00.274821 5057 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e52f656e-cf07-4fa7-aed0-21d17554a880-istio-envoy\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:28:00.274966 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:00.274833 5057 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e52f656e-cf07-4fa7-aed0-21d17554a880-istiod-ca-cert\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:28:00.414472 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:00.414318 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:28:00.538125 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:00.538059 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-nxkfc"] Oct 29 23:28:00.544751 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:00.544716 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-nxkfc"] Oct 29 23:28:01.333172 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:01.333137 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e52f656e-cf07-4fa7-aed0-21d17554a880" path="/var/lib/kubelet/pods/e52f656e-cf07-4fa7-aed0-21d17554a880/volumes" Oct 29 23:28:01.853987 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:01.848459 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:28:02.488221 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:02.483824 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:28:02.615387 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:02.615302 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:28:02.636416 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:02.636386 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-c880c7-s2nck"] Oct 29 23:28:02.671419 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:02.644579 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-c880c7-s2nck"] Oct 29 23:28:02.727268 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:02.727228 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:28:03.136175 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:03.136087 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:28:03.249119 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:03.249085 5057 generic.go:356] "Generic (PLEG): container finished" podID="9c02e87f-b552-45f4-97d2-d54429170748" containerID="ffe99f5d1703980ea3c268cb83c005ab697c8dacbb0e060ef257ad2e96a03df2" exitCode=0 Oct 29 23:28:03.249312 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:03.249161 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f3d24e75-openshift-default-7b4d9c69f7-rd4pc" event={"ID":"9c02e87f-b552-45f4-97d2-d54429170748","Type":"ContainerDied","Data":"ffe99f5d1703980ea3c268cb83c005ab697c8dacbb0e060ef257ad2e96a03df2"} Oct 29 23:28:03.249693 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:03.249677 5057 scope.go:122] "RemoveContainer" containerID="ffe99f5d1703980ea3c268cb83c005ab697c8dacbb0e060ef257ad2e96a03df2" Oct 29 23:28:03.251861 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:03.251827 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 23:28:03.251967 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:03.251915 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 23:28:03.251967 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:03.251959 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 23:28:03.348237 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:03.348202 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="451b200e-e1c1-4acc-968c-c79f0e1425de" path="/var/lib/kubelet/pods/451b200e-e1c1-4acc-968c-c79f0e1425de/volumes" Oct 29 23:28:03.968345 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:03.967457 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-cc6328-59f5f5554d-rgnmt"] Oct 29 23:28:03.968345 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:03.967711 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cc6328-59f5f5554d-rgnmt" podUID="04c996bb-8ee7-465f-9c69-ba118111d6f2" containerName="maas-api" containerID="cri-o://e0e16310fb9a9d5d25ac6d46d902666bb483f1dad6d0a020099547eaede51c88" gracePeriod=30 Oct 29 23:28:04.046354 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:04.045611 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:28:04.550399 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:04.549883 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f3d24e75-openshift-default-7b4d9c69f7-rd4pc" event={"ID":"9c02e87f-b552-45f4-97d2-d54429170748","Type":"ContainerStarted","Data":"6d483f81071b9b331fcb3bb241eac4037d0ef484096e892adccddc5d082273df"} Oct 29 23:28:04.553425 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:04.553387 5057 generic.go:356] "Generic (PLEG): container finished" podID="04c996bb-8ee7-465f-9c69-ba118111d6f2" containerID="e0e16310fb9a9d5d25ac6d46d902666bb483f1dad6d0a020099547eaede51c88" exitCode=0 Oct 29 23:28:04.553602 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:04.553480 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cc6328-59f5f5554d-rgnmt" Oct 29 23:28:04.553602 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:04.553563 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cc6328-59f5f5554d-rgnmt" event={"ID":"04c996bb-8ee7-465f-9c69-ba118111d6f2","Type":"ContainerDied","Data":"e0e16310fb9a9d5d25ac6d46d902666bb483f1dad6d0a020099547eaede51c88"} Oct 29 23:28:04.553742 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:04.553637 5057 scope.go:122] "RemoveContainer" containerID="e0e16310fb9a9d5d25ac6d46d902666bb483f1dad6d0a020099547eaede51c88" Oct 29 23:28:04.656865 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:04.656775 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:28:04.730380 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:04.724137 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04c996bb-8ee7-465f-9c69-ba118111d6f2-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/04c996bb-8ee7-465f-9c69-ba118111d6f2-openshift-service-ca\") pod \"04c996bb-8ee7-465f-9c69-ba118111d6f2\" (UID: \"04c996bb-8ee7-465f-9c69-ba118111d6f2\") " Oct 29 23:28:04.730380 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:04.724738 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04c996bb-8ee7-465f-9c69-ba118111d6f2-openshift-service-ca" pod "04c996bb-8ee7-465f-9c69-ba118111d6f2" (UID: "04c996bb-8ee7-465f-9c69-ba118111d6f2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:28:04.730380 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:04.724241 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/04c996bb-8ee7-465f-9c69-ba118111d6f2-metrics-certs\" (UniqueName: \"kubernetes.io/secret/04c996bb-8ee7-465f-9c69-ba118111d6f2-metrics-certs\") pod \"04c996bb-8ee7-465f-9c69-ba118111d6f2\" (UID: \"04c996bb-8ee7-465f-9c69-ba118111d6f2\") " Oct 29 23:28:04.730380 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:04.724928 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04c996bb-8ee7-465f-9c69-ba118111d6f2-kube-api-access-7x6dh\" (UniqueName: \"kubernetes.io/projected/04c996bb-8ee7-465f-9c69-ba118111d6f2-kube-api-access-7x6dh\") pod \"04c996bb-8ee7-465f-9c69-ba118111d6f2\" (UID: \"04c996bb-8ee7-465f-9c69-ba118111d6f2\") " Oct 29 23:28:04.730380 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:04.725286 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/04c996bb-8ee7-465f-9c69-ba118111d6f2-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/04c996bb-8ee7-465f-9c69-ba118111d6f2-maas-api-tls\") pod \"04c996bb-8ee7-465f-9c69-ba118111d6f2\" (UID: \"04c996bb-8ee7-465f-9c69-ba118111d6f2\") " Oct 29 23:28:04.730380 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:04.726302 5057 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/04c996bb-8ee7-465f-9c69-ba118111d6f2-openshift-service-ca\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:28:04.733203 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:04.733169 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04c996bb-8ee7-465f-9c69-ba118111d6f2-metrics-certs" pod "04c996bb-8ee7-465f-9c69-ba118111d6f2" (UID: "04c996bb-8ee7-465f-9c69-ba118111d6f2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:28:04.734723 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:04.733468 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04c996bb-8ee7-465f-9c69-ba118111d6f2-maas-api-tls" pod "04c996bb-8ee7-465f-9c69-ba118111d6f2" (UID: "04c996bb-8ee7-465f-9c69-ba118111d6f2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:28:04.742443 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:04.742416 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04c996bb-8ee7-465f-9c69-ba118111d6f2-kube-api-access-7x6dh" pod "04c996bb-8ee7-465f-9c69-ba118111d6f2" (UID: "04c996bb-8ee7-465f-9c69-ba118111d6f2"). InnerVolumeSpecName "kube-api-access-7x6dh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:28:04.770988 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:04.770959 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:28:04.827945 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:04.827909 5057 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/04c996bb-8ee7-465f-9c69-ba118111d6f2-metrics-certs\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:28:04.827945 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:04.827941 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7x6dh\" (UniqueName: \"kubernetes.io/projected/04c996bb-8ee7-465f-9c69-ba118111d6f2-kube-api-access-7x6dh\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:28:04.827945 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:04.827950 5057 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/04c996bb-8ee7-465f-9c69-ba118111d6f2-maas-api-tls\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:28:04.839480 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:04.839453 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-f3d24e75-openshift-default-7b4d9c69f7-rd4pc" Oct 29 23:28:04.839621 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:04.839599 5057 patch_prober.go:29] interesting pod/e2e-mt-f3d24e75-openshift-default-7b4d9c69f7-rd4pc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.250:15021/healthz/ready\": dial tcp 10.132.0.250:15021: connect: connection refused" start-of-body= Oct 29 23:28:04.839681 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:04.839662 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f3d24e75-openshift-default-7b4d9c69f7-rd4pc" podUID="9c02e87f-b552-45f4-97d2-d54429170748" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.250:15021/healthz/ready\": dial tcp 10.132.0.250:15021: connect: connection refused" Oct 29 23:28:04.899352 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:04.896220 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:28:05.297354 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:05.296810 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:28:05.564853 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:05.564774 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cc6328-59f5f5554d-rgnmt" Oct 29 23:28:05.564853 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:05.564834 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cc6328-59f5f5554d-rgnmt" event={"ID":"04c996bb-8ee7-465f-9c69-ba118111d6f2","Type":"ContainerDied","Data":"f2f532b17efc4d19211f7fd79863e9733d2200e6dfc2930f5d612f55f1b9f47e"} Oct 29 23:28:05.641050 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:05.641019 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-cc6328-59f5f5554d-rgnmt"] Oct 29 23:28:05.651356 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:05.651314 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-cc6328-59f5f5554d-rgnmt"] Oct 29 23:28:05.699355 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:05.692527 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-755453-9bd867db-hdlwb"] Oct 29 23:28:05.715429 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:05.715399 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-755453-788f5c78c-svk2k"] Oct 29 23:28:05.840735 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:05.840647 5057 patch_prober.go:29] interesting pod/e2e-mt-f3d24e75-openshift-default-7b4d9c69f7-rd4pc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.250:15021/healthz/ready\": dial tcp 10.132.0.250:15021: connect: connection refused" start-of-body= Oct 29 23:28:05.840735 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:05.840704 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f3d24e75-openshift-default-7b4d9c69f7-rd4pc" podUID="9c02e87f-b552-45f4-97d2-d54429170748" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.250:15021/healthz/ready\": dial tcp 10.132.0.250:15021: connect: connection refused" Oct 29 23:28:05.849713 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:05.849682 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:28:06.000352 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:05.997177 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:28:06.713129 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:06.713092 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-b8f5c4-584577d98b-tpwp2"] Oct 29 23:28:06.747380 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:06.747345 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-b8f5c4-cdd8644c9-hpdp5"] Oct 29 23:28:06.844151 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:06.844105 5057 patch_prober.go:29] interesting pod/e2e-mt-f3d24e75-openshift-default-7b4d9c69f7-rd4pc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.250:15021/healthz/ready\": dial tcp 10.132.0.250:15021: connect: connection refused" start-of-body= Oct 29 23:28:06.844388 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:06.844167 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f3d24e75-openshift-default-7b4d9c69f7-rd4pc" podUID="9c02e87f-b552-45f4-97d2-d54429170748" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.250:15021/healthz/ready\": dial tcp 10.132.0.250:15021: connect: connection refused" Oct 29 23:28:06.848656 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:06.848628 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:28:06.959535 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:06.959499 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:28:07.254438 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:07.254408 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w2-ed3bf2/rate-test-model-ed3bf2-kserve-7bbcbb9dd9-xpmj5"] Oct 29 23:28:07.255364 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:07.255310 5057 cpu_manager.go:414] "containerMap: removing container" podUID="04c996bb-8ee7-465f-9c69-ba118111d6f2" containerName="maas-api" Oct 29 23:28:07.255364 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:07.255343 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04c996bb-8ee7-465f-9c69-ba118111d6f2" containerName="maas-api" Oct 29 23:28:07.255526 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:07.255372 5057 cpu_manager.go:414] "containerMap: removing container" podUID="e52f656e-cf07-4fa7-aed0-21d17554a880" containerName="istio-proxy" Oct 29 23:28:07.255526 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:07.255377 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e52f656e-cf07-4fa7-aed0-21d17554a880" containerName="istio-proxy" Oct 29 23:28:07.255526 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:07.255409 5057 cpu_manager.go:414] "containerMap: removing container" podUID="f7be7f1f-537b-4b1f-abd6-97727768c84c" containerName="revoke-keys" Oct 29 23:28:07.255526 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:07.255414 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7be7f1f-537b-4b1f-abd6-97727768c84c" containerName="revoke-keys" Oct 29 23:28:07.255673 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:07.255561 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="e52f656e-cf07-4fa7-aed0-21d17554a880" containerName="istio-proxy" Oct 29 23:28:07.255673 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:07.255575 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7be7f1f-537b-4b1f-abd6-97727768c84c" containerName="revoke-keys" Oct 29 23:28:07.255673 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:07.255586 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="04c996bb-8ee7-465f-9c69-ba118111d6f2" containerName="maas-api" Oct 29 23:28:07.260872 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:07.260849 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-ed3bf2/rate-test-model-ed3bf2-kserve-7bbcbb9dd9-xpmj5" Oct 29 23:28:07.285374 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:07.276078 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-ed3bf2\"/\"rate-test-model-ed3bf2-kserve-self-signed-certs\"" Oct 29 23:28:07.285374 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:07.276201 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-ed3bf2\"/\"default-dockercfg-g7m9d\"" Oct 29 23:28:07.340390 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:07.339007 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-ed3bf2\"/\"kube-root-ca.crt\"" Oct 29 23:28:07.340390 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:07.339610 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-ed3bf2\"/\"openshift-service-ca.crt\"" Oct 29 23:28:07.342364 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:07.341233 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:28:07.363612 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:07.363581 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04c996bb-8ee7-465f-9c69-ba118111d6f2" path="/var/lib/kubelet/pods/04c996bb-8ee7-465f-9c69-ba118111d6f2/volumes" Oct 29 23:28:07.364016 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:07.363998 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-ed3bf2/rate-test-model-ed3bf2-kserve-7bbcbb9dd9-xpmj5"] Oct 29 23:28:07.375762 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:07.375732 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6084f4e5-c0ca-4f04-9aec-b1236e3d66db-dshm\") pod \"rate-test-model-ed3bf2-kserve-7bbcbb9dd9-xpmj5\" (UID: \"6084f4e5-c0ca-4f04-9aec-b1236e3d66db\") " pod="ai-tenant-e2e-shared-b-w2-ed3bf2/rate-test-model-ed3bf2-kserve-7bbcbb9dd9-xpmj5" Oct 29 23:28:07.375876 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:07.375775 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6084f4e5-c0ca-4f04-9aec-b1236e3d66db-tmp-dir\") pod \"rate-test-model-ed3bf2-kserve-7bbcbb9dd9-xpmj5\" (UID: \"6084f4e5-c0ca-4f04-9aec-b1236e3d66db\") " pod="ai-tenant-e2e-shared-b-w2-ed3bf2/rate-test-model-ed3bf2-kserve-7bbcbb9dd9-xpmj5" Oct 29 23:28:07.375876 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:07.375863 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6084f4e5-c0ca-4f04-9aec-b1236e3d66db-home\") pod \"rate-test-model-ed3bf2-kserve-7bbcbb9dd9-xpmj5\" (UID: \"6084f4e5-c0ca-4f04-9aec-b1236e3d66db\") " pod="ai-tenant-e2e-shared-b-w2-ed3bf2/rate-test-model-ed3bf2-kserve-7bbcbb9dd9-xpmj5" Oct 29 23:28:07.376008 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:07.375988 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6084f4e5-c0ca-4f04-9aec-b1236e3d66db-tls-certs\") pod \"rate-test-model-ed3bf2-kserve-7bbcbb9dd9-xpmj5\" (UID: \"6084f4e5-c0ca-4f04-9aec-b1236e3d66db\") " pod="ai-tenant-e2e-shared-b-w2-ed3bf2/rate-test-model-ed3bf2-kserve-7bbcbb9dd9-xpmj5" Oct 29 23:28:07.376158 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:07.376140 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6084f4e5-c0ca-4f04-9aec-b1236e3d66db-model-cache\") pod \"rate-test-model-ed3bf2-kserve-7bbcbb9dd9-xpmj5\" (UID: \"6084f4e5-c0ca-4f04-9aec-b1236e3d66db\") " pod="ai-tenant-e2e-shared-b-w2-ed3bf2/rate-test-model-ed3bf2-kserve-7bbcbb9dd9-xpmj5" Oct 29 23:28:07.376278 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:07.376191 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pp5bq\" (UniqueName: \"kubernetes.io/projected/6084f4e5-c0ca-4f04-9aec-b1236e3d66db-kube-api-access-pp5bq\") pod \"rate-test-model-ed3bf2-kserve-7bbcbb9dd9-xpmj5\" (UID: \"6084f4e5-c0ca-4f04-9aec-b1236e3d66db\") " pod="ai-tenant-e2e-shared-b-w2-ed3bf2/rate-test-model-ed3bf2-kserve-7bbcbb9dd9-xpmj5" Oct 29 23:28:07.487320 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:07.482305 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6084f4e5-c0ca-4f04-9aec-b1236e3d66db-tls-certs\") pod \"rate-test-model-ed3bf2-kserve-7bbcbb9dd9-xpmj5\" (UID: \"6084f4e5-c0ca-4f04-9aec-b1236e3d66db\") " pod="ai-tenant-e2e-shared-b-w2-ed3bf2/rate-test-model-ed3bf2-kserve-7bbcbb9dd9-xpmj5" Oct 29 23:28:07.487320 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:07.482514 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6084f4e5-c0ca-4f04-9aec-b1236e3d66db-model-cache\") pod \"rate-test-model-ed3bf2-kserve-7bbcbb9dd9-xpmj5\" (UID: \"6084f4e5-c0ca-4f04-9aec-b1236e3d66db\") " pod="ai-tenant-e2e-shared-b-w2-ed3bf2/rate-test-model-ed3bf2-kserve-7bbcbb9dd9-xpmj5" Oct 29 23:28:07.487320 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:07.482591 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pp5bq\" (UniqueName: \"kubernetes.io/projected/6084f4e5-c0ca-4f04-9aec-b1236e3d66db-kube-api-access-pp5bq\") pod \"rate-test-model-ed3bf2-kserve-7bbcbb9dd9-xpmj5\" (UID: \"6084f4e5-c0ca-4f04-9aec-b1236e3d66db\") " pod="ai-tenant-e2e-shared-b-w2-ed3bf2/rate-test-model-ed3bf2-kserve-7bbcbb9dd9-xpmj5" Oct 29 23:28:07.487320 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:07.482674 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6084f4e5-c0ca-4f04-9aec-b1236e3d66db-dshm\") pod \"rate-test-model-ed3bf2-kserve-7bbcbb9dd9-xpmj5\" (UID: \"6084f4e5-c0ca-4f04-9aec-b1236e3d66db\") " pod="ai-tenant-e2e-shared-b-w2-ed3bf2/rate-test-model-ed3bf2-kserve-7bbcbb9dd9-xpmj5" Oct 29 23:28:07.487320 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:07.482732 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6084f4e5-c0ca-4f04-9aec-b1236e3d66db-tmp-dir\") pod \"rate-test-model-ed3bf2-kserve-7bbcbb9dd9-xpmj5\" (UID: \"6084f4e5-c0ca-4f04-9aec-b1236e3d66db\") " pod="ai-tenant-e2e-shared-b-w2-ed3bf2/rate-test-model-ed3bf2-kserve-7bbcbb9dd9-xpmj5" Oct 29 23:28:07.487320 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:07.482778 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6084f4e5-c0ca-4f04-9aec-b1236e3d66db-home\") pod \"rate-test-model-ed3bf2-kserve-7bbcbb9dd9-xpmj5\" (UID: \"6084f4e5-c0ca-4f04-9aec-b1236e3d66db\") " pod="ai-tenant-e2e-shared-b-w2-ed3bf2/rate-test-model-ed3bf2-kserve-7bbcbb9dd9-xpmj5" Oct 29 23:28:07.487320 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:07.483236 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6084f4e5-c0ca-4f04-9aec-b1236e3d66db-home\") pod \"rate-test-model-ed3bf2-kserve-7bbcbb9dd9-xpmj5\" (UID: \"6084f4e5-c0ca-4f04-9aec-b1236e3d66db\") " pod="ai-tenant-e2e-shared-b-w2-ed3bf2/rate-test-model-ed3bf2-kserve-7bbcbb9dd9-xpmj5" Oct 29 23:28:07.488991 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:07.487903 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6084f4e5-c0ca-4f04-9aec-b1236e3d66db-model-cache\") pod \"rate-test-model-ed3bf2-kserve-7bbcbb9dd9-xpmj5\" (UID: \"6084f4e5-c0ca-4f04-9aec-b1236e3d66db\") " pod="ai-tenant-e2e-shared-b-w2-ed3bf2/rate-test-model-ed3bf2-kserve-7bbcbb9dd9-xpmj5" Oct 29 23:28:07.488991 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:07.488483 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6084f4e5-c0ca-4f04-9aec-b1236e3d66db-tmp-dir\") pod \"rate-test-model-ed3bf2-kserve-7bbcbb9dd9-xpmj5\" (UID: \"6084f4e5-c0ca-4f04-9aec-b1236e3d66db\") " pod="ai-tenant-e2e-shared-b-w2-ed3bf2/rate-test-model-ed3bf2-kserve-7bbcbb9dd9-xpmj5" Oct 29 23:28:07.508829 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:07.507882 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6084f4e5-c0ca-4f04-9aec-b1236e3d66db-dshm\") pod \"rate-test-model-ed3bf2-kserve-7bbcbb9dd9-xpmj5\" (UID: \"6084f4e5-c0ca-4f04-9aec-b1236e3d66db\") " pod="ai-tenant-e2e-shared-b-w2-ed3bf2/rate-test-model-ed3bf2-kserve-7bbcbb9dd9-xpmj5" Oct 29 23:28:07.512857 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:07.511528 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6084f4e5-c0ca-4f04-9aec-b1236e3d66db-tls-certs\") pod \"rate-test-model-ed3bf2-kserve-7bbcbb9dd9-xpmj5\" (UID: \"6084f4e5-c0ca-4f04-9aec-b1236e3d66db\") " pod="ai-tenant-e2e-shared-b-w2-ed3bf2/rate-test-model-ed3bf2-kserve-7bbcbb9dd9-xpmj5" Oct 29 23:28:07.519433 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:07.519138 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pp5bq\" (UniqueName: \"kubernetes.io/projected/6084f4e5-c0ca-4f04-9aec-b1236e3d66db-kube-api-access-pp5bq\") pod \"rate-test-model-ed3bf2-kserve-7bbcbb9dd9-xpmj5\" (UID: \"6084f4e5-c0ca-4f04-9aec-b1236e3d66db\") " pod="ai-tenant-e2e-shared-b-w2-ed3bf2/rate-test-model-ed3bf2-kserve-7bbcbb9dd9-xpmj5" Oct 29 23:28:07.585354 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:07.585098 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-ed3bf2/rate-test-model-ed3bf2-kserve-7bbcbb9dd9-xpmj5" Oct 29 23:28:07.839990 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:07.839957 5057 patch_prober.go:29] interesting pod/e2e-mt-f3d24e75-openshift-default-7b4d9c69f7-rd4pc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.250:15021/healthz/ready\": dial tcp 10.132.0.250:15021: connect: connection refused" start-of-body= Oct 29 23:28:07.840405 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:07.840034 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f3d24e75-openshift-default-7b4d9c69f7-rd4pc" podUID="9c02e87f-b552-45f4-97d2-d54429170748" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.250:15021/healthz/ready\": dial tcp 10.132.0.250:15021: connect: connection refused" Oct 29 23:28:07.973464 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:07.973436 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cc6328-64879d8c8c-62gfk" Oct 29 23:28:08.020382 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:08.020355 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-ed3bf2/rate-test-model-ed3bf2-kserve-7bbcbb9dd9-xpmj5"] Oct 29 23:28:08.020863 ip-10-0-140-74 kubenswrapper[5057]: W1029 23:28:08.020824 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6084f4e5_c0ca_4f04_9aec_b1236e3d66db.slice/crio-319b9103138b6f2241c89f9cab27ff4049f5a6ec9123fc3235cf6cd57e13ac95 WatchSource:0}: Error finding container 319b9103138b6f2241c89f9cab27ff4049f5a6ec9123fc3235cf6cd57e13ac95: Status 404 returned error can't find the container with id 319b9103138b6f2241c89f9cab27ff4049f5a6ec9123fc3235cf6cd57e13ac95 Oct 29 23:28:08.119879 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:08.118245 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3eb0219c-cf7b-4f66-b484-a78a2a5514a1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3eb0219c-cf7b-4f66-b484-a78a2a5514a1-plugins-config-volume\") pod \"3eb0219c-cf7b-4f66-b484-a78a2a5514a1\" (UID: \"3eb0219c-cf7b-4f66-b484-a78a2a5514a1\") " Oct 29 23:28:08.119879 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:08.118429 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3eb0219c-cf7b-4f66-b484-a78a2a5514a1-kube-api-access-6lp6g\" (UniqueName: \"kubernetes.io/projected/3eb0219c-cf7b-4f66-b484-a78a2a5514a1-kube-api-access-6lp6g\") pod \"3eb0219c-cf7b-4f66-b484-a78a2a5514a1\" (UID: \"3eb0219c-cf7b-4f66-b484-a78a2a5514a1\") " Oct 29 23:28:08.120392 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:08.120363 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3eb0219c-cf7b-4f66-b484-a78a2a5514a1-plugins-config-volume" pod "3eb0219c-cf7b-4f66-b484-a78a2a5514a1" (UID: "3eb0219c-cf7b-4f66-b484-a78a2a5514a1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:28:08.122761 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:08.122736 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3eb0219c-cf7b-4f66-b484-a78a2a5514a1-kube-api-access-6lp6g" pod "3eb0219c-cf7b-4f66-b484-a78a2a5514a1" (UID: "3eb0219c-cf7b-4f66-b484-a78a2a5514a1"). InnerVolumeSpecName "kube-api-access-6lp6g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:28:08.221401 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:08.221366 5057 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3eb0219c-cf7b-4f66-b484-a78a2a5514a1-plugins-config-volume\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:28:08.221401 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:08.221400 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6lp6g\" (UniqueName: \"kubernetes.io/projected/3eb0219c-cf7b-4f66-b484-a78a2a5514a1-kube-api-access-6lp6g\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:28:08.520182 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:08.520139 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:28:08.664994 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:08.664956 5057 generic.go:356] "Generic (PLEG): container finished" podID="3eb0219c-cf7b-4f66-b484-a78a2a5514a1" containerID="adc446a0e96784a34afcefe74ac6ed13c71698783c9d1dd2d33149600489864d" exitCode=0 Oct 29 23:28:08.665195 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:08.665011 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cc6328-64879d8c8c-62gfk" Oct 29 23:28:08.665195 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:08.665042 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cc6328-64879d8c8c-62gfk" event={"ID":"3eb0219c-cf7b-4f66-b484-a78a2a5514a1","Type":"ContainerDied","Data":"adc446a0e96784a34afcefe74ac6ed13c71698783c9d1dd2d33149600489864d"} Oct 29 23:28:08.665195 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:08.665097 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cc6328-64879d8c8c-62gfk" event={"ID":"3eb0219c-cf7b-4f66-b484-a78a2a5514a1","Type":"ContainerDied","Data":"11f5c4498fff2580c43c3747076241b60bfde742327f922dd283265e44eef89d"} Oct 29 23:28:08.665195 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:08.665125 5057 scope.go:122] "RemoveContainer" containerID="adc446a0e96784a34afcefe74ac6ed13c71698783c9d1dd2d33149600489864d" Oct 29 23:28:08.667130 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:08.667108 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-ed3bf2/rate-test-model-ed3bf2-kserve-7bbcbb9dd9-xpmj5" event={"ID":"6084f4e5-c0ca-4f04-9aec-b1236e3d66db","Type":"ContainerStarted","Data":"c6747472cb2c4c62b7ad8448c9ee1151edf2210f7c43034b63f2f28f1620a975"} Oct 29 23:28:08.667251 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:08.667139 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-ed3bf2/rate-test-model-ed3bf2-kserve-7bbcbb9dd9-xpmj5" event={"ID":"6084f4e5-c0ca-4f04-9aec-b1236e3d66db","Type":"ContainerStarted","Data":"319b9103138b6f2241c89f9cab27ff4049f5a6ec9123fc3235cf6cd57e13ac95"} Oct 29 23:28:08.710300 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:08.710273 5057 scope.go:122] "RemoveContainer" containerID="adc446a0e96784a34afcefe74ac6ed13c71698783c9d1dd2d33149600489864d" Oct 29 23:28:08.710703 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:28:08.710675 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"adc446a0e96784a34afcefe74ac6ed13c71698783c9d1dd2d33149600489864d\": container with ID starting with adc446a0e96784a34afcefe74ac6ed13c71698783c9d1dd2d33149600489864d not found: ID does not exist" containerID="adc446a0e96784a34afcefe74ac6ed13c71698783c9d1dd2d33149600489864d" Oct 29 23:28:08.710779 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:08.710716 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"adc446a0e96784a34afcefe74ac6ed13c71698783c9d1dd2d33149600489864d"} err="failed to get container status \"adc446a0e96784a34afcefe74ac6ed13c71698783c9d1dd2d33149600489864d\": rpc error: code = NotFound desc = could not find container \"adc446a0e96784a34afcefe74ac6ed13c71698783c9d1dd2d33149600489864d\": container with ID starting with adc446a0e96784a34afcefe74ac6ed13c71698783c9d1dd2d33149600489864d not found: ID does not exist" Oct 29 23:28:08.736120 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:08.736077 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w2-ed3bf2/rate-test-model-ed3bf2-kserve-7bbcbb9dd9-xpmj5" podStartSLOduration=1.736064066 podStartE2EDuration="1.736064066s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:28:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:28:08.734779791 +0000 UTC m=+3269.517993218" watchObservedRunningTime="2026-10-29 23:28:08.736064066 +0000 UTC m=+3269.519277493" Oct 29 23:28:08.776109 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:08.776046 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-cc6328-64879d8c8c-62gfk"] Oct 29 23:28:08.813118 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:08.813082 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-cc6328-64879d8c8c-62gfk"] Oct 29 23:28:08.839796 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:08.839768 5057 patch_prober.go:29] interesting pod/e2e-mt-f3d24e75-openshift-default-7b4d9c69f7-rd4pc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.250:15021/healthz/ready\": dial tcp 10.132.0.250:15021: connect: connection refused" start-of-body= Oct 29 23:28:08.839943 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:08.839816 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f3d24e75-openshift-default-7b4d9c69f7-rd4pc" podUID="9c02e87f-b552-45f4-97d2-d54429170748" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.250:15021/healthz/ready\": dial tcp 10.132.0.250:15021: connect: connection refused" Oct 29 23:28:09.332428 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:09.332385 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3eb0219c-cf7b-4f66-b484-a78a2a5514a1" path="/var/lib/kubelet/pods/3eb0219c-cf7b-4f66-b484-a78a2a5514a1/volumes" Oct 29 23:28:09.586065 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:09.585982 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w2-ed3bf2/rate-test-model-ed3bf2-kserve-7bbcbb9dd9-xpmj5" Oct 29 23:28:09.800838 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:09.800798 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:28:09.845296 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:09.845219 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-f3d24e75-openshift-default-7b4d9c69f7-rd4pc" Oct 29 23:28:09.845690 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:09.845670 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-f3d24e75-openshift-default-7b4d9c69f7-rd4pc" Oct 29 23:28:09.846427 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:09.846405 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-f3d24e75-openshift-default-7b4d9c69f7-rd4pc" Oct 29 23:28:09.923645 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:09.923591 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:28:10.201406 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:10.190465 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:28:10.322634 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:10.322567 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:28:10.693013 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:10.692924 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-755453-9bd867db-hdlwb" podUID="6760c386-e49a-4d99-b811-31a8d9df5b41" containerName="payload-processing" containerID="cri-o://5ec94988b8438146a3c75eb716cc8925ef57b4b532a947dcc1c377448579dc3d" gracePeriod=25 Oct 29 23:28:10.716488 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:10.716438 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-755453-788f5c78c-svk2k" podUID="0e08ea7b-6096-4253-a757-b092cf07bc7c" containerName="payload-pre-processing" containerID="cri-o://e15440c29d67c92cf76444e89c189c274e005fdcd089d174448a8814ee0c202a" gracePeriod=25 Oct 29 23:28:11.713705 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:11.713667 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-b8f5c4-584577d98b-tpwp2" podUID="074dc860-7008-4c4f-9c9e-0fe9059b1947" containerName="payload-processing" containerID="cri-o://8dbf9fe6b03d54fa2705a50cb4b71157296dc4dd80f4aa33754a7d86334f3178" gracePeriod=25 Oct 29 23:28:11.747791 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:11.747760 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-b8f5c4-cdd8644c9-hpdp5" podUID="e6c69267-32e0-4a27-a376-d9918d3f2e3b" containerName="payload-pre-processing" containerID="cri-o://31e3629a311ce5d04d9e20fc37fe8042d0a167af1e267f1769c698ee9fc31567" gracePeriod=25 Oct 29 23:28:12.054701 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:12.054670 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-b8f5c4-584577d98b-tpwp2" Oct 29 23:28:12.090684 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:12.090655 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-b8f5c4-cdd8644c9-hpdp5" Oct 29 23:28:12.120134 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:12.120095 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:28:12.196849 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:12.196811 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6c69267-32e0-4a27-a376-d9918d3f2e3b-kube-api-access-pwfbd\" (UniqueName: \"kubernetes.io/projected/e6c69267-32e0-4a27-a376-d9918d3f2e3b-kube-api-access-pwfbd\") pod \"e6c69267-32e0-4a27-a376-d9918d3f2e3b\" (UID: \"e6c69267-32e0-4a27-a376-d9918d3f2e3b\") " Oct 29 23:28:12.197068 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:12.197001 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/074dc860-7008-4c4f-9c9e-0fe9059b1947-kube-api-access-4lzvk\" (UniqueName: \"kubernetes.io/projected/074dc860-7008-4c4f-9c9e-0fe9059b1947-kube-api-access-4lzvk\") pod \"074dc860-7008-4c4f-9c9e-0fe9059b1947\" (UID: \"074dc860-7008-4c4f-9c9e-0fe9059b1947\") " Oct 29 23:28:12.197173 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:12.197153 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e6c69267-32e0-4a27-a376-d9918d3f2e3b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6c69267-32e0-4a27-a376-d9918d3f2e3b-plugins-config-volume\") pod \"e6c69267-32e0-4a27-a376-d9918d3f2e3b\" (UID: \"e6c69267-32e0-4a27-a376-d9918d3f2e3b\") " Oct 29 23:28:12.197243 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:12.197190 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/074dc860-7008-4c4f-9c9e-0fe9059b1947-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/074dc860-7008-4c4f-9c9e-0fe9059b1947-plugins-config-volume\") pod \"074dc860-7008-4c4f-9c9e-0fe9059b1947\" (UID: \"074dc860-7008-4c4f-9c9e-0fe9059b1947\") " Oct 29 23:28:12.197922 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:12.197616 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6c69267-32e0-4a27-a376-d9918d3f2e3b-plugins-config-volume" pod "e6c69267-32e0-4a27-a376-d9918d3f2e3b" (UID: "e6c69267-32e0-4a27-a376-d9918d3f2e3b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:28:12.198237 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:12.198204 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/074dc860-7008-4c4f-9c9e-0fe9059b1947-plugins-config-volume" pod "074dc860-7008-4c4f-9c9e-0fe9059b1947" (UID: "074dc860-7008-4c4f-9c9e-0fe9059b1947"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:28:12.199030 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:12.198994 5057 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6c69267-32e0-4a27-a376-d9918d3f2e3b-plugins-config-volume\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:28:12.199030 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:12.199026 5057 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/074dc860-7008-4c4f-9c9e-0fe9059b1947-plugins-config-volume\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:28:12.200705 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:12.200678 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6c69267-32e0-4a27-a376-d9918d3f2e3b-kube-api-access-pwfbd" pod "e6c69267-32e0-4a27-a376-d9918d3f2e3b" (UID: "e6c69267-32e0-4a27-a376-d9918d3f2e3b"). InnerVolumeSpecName "kube-api-access-pwfbd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:28:12.201998 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:12.201970 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/074dc860-7008-4c4f-9c9e-0fe9059b1947-kube-api-access-4lzvk" pod "074dc860-7008-4c4f-9c9e-0fe9059b1947" (UID: "074dc860-7008-4c4f-9c9e-0fe9059b1947"). InnerVolumeSpecName "kube-api-access-4lzvk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:28:12.256201 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:12.256160 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-755453-788f5c78c-svk2k" podUID="0e08ea7b-6096-4253-a757-b092cf07bc7c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:28:12.262838 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:12.262798 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:28:12.301170 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:12.301062 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pwfbd\" (UniqueName: \"kubernetes.io/projected/e6c69267-32e0-4a27-a376-d9918d3f2e3b-kube-api-access-pwfbd\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:28:12.301170 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:12.301101 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4lzvk\" (UniqueName: \"kubernetes.io/projected/074dc860-7008-4c4f-9c9e-0fe9059b1947-kube-api-access-4lzvk\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:28:12.372662 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:12.372629 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-cc6328-openshift-default-84c4dfccc9-8k9hr"] Oct 29 23:28:12.372840 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:12.372822 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-cc6328-openshift-default-84c4dfccc9-8k9hr" podUID="268ae24d-3a42-472a-97d7-aeacd9989632" containerName="istio-proxy" containerID="cri-o://64462bcdeb77a27dc04b5f2652ad8fcb2fd0f9841fd47f70ac2fa2eded568cf4" gracePeriod=30 Oct 29 23:28:12.386347 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:12.381139 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:28:12.507715 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:12.506063 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:28:12.715953 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:12.715922 5057 generic.go:356] "Generic (PLEG): container finished" podID="074dc860-7008-4c4f-9c9e-0fe9059b1947" containerID="8dbf9fe6b03d54fa2705a50cb4b71157296dc4dd80f4aa33754a7d86334f3178" exitCode=0 Oct 29 23:28:12.716450 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:12.715966 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-b8f5c4-584577d98b-tpwp2" Oct 29 23:28:12.716450 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:12.715991 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-b8f5c4-584577d98b-tpwp2" event={"ID":"074dc860-7008-4c4f-9c9e-0fe9059b1947","Type":"ContainerDied","Data":"8dbf9fe6b03d54fa2705a50cb4b71157296dc4dd80f4aa33754a7d86334f3178"} Oct 29 23:28:12.716450 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:12.716029 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-b8f5c4-584577d98b-tpwp2" event={"ID":"074dc860-7008-4c4f-9c9e-0fe9059b1947","Type":"ContainerDied","Data":"bfa48e61bc90a5cc416507c3042d79ce002e5792e16f7ccb3d807732da46825c"} Oct 29 23:28:12.716450 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:12.716049 5057 scope.go:122] "RemoveContainer" containerID="8dbf9fe6b03d54fa2705a50cb4b71157296dc4dd80f4aa33754a7d86334f3178" Oct 29 23:28:12.717694 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:12.717669 5057 generic.go:356] "Generic (PLEG): container finished" podID="e6c69267-32e0-4a27-a376-d9918d3f2e3b" containerID="31e3629a311ce5d04d9e20fc37fe8042d0a167af1e267f1769c698ee9fc31567" exitCode=0 Oct 29 23:28:12.717818 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:12.717723 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-b8f5c4-cdd8644c9-hpdp5" Oct 29 23:28:12.717818 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:12.717762 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-b8f5c4-cdd8644c9-hpdp5" event={"ID":"e6c69267-32e0-4a27-a376-d9918d3f2e3b","Type":"ContainerDied","Data":"31e3629a311ce5d04d9e20fc37fe8042d0a167af1e267f1769c698ee9fc31567"} Oct 29 23:28:12.717818 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:12.717815 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-b8f5c4-cdd8644c9-hpdp5" event={"ID":"e6c69267-32e0-4a27-a376-d9918d3f2e3b","Type":"ContainerDied","Data":"0a3f9f5dc69ce624677091fb1efb08b8177790c929e103a31a326181e5590983"} Oct 29 23:28:12.747568 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:12.747545 5057 scope.go:122] "RemoveContainer" containerID="8dbf9fe6b03d54fa2705a50cb4b71157296dc4dd80f4aa33754a7d86334f3178" Oct 29 23:28:12.747902 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:28:12.747868 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8dbf9fe6b03d54fa2705a50cb4b71157296dc4dd80f4aa33754a7d86334f3178\": container with ID starting with 8dbf9fe6b03d54fa2705a50cb4b71157296dc4dd80f4aa33754a7d86334f3178 not found: ID does not exist" containerID="8dbf9fe6b03d54fa2705a50cb4b71157296dc4dd80f4aa33754a7d86334f3178" Oct 29 23:28:12.748014 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:12.747909 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8dbf9fe6b03d54fa2705a50cb4b71157296dc4dd80f4aa33754a7d86334f3178"} err="failed to get container status \"8dbf9fe6b03d54fa2705a50cb4b71157296dc4dd80f4aa33754a7d86334f3178\": rpc error: code = NotFound desc = could not find container \"8dbf9fe6b03d54fa2705a50cb4b71157296dc4dd80f4aa33754a7d86334f3178\": container with ID starting with 8dbf9fe6b03d54fa2705a50cb4b71157296dc4dd80f4aa33754a7d86334f3178 not found: ID does not exist" Oct 29 23:28:12.748014 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:12.747932 5057 scope.go:122] "RemoveContainer" containerID="31e3629a311ce5d04d9e20fc37fe8042d0a167af1e267f1769c698ee9fc31567" Oct 29 23:28:12.758714 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:12.758696 5057 scope.go:122] "RemoveContainer" containerID="31e3629a311ce5d04d9e20fc37fe8042d0a167af1e267f1769c698ee9fc31567" Oct 29 23:28:12.758999 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:28:12.758975 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31e3629a311ce5d04d9e20fc37fe8042d0a167af1e267f1769c698ee9fc31567\": container with ID starting with 31e3629a311ce5d04d9e20fc37fe8042d0a167af1e267f1769c698ee9fc31567 not found: ID does not exist" containerID="31e3629a311ce5d04d9e20fc37fe8042d0a167af1e267f1769c698ee9fc31567" Oct 29 23:28:12.759052 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:12.759011 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31e3629a311ce5d04d9e20fc37fe8042d0a167af1e267f1769c698ee9fc31567"} err="failed to get container status \"31e3629a311ce5d04d9e20fc37fe8042d0a167af1e267f1769c698ee9fc31567\": rpc error: code = NotFound desc = could not find container \"31e3629a311ce5d04d9e20fc37fe8042d0a167af1e267f1769c698ee9fc31567\": container with ID starting with 31e3629a311ce5d04d9e20fc37fe8042d0a167af1e267f1769c698ee9fc31567 not found: ID does not exist" Oct 29 23:28:12.786472 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:12.786445 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-b8f5c4-cdd8644c9-hpdp5"] Oct 29 23:28:12.791009 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:12.790984 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-b8f5c4-cdd8644c9-hpdp5"] Oct 29 23:28:12.819723 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:12.819696 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-b8f5c4-584577d98b-tpwp2"] Oct 29 23:28:12.824221 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:12.824198 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-b8f5c4-584577d98b-tpwp2"] Oct 29 23:28:13.263060 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:13.263018 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-755453-9bd867db-hdlwb" podUID="6760c386-e49a-4d99-b811-31a8d9df5b41" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:28:13.333946 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:13.333916 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="074dc860-7008-4c4f-9c9e-0fe9059b1947" path="/var/lib/kubelet/pods/074dc860-7008-4c4f-9c9e-0fe9059b1947/volumes" Oct 29 23:28:13.334393 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:13.334377 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e6c69267-32e0-4a27-a376-d9918d3f2e3b" path="/var/lib/kubelet/pods/e6c69267-32e0-4a27-a376-d9918d3f2e3b/volumes" Oct 29 23:28:13.456269 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:13.456216 5057 patch_prober.go:29] interesting pod/e2e-inf-b-cc6328-openshift-default-84c4dfccc9-8k9hr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 23:28:13.456469 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:13.456294 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-cc6328-openshift-default-84c4dfccc9-8k9hr" podUID="268ae24d-3a42-472a-97d7-aeacd9989632" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 23:28:14.002500 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:14.002467 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:28:14.032653 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:14.032623 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d836f867-m6fqr"] Oct 29 23:28:14.043688 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:14.043660 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d836f867-m6fqr"] Oct 29 23:28:15.332806 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:15.332769 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c1683db-89f4-4cff-ae7a-5170fc9645e4" path="/var/lib/kubelet/pods/6c1683db-89f4-4cff-ae7a-5170fc9645e4/volumes" Oct 29 23:28:15.490480 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:15.490441 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:28:17.118579 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:17.118530 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:28:17.588108 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:17.586661 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w2-ed3bf2/rate-test-model-ed3bf2-kserve-7bbcbb9dd9-xpmj5" Oct 29 23:28:17.604504 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:17.604157 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w2-ed3bf2/rate-test-model-ed3bf2-kserve-7bbcbb9dd9-xpmj5" Oct 29 23:28:17.684613 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:17.684583 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-cc6328-openshift-default-84c4dfccc9-8k9hr" Oct 29 23:28:17.775408 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:17.775326 5057 generic.go:356] "Generic (PLEG): container finished" podID="268ae24d-3a42-472a-97d7-aeacd9989632" containerID="64462bcdeb77a27dc04b5f2652ad8fcb2fd0f9841fd47f70ac2fa2eded568cf4" exitCode=0 Oct 29 23:28:17.775667 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:17.775382 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-cc6328-openshift-default-84c4dfccc9-8k9hr" event={"ID":"268ae24d-3a42-472a-97d7-aeacd9989632","Type":"ContainerDied","Data":"64462bcdeb77a27dc04b5f2652ad8fcb2fd0f9841fd47f70ac2fa2eded568cf4"} Oct 29 23:28:17.775768 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:17.775718 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-cc6328-openshift-default-84c4dfccc9-8k9hr" event={"ID":"268ae24d-3a42-472a-97d7-aeacd9989632","Type":"ContainerDied","Data":"1a713655ceb50ffbcdb5ef6492d3c0a9719422a3f8340447e84982c26bb40184"} Oct 29 23:28:17.775768 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:17.775423 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-cc6328-openshift-default-84c4dfccc9-8k9hr" Oct 29 23:28:17.775768 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:17.775757 5057 scope.go:122] "RemoveContainer" containerID="64462bcdeb77a27dc04b5f2652ad8fcb2fd0f9841fd47f70ac2fa2eded568cf4" Oct 29 23:28:17.781051 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:17.781016 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/268ae24d-3a42-472a-97d7-aeacd9989632-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/268ae24d-3a42-472a-97d7-aeacd9989632-istio-podinfo\") pod \"268ae24d-3a42-472a-97d7-aeacd9989632\" (UID: \"268ae24d-3a42-472a-97d7-aeacd9989632\") " Oct 29 23:28:17.781620 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:17.781596 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/268ae24d-3a42-472a-97d7-aeacd9989632-istio-token\" (UniqueName: \"kubernetes.io/projected/268ae24d-3a42-472a-97d7-aeacd9989632-istio-token\") pod \"268ae24d-3a42-472a-97d7-aeacd9989632\" (UID: \"268ae24d-3a42-472a-97d7-aeacd9989632\") " Oct 29 23:28:17.781751 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:17.781694 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/268ae24d-3a42-472a-97d7-aeacd9989632-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/268ae24d-3a42-472a-97d7-aeacd9989632-istio-envoy\") pod \"268ae24d-3a42-472a-97d7-aeacd9989632\" (UID: \"268ae24d-3a42-472a-97d7-aeacd9989632\") " Oct 29 23:28:17.783083 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:17.781824 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/268ae24d-3a42-472a-97d7-aeacd9989632-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/268ae24d-3a42-472a-97d7-aeacd9989632-istio-data\") pod \"268ae24d-3a42-472a-97d7-aeacd9989632\" (UID: \"268ae24d-3a42-472a-97d7-aeacd9989632\") " Oct 29 23:28:17.783083 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:17.781930 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/268ae24d-3a42-472a-97d7-aeacd9989632-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/268ae24d-3a42-472a-97d7-aeacd9989632-workload-socket\") pod \"268ae24d-3a42-472a-97d7-aeacd9989632\" (UID: \"268ae24d-3a42-472a-97d7-aeacd9989632\") " Oct 29 23:28:17.783083 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:17.782007 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/268ae24d-3a42-472a-97d7-aeacd9989632-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/268ae24d-3a42-472a-97d7-aeacd9989632-credential-socket\") pod \"268ae24d-3a42-472a-97d7-aeacd9989632\" (UID: \"268ae24d-3a42-472a-97d7-aeacd9989632\") " Oct 29 23:28:17.783083 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:17.782059 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/268ae24d-3a42-472a-97d7-aeacd9989632-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/268ae24d-3a42-472a-97d7-aeacd9989632-workload-certs\") pod \"268ae24d-3a42-472a-97d7-aeacd9989632\" (UID: \"268ae24d-3a42-472a-97d7-aeacd9989632\") " Oct 29 23:28:17.783083 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:17.782124 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/268ae24d-3a42-472a-97d7-aeacd9989632-kube-api-access-9l5l4\" (UniqueName: \"kubernetes.io/projected/268ae24d-3a42-472a-97d7-aeacd9989632-kube-api-access-9l5l4\") pod \"268ae24d-3a42-472a-97d7-aeacd9989632\" (UID: \"268ae24d-3a42-472a-97d7-aeacd9989632\") " Oct 29 23:28:17.783083 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:17.782199 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/268ae24d-3a42-472a-97d7-aeacd9989632-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/268ae24d-3a42-472a-97d7-aeacd9989632-istiod-ca-cert\") pod \"268ae24d-3a42-472a-97d7-aeacd9989632\" (UID: \"268ae24d-3a42-472a-97d7-aeacd9989632\") " Oct 29 23:28:17.783083 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:17.782408 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/268ae24d-3a42-472a-97d7-aeacd9989632-istio-data" pod "268ae24d-3a42-472a-97d7-aeacd9989632" (UID: "268ae24d-3a42-472a-97d7-aeacd9989632"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:28:17.783083 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:17.782927 5057 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/268ae24d-3a42-472a-97d7-aeacd9989632-istio-data\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:28:17.783083 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:17.782934 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/268ae24d-3a42-472a-97d7-aeacd9989632-workload-certs" pod "268ae24d-3a42-472a-97d7-aeacd9989632" (UID: "268ae24d-3a42-472a-97d7-aeacd9989632"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:28:17.783083 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:17.783058 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/268ae24d-3a42-472a-97d7-aeacd9989632-credential-socket" pod "268ae24d-3a42-472a-97d7-aeacd9989632" (UID: "268ae24d-3a42-472a-97d7-aeacd9989632"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:28:17.783928 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:17.783108 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/268ae24d-3a42-472a-97d7-aeacd9989632-workload-socket" pod "268ae24d-3a42-472a-97d7-aeacd9989632" (UID: "268ae24d-3a42-472a-97d7-aeacd9989632"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:28:17.783928 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:17.783190 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/268ae24d-3a42-472a-97d7-aeacd9989632-istiod-ca-cert" pod "268ae24d-3a42-472a-97d7-aeacd9989632" (UID: "268ae24d-3a42-472a-97d7-aeacd9989632"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:28:17.787500 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:17.787472 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/268ae24d-3a42-472a-97d7-aeacd9989632-istio-envoy" pod "268ae24d-3a42-472a-97d7-aeacd9989632" (UID: "268ae24d-3a42-472a-97d7-aeacd9989632"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:28:17.799585 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:17.799503 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/268ae24d-3a42-472a-97d7-aeacd9989632-istio-token" pod "268ae24d-3a42-472a-97d7-aeacd9989632" (UID: "268ae24d-3a42-472a-97d7-aeacd9989632"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:28:17.799705 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:17.799601 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/268ae24d-3a42-472a-97d7-aeacd9989632-kube-api-access-9l5l4" pod "268ae24d-3a42-472a-97d7-aeacd9989632" (UID: "268ae24d-3a42-472a-97d7-aeacd9989632"). InnerVolumeSpecName "kube-api-access-9l5l4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:28:17.800418 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:17.799707 5057 scope.go:122] "RemoveContainer" containerID="060923ec45c1ee57ef9922829f4a5166a251b06042203ba607f48932a190b092" Oct 29 23:28:17.802088 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:17.801795 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w2-ed3bf2/rate-test-model-ed3bf2-kserve-7bbcbb9dd9-xpmj5" Oct 29 23:28:17.803449 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:17.803421 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/268ae24d-3a42-472a-97d7-aeacd9989632-istio-podinfo" pod "268ae24d-3a42-472a-97d7-aeacd9989632" (UID: "268ae24d-3a42-472a-97d7-aeacd9989632"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:28:17.827053 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:17.827031 5057 scope.go:122] "RemoveContainer" containerID="64462bcdeb77a27dc04b5f2652ad8fcb2fd0f9841fd47f70ac2fa2eded568cf4" Oct 29 23:28:17.827371 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:28:17.827321 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64462bcdeb77a27dc04b5f2652ad8fcb2fd0f9841fd47f70ac2fa2eded568cf4\": container with ID starting with 64462bcdeb77a27dc04b5f2652ad8fcb2fd0f9841fd47f70ac2fa2eded568cf4 not found: ID does not exist" containerID="64462bcdeb77a27dc04b5f2652ad8fcb2fd0f9841fd47f70ac2fa2eded568cf4" Oct 29 23:28:17.827480 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:17.827387 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64462bcdeb77a27dc04b5f2652ad8fcb2fd0f9841fd47f70ac2fa2eded568cf4"} err="failed to get container status \"64462bcdeb77a27dc04b5f2652ad8fcb2fd0f9841fd47f70ac2fa2eded568cf4\": rpc error: code = NotFound desc = could not find container \"64462bcdeb77a27dc04b5f2652ad8fcb2fd0f9841fd47f70ac2fa2eded568cf4\": container with ID starting with 64462bcdeb77a27dc04b5f2652ad8fcb2fd0f9841fd47f70ac2fa2eded568cf4 not found: ID does not exist" Oct 29 23:28:17.827480 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:17.827413 5057 scope.go:122] "RemoveContainer" containerID="060923ec45c1ee57ef9922829f4a5166a251b06042203ba607f48932a190b092" Oct 29 23:28:17.827713 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:28:17.827693 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"060923ec45c1ee57ef9922829f4a5166a251b06042203ba607f48932a190b092\": container with ID starting with 060923ec45c1ee57ef9922829f4a5166a251b06042203ba607f48932a190b092 not found: ID does not exist" containerID="060923ec45c1ee57ef9922829f4a5166a251b06042203ba607f48932a190b092" Oct 29 23:28:17.827762 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:17.827719 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"060923ec45c1ee57ef9922829f4a5166a251b06042203ba607f48932a190b092"} err="failed to get container status \"060923ec45c1ee57ef9922829f4a5166a251b06042203ba607f48932a190b092\": rpc error: code = NotFound desc = could not find container \"060923ec45c1ee57ef9922829f4a5166a251b06042203ba607f48932a190b092\": container with ID starting with 060923ec45c1ee57ef9922829f4a5166a251b06042203ba607f48932a190b092 not found: ID does not exist" Oct 29 23:28:17.884298 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:17.884263 5057 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/268ae24d-3a42-472a-97d7-aeacd9989632-istio-podinfo\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:28:17.884298 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:17.884297 5057 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/268ae24d-3a42-472a-97d7-aeacd9989632-istio-token\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:28:17.884557 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:17.884311 5057 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/268ae24d-3a42-472a-97d7-aeacd9989632-istio-envoy\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:28:17.884557 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:17.884325 5057 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/268ae24d-3a42-472a-97d7-aeacd9989632-workload-socket\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:28:17.884557 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:17.884354 5057 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/268ae24d-3a42-472a-97d7-aeacd9989632-credential-socket\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:28:17.884557 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:17.884388 5057 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/268ae24d-3a42-472a-97d7-aeacd9989632-workload-certs\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:28:17.884557 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:17.884404 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9l5l4\" (UniqueName: \"kubernetes.io/projected/268ae24d-3a42-472a-97d7-aeacd9989632-kube-api-access-9l5l4\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:28:17.884557 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:17.884418 5057 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/268ae24d-3a42-472a-97d7-aeacd9989632-istiod-ca-cert\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:28:18.130678 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:18.130643 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-cc6328-openshift-default-84c4dfccc9-8k9hr"] Oct 29 23:28:18.139432 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:18.139404 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-cc6328-openshift-default-84c4dfccc9-8k9hr"] Oct 29 23:28:18.703457 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:18.703422 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:28:19.029852 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:28:19.029771 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d2dnn" containerName="extractor" Oct 29 23:28:19.333656 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:19.333579 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="268ae24d-3a42-472a-97d7-aeacd9989632" path="/var/lib/kubelet/pods/268ae24d-3a42-472a-97d7-aeacd9989632/volumes" Oct 29 23:28:20.293695 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:20.293658 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:28:21.351668 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:21.351544 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:28:21.466834 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:21.466801 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:28:21.893910 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:21.893873 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:28:22.255465 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:22.255429 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-755453-788f5c78c-svk2k" podUID="0e08ea7b-6096-4253-a757-b092cf07bc7c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:28:23.262927 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:23.262890 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-755453-9bd867db-hdlwb" podUID="6760c386-e49a-4d99-b811-31a8d9df5b41" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:28:23.511654 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:23.511606 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:28:25.001968 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:25.001928 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:28:26.038468 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:26.038434 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ffd0336a-kz9q8"] Oct 29 23:28:26.049108 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:26.049080 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ffd0336a-kz9q8"] Oct 29 23:28:26.494112 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:26.494078 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:28:27.031442 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:27.031399 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b6acce1-qzn47"] Oct 29 23:28:27.043058 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:27.043028 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c5febaf3-x49n2"] Oct 29 23:28:27.047424 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:27.047398 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b6acce1-qzn47"] Oct 29 23:28:27.051084 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:27.051064 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c5febaf3-x49n2"] Oct 29 23:28:27.331592 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:27.331509 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4fab3853-8396-4cf1-af41-96c367c1e4f4" path="/var/lib/kubelet/pods/4fab3853-8396-4cf1-af41-96c367c1e4f4/volumes" Oct 29 23:28:27.332000 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:27.331985 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8eb9f5be-be7c-4469-a840-92fe5a4600fd" path="/var/lib/kubelet/pods/8eb9f5be-be7c-4469-a840-92fe5a4600fd/volumes" Oct 29 23:28:27.332435 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:27.332420 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9570ca18-a7cc-4041-b505-e11161579302" path="/var/lib/kubelet/pods/9570ca18-a7cc-4041-b505-e11161579302/volumes" Oct 29 23:28:27.834496 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:27.834460 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:28:28.193125 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:28.193034 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:28:28.355581 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:28.355525 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:28:28.463708 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:28.463669 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:28:28.542748 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:28.542713 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-755453-cnmm7"] Oct 29 23:28:28.554308 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:28.554276 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-755453-cnmm7"] Oct 29 23:28:29.058660 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:29.058624 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:28:29.171555 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:29.171509 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:28:29.281112 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:29.281078 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:28:29.332298 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:29.332226 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7be7f1f-537b-4b1f-abd6-97727768c84c" path="/var/lib/kubelet/pods/f7be7f1f-537b-4b1f-abd6-97727768c84c/volumes" Oct 29 23:28:29.683356 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:29.683249 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:28:30.908741 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:30.908707 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:28:30.928347 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:30.928034 5057 generic.go:356] "Generic (PLEG): container finished" podID="6760c386-e49a-4d99-b811-31a8d9df5b41" containerID="5ec94988b8438146a3c75eb716cc8925ef57b4b532a947dcc1c377448579dc3d" exitCode=0 Oct 29 23:28:30.928347 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:30.928253 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-755453-9bd867db-hdlwb" event={"ID":"6760c386-e49a-4d99-b811-31a8d9df5b41","Type":"ContainerDied","Data":"5ec94988b8438146a3c75eb716cc8925ef57b4b532a947dcc1c377448579dc3d"} Oct 29 23:28:30.939189 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:30.939159 5057 generic.go:356] "Generic (PLEG): container finished" podID="0e08ea7b-6096-4253-a757-b092cf07bc7c" containerID="e15440c29d67c92cf76444e89c189c274e005fdcd089d174448a8814ee0c202a" exitCode=0 Oct 29 23:28:30.939385 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:30.939231 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-755453-788f5c78c-svk2k" event={"ID":"0e08ea7b-6096-4253-a757-b092cf07bc7c","Type":"ContainerDied","Data":"e15440c29d67c92cf76444e89c189c274e005fdcd089d174448a8814ee0c202a"} Oct 29 23:28:31.040751 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:31.040586 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-755453-9bd867db-hdlwb" Oct 29 23:28:31.064814 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:31.064784 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-755453-788f5c78c-svk2k" Oct 29 23:28:31.135177 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:31.135144 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e08ea7b-6096-4253-a757-b092cf07bc7c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e08ea7b-6096-4253-a757-b092cf07bc7c-plugins-config-volume\") pod \"0e08ea7b-6096-4253-a757-b092cf07bc7c\" (UID: \"0e08ea7b-6096-4253-a757-b092cf07bc7c\") " Oct 29 23:28:31.135395 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:31.135222 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6760c386-e49a-4d99-b811-31a8d9df5b41-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6760c386-e49a-4d99-b811-31a8d9df5b41-plugins-config-volume\") pod \"6760c386-e49a-4d99-b811-31a8d9df5b41\" (UID: \"6760c386-e49a-4d99-b811-31a8d9df5b41\") " Oct 29 23:28:31.135395 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:31.135276 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e08ea7b-6096-4253-a757-b092cf07bc7c-kube-api-access-d6qfs\" (UniqueName: \"kubernetes.io/projected/0e08ea7b-6096-4253-a757-b092cf07bc7c-kube-api-access-d6qfs\") pod \"0e08ea7b-6096-4253-a757-b092cf07bc7c\" (UID: \"0e08ea7b-6096-4253-a757-b092cf07bc7c\") " Oct 29 23:28:31.135395 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:31.135360 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6760c386-e49a-4d99-b811-31a8d9df5b41-kube-api-access-8wkb6\" (UniqueName: \"kubernetes.io/projected/6760c386-e49a-4d99-b811-31a8d9df5b41-kube-api-access-8wkb6\") pod \"6760c386-e49a-4d99-b811-31a8d9df5b41\" (UID: \"6760c386-e49a-4d99-b811-31a8d9df5b41\") " Oct 29 23:28:31.135595 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:31.135558 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e08ea7b-6096-4253-a757-b092cf07bc7c-plugins-config-volume" pod "0e08ea7b-6096-4253-a757-b092cf07bc7c" (UID: "0e08ea7b-6096-4253-a757-b092cf07bc7c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:28:31.135691 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:31.135663 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6760c386-e49a-4d99-b811-31a8d9df5b41-plugins-config-volume" pod "6760c386-e49a-4d99-b811-31a8d9df5b41" (UID: "6760c386-e49a-4d99-b811-31a8d9df5b41"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:28:31.135827 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:31.135809 5057 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e08ea7b-6096-4253-a757-b092cf07bc7c-plugins-config-volume\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:28:31.135827 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:31.135824 5057 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6760c386-e49a-4d99-b811-31a8d9df5b41-plugins-config-volume\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:28:31.137615 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:31.137589 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6760c386-e49a-4d99-b811-31a8d9df5b41-kube-api-access-8wkb6" pod "6760c386-e49a-4d99-b811-31a8d9df5b41" (UID: "6760c386-e49a-4d99-b811-31a8d9df5b41"). InnerVolumeSpecName "kube-api-access-8wkb6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:28:31.137704 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:31.137683 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e08ea7b-6096-4253-a757-b092cf07bc7c-kube-api-access-d6qfs" pod "0e08ea7b-6096-4253-a757-b092cf07bc7c" (UID: "0e08ea7b-6096-4253-a757-b092cf07bc7c"). InnerVolumeSpecName "kube-api-access-d6qfs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:28:31.237439 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:31.237408 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d6qfs\" (UniqueName: \"kubernetes.io/projected/0e08ea7b-6096-4253-a757-b092cf07bc7c-kube-api-access-d6qfs\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:28:31.237439 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:31.237434 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8wkb6\" (UniqueName: \"kubernetes.io/projected/6760c386-e49a-4d99-b811-31a8d9df5b41-kube-api-access-8wkb6\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:28:31.949722 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:31.949690 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-755453-788f5c78c-svk2k" Oct 29 23:28:31.950166 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:31.949720 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-755453-788f5c78c-svk2k" event={"ID":"0e08ea7b-6096-4253-a757-b092cf07bc7c","Type":"ContainerDied","Data":"8a777633783aa760a65bff963846cdd4b36a40dc54931b2ccebcc7b43c665d18"} Oct 29 23:28:31.950166 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:31.949780 5057 scope.go:122] "RemoveContainer" containerID="e15440c29d67c92cf76444e89c189c274e005fdcd089d174448a8814ee0c202a" Oct 29 23:28:31.951572 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:31.951553 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-755453-9bd867db-hdlwb" Oct 29 23:28:31.951693 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:31.951567 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-755453-9bd867db-hdlwb" event={"ID":"6760c386-e49a-4d99-b811-31a8d9df5b41","Type":"ContainerDied","Data":"a9b37d08537dcbeb0f9f61565f174653a49b789bcfa8e8c6d6d85224a9d8f047"} Oct 29 23:28:31.962376 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:31.962324 5057 scope.go:122] "RemoveContainer" containerID="5ec94988b8438146a3c75eb716cc8925ef57b4b532a947dcc1c377448579dc3d" Oct 29 23:28:32.007684 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:32.007655 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-755453-788f5c78c-svk2k"] Oct 29 23:28:32.018042 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:32.018013 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-755453-788f5c78c-svk2k"] Oct 29 23:28:32.047634 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:32.047604 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-755453-9bd867db-hdlwb"] Oct 29 23:28:32.058968 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:32.058930 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-755453-9bd867db-hdlwb"] Oct 29 23:28:32.514035 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:32.513988 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:28:33.330659 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:33.330623 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e08ea7b-6096-4253-a757-b092cf07bc7c" path="/var/lib/kubelet/pods/0e08ea7b-6096-4253-a757-b092cf07bc7c/volumes" Oct 29 23:28:33.331144 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:33.331128 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6760c386-e49a-4d99-b811-31a8d9df5b41" path="/var/lib/kubelet/pods/6760c386-e49a-4d99-b811-31a8d9df5b41/volumes" Oct 29 23:28:34.749356 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:34.749192 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:28:35.166124 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:35.165977 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w1-59eefa/rate-test-model-59eefa-kserve-5c6dcf6849-6wq8t"] Oct 29 23:28:35.168943 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:35.168899 5057 cpu_manager.go:414] "containerMap: removing container" podUID="e6c69267-32e0-4a27-a376-d9918d3f2e3b" containerName="payload-pre-processing" Oct 29 23:28:35.169110 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:35.169092 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6c69267-32e0-4a27-a376-d9918d3f2e3b" containerName="payload-pre-processing" Oct 29 23:28:35.169257 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:35.169243 5057 cpu_manager.go:414] "containerMap: removing container" podUID="074dc860-7008-4c4f-9c9e-0fe9059b1947" containerName="payload-processing" Oct 29 23:28:35.171969 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:35.171944 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="074dc860-7008-4c4f-9c9e-0fe9059b1947" containerName="payload-processing" Oct 29 23:28:35.172194 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:35.172177 5057 cpu_manager.go:414] "containerMap: removing container" podUID="268ae24d-3a42-472a-97d7-aeacd9989632" containerName="istio-proxy" Oct 29 23:28:35.172311 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:35.172296 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="268ae24d-3a42-472a-97d7-aeacd9989632" containerName="istio-proxy" Oct 29 23:28:35.172475 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:35.172458 5057 cpu_manager.go:414] "containerMap: removing container" podUID="0e08ea7b-6096-4253-a757-b092cf07bc7c" containerName="payload-pre-processing" Oct 29 23:28:35.172614 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:35.172587 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e08ea7b-6096-4253-a757-b092cf07bc7c" containerName="payload-pre-processing" Oct 29 23:28:35.213375 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:35.172739 5057 cpu_manager.go:414] "containerMap: removing container" podUID="3eb0219c-cf7b-4f66-b484-a78a2a5514a1" containerName="payload-pre-processing" Oct 29 23:28:35.213605 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:35.213577 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3eb0219c-cf7b-4f66-b484-a78a2a5514a1" containerName="payload-pre-processing" Oct 29 23:28:35.213730 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:35.213716 5057 cpu_manager.go:414] "containerMap: removing container" podUID="268ae24d-3a42-472a-97d7-aeacd9989632" containerName="istio-proxy" Oct 29 23:28:35.213827 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:35.213811 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="268ae24d-3a42-472a-97d7-aeacd9989632" containerName="istio-proxy" Oct 29 23:28:35.213930 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:35.213918 5057 cpu_manager.go:414] "containerMap: removing container" podUID="6760c386-e49a-4d99-b811-31a8d9df5b41" containerName="payload-processing" Oct 29 23:28:35.214015 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:35.214004 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6760c386-e49a-4d99-b811-31a8d9df5b41" containerName="payload-processing" Oct 29 23:28:35.214426 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:35.214403 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6c69267-32e0-4a27-a376-d9918d3f2e3b" containerName="payload-pre-processing" Oct 29 23:28:35.214563 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:35.214550 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="074dc860-7008-4c4f-9c9e-0fe9059b1947" containerName="payload-processing" Oct 29 23:28:35.214671 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:35.214659 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="268ae24d-3a42-472a-97d7-aeacd9989632" containerName="istio-proxy" Oct 29 23:28:35.260362 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:35.244480 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e08ea7b-6096-4253-a757-b092cf07bc7c" containerName="payload-pre-processing" Oct 29 23:28:35.260362 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:35.237279 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:28:35.260362 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:35.244982 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="3eb0219c-cf7b-4f66-b484-a78a2a5514a1" containerName="payload-pre-processing" Oct 29 23:28:35.260362 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:35.245003 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="268ae24d-3a42-472a-97d7-aeacd9989632" containerName="istio-proxy" Oct 29 23:28:35.260362 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:35.245013 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="6760c386-e49a-4d99-b811-31a8d9df5b41" containerName="payload-processing" Oct 29 23:28:35.373368 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:35.355415 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-59eefa/rate-test-model-59eefa-kserve-5c6dcf6849-6wq8t"] Oct 29 23:28:35.373368 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:35.360347 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-59eefa/rate-test-model-59eefa-kserve-5c6dcf6849-6wq8t" Oct 29 23:28:35.433446 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:35.409972 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-59eefa\"/\"rate-test-model-59eefa-kserve-self-signed-certs\"" Oct 29 23:28:35.499361 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:35.483491 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ae48191-676a-43fa-9ffd-a872f474e461-tmp-dir\") pod \"rate-test-model-59eefa-kserve-5c6dcf6849-6wq8t\" (UID: \"4ae48191-676a-43fa-9ffd-a872f474e461\") " pod="ai-tenant-e2e-shared-a-w1-59eefa/rate-test-model-59eefa-kserve-5c6dcf6849-6wq8t" Oct 29 23:28:35.499361 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:35.483568 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ae48191-676a-43fa-9ffd-a872f474e461-home\") pod \"rate-test-model-59eefa-kserve-5c6dcf6849-6wq8t\" (UID: \"4ae48191-676a-43fa-9ffd-a872f474e461\") " pod="ai-tenant-e2e-shared-a-w1-59eefa/rate-test-model-59eefa-kserve-5c6dcf6849-6wq8t" Oct 29 23:28:35.499361 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:35.483676 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ae48191-676a-43fa-9ffd-a872f474e461-tls-certs\") pod \"rate-test-model-59eefa-kserve-5c6dcf6849-6wq8t\" (UID: \"4ae48191-676a-43fa-9ffd-a872f474e461\") " pod="ai-tenant-e2e-shared-a-w1-59eefa/rate-test-model-59eefa-kserve-5c6dcf6849-6wq8t" Oct 29 23:28:35.499361 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:35.483757 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ae48191-676a-43fa-9ffd-a872f474e461-model-cache\") pod \"rate-test-model-59eefa-kserve-5c6dcf6849-6wq8t\" (UID: \"4ae48191-676a-43fa-9ffd-a872f474e461\") " pod="ai-tenant-e2e-shared-a-w1-59eefa/rate-test-model-59eefa-kserve-5c6dcf6849-6wq8t" Oct 29 23:28:35.499361 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:35.483842 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jd6qt\" (UniqueName: \"kubernetes.io/projected/4ae48191-676a-43fa-9ffd-a872f474e461-kube-api-access-jd6qt\") pod \"rate-test-model-59eefa-kserve-5c6dcf6849-6wq8t\" (UID: \"4ae48191-676a-43fa-9ffd-a872f474e461\") " pod="ai-tenant-e2e-shared-a-w1-59eefa/rate-test-model-59eefa-kserve-5c6dcf6849-6wq8t" Oct 29 23:28:35.499361 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:35.483960 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ae48191-676a-43fa-9ffd-a872f474e461-dshm\") pod \"rate-test-model-59eefa-kserve-5c6dcf6849-6wq8t\" (UID: \"4ae48191-676a-43fa-9ffd-a872f474e461\") " pod="ai-tenant-e2e-shared-a-w1-59eefa/rate-test-model-59eefa-kserve-5c6dcf6849-6wq8t" Oct 29 23:28:35.599746 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:35.599706 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jd6qt\" (UniqueName: \"kubernetes.io/projected/4ae48191-676a-43fa-9ffd-a872f474e461-kube-api-access-jd6qt\") pod \"rate-test-model-59eefa-kserve-5c6dcf6849-6wq8t\" (UID: \"4ae48191-676a-43fa-9ffd-a872f474e461\") " pod="ai-tenant-e2e-shared-a-w1-59eefa/rate-test-model-59eefa-kserve-5c6dcf6849-6wq8t" Oct 29 23:28:35.610355 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:35.600089 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ae48191-676a-43fa-9ffd-a872f474e461-dshm\") pod \"rate-test-model-59eefa-kserve-5c6dcf6849-6wq8t\" (UID: \"4ae48191-676a-43fa-9ffd-a872f474e461\") " pod="ai-tenant-e2e-shared-a-w1-59eefa/rate-test-model-59eefa-kserve-5c6dcf6849-6wq8t" Oct 29 23:28:35.610603 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:35.610579 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ae48191-676a-43fa-9ffd-a872f474e461-tmp-dir\") pod \"rate-test-model-59eefa-kserve-5c6dcf6849-6wq8t\" (UID: \"4ae48191-676a-43fa-9ffd-a872f474e461\") " pod="ai-tenant-e2e-shared-a-w1-59eefa/rate-test-model-59eefa-kserve-5c6dcf6849-6wq8t" Oct 29 23:28:35.610781 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:35.610762 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ae48191-676a-43fa-9ffd-a872f474e461-home\") pod \"rate-test-model-59eefa-kserve-5c6dcf6849-6wq8t\" (UID: \"4ae48191-676a-43fa-9ffd-a872f474e461\") " pod="ai-tenant-e2e-shared-a-w1-59eefa/rate-test-model-59eefa-kserve-5c6dcf6849-6wq8t" Oct 29 23:28:35.611021 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:35.611000 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ae48191-676a-43fa-9ffd-a872f474e461-tls-certs\") pod \"rate-test-model-59eefa-kserve-5c6dcf6849-6wq8t\" (UID: \"4ae48191-676a-43fa-9ffd-a872f474e461\") " pod="ai-tenant-e2e-shared-a-w1-59eefa/rate-test-model-59eefa-kserve-5c6dcf6849-6wq8t" Oct 29 23:28:35.611210 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:35.611191 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ae48191-676a-43fa-9ffd-a872f474e461-model-cache\") pod \"rate-test-model-59eefa-kserve-5c6dcf6849-6wq8t\" (UID: \"4ae48191-676a-43fa-9ffd-a872f474e461\") " pod="ai-tenant-e2e-shared-a-w1-59eefa/rate-test-model-59eefa-kserve-5c6dcf6849-6wq8t" Oct 29 23:28:35.611801 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:35.611776 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ae48191-676a-43fa-9ffd-a872f474e461-model-cache\") pod \"rate-test-model-59eefa-kserve-5c6dcf6849-6wq8t\" (UID: \"4ae48191-676a-43fa-9ffd-a872f474e461\") " pod="ai-tenant-e2e-shared-a-w1-59eefa/rate-test-model-59eefa-kserve-5c6dcf6849-6wq8t" Oct 29 23:28:35.611953 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:35.605896 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-59eefa\"/\"kube-root-ca.crt\"" Oct 29 23:28:35.632443 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:35.625630 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ae48191-676a-43fa-9ffd-a872f474e461-tmp-dir\") pod \"rate-test-model-59eefa-kserve-5c6dcf6849-6wq8t\" (UID: \"4ae48191-676a-43fa-9ffd-a872f474e461\") " pod="ai-tenant-e2e-shared-a-w1-59eefa/rate-test-model-59eefa-kserve-5c6dcf6849-6wq8t" Oct 29 23:28:35.632625 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:35.606481 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-59eefa\"/\"default-dockercfg-9f9pf\"" Oct 29 23:28:35.632728 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:35.625837 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ae48191-676a-43fa-9ffd-a872f474e461-home\") pod \"rate-test-model-59eefa-kserve-5c6dcf6849-6wq8t\" (UID: \"4ae48191-676a-43fa-9ffd-a872f474e461\") " pod="ai-tenant-e2e-shared-a-w1-59eefa/rate-test-model-59eefa-kserve-5c6dcf6849-6wq8t" Oct 29 23:28:35.632823 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:35.606546 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-59eefa\"/\"openshift-service-ca.crt\"" Oct 29 23:28:35.736865 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:35.679233 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jd6qt\" (UniqueName: \"kubernetes.io/projected/4ae48191-676a-43fa-9ffd-a872f474e461-kube-api-access-jd6qt\") pod \"rate-test-model-59eefa-kserve-5c6dcf6849-6wq8t\" (UID: \"4ae48191-676a-43fa-9ffd-a872f474e461\") " pod="ai-tenant-e2e-shared-a-w1-59eefa/rate-test-model-59eefa-kserve-5c6dcf6849-6wq8t" Oct 29 23:28:35.736865 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:35.685074 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ae48191-676a-43fa-9ffd-a872f474e461-tls-certs\") pod \"rate-test-model-59eefa-kserve-5c6dcf6849-6wq8t\" (UID: \"4ae48191-676a-43fa-9ffd-a872f474e461\") " pod="ai-tenant-e2e-shared-a-w1-59eefa/rate-test-model-59eefa-kserve-5c6dcf6849-6wq8t" Oct 29 23:28:35.770461 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:35.770396 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ae48191-676a-43fa-9ffd-a872f474e461-dshm\") pod \"rate-test-model-59eefa-kserve-5c6dcf6849-6wq8t\" (UID: \"4ae48191-676a-43fa-9ffd-a872f474e461\") " pod="ai-tenant-e2e-shared-a-w1-59eefa/rate-test-model-59eefa-kserve-5c6dcf6849-6wq8t" Oct 29 23:28:35.970381 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:35.964405 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-59eefa/rate-test-model-59eefa-kserve-5c6dcf6849-6wq8t" Oct 29 23:28:36.210354 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:36.206534 5057 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-cm4wj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 23:28:36.210354 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:36.206595 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-cm4wj" podUID="4e861691-eac4-4f06-a6aa-b6553efdd358" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 23:28:36.379795 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:36.379553 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:28:36.503045 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:36.503007 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:28:36.557118 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:36.557090 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-59eefa/rate-test-model-59eefa-kserve-5c6dcf6849-6wq8t"] Oct 29 23:28:36.557934 ip-10-0-140-74 kubenswrapper[5057]: W1029 23:28:36.557902 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4ae48191_676a_43fa_9ffd_a872f474e461.slice/crio-c8387bd6be7f0c8c65dd8efacb609fa6350c51d2f77ce239436fd4c46b3cfbf5 WatchSource:0}: Error finding container c8387bd6be7f0c8c65dd8efacb609fa6350c51d2f77ce239436fd4c46b3cfbf5: Status 404 returned error can't find the container with id c8387bd6be7f0c8c65dd8efacb609fa6350c51d2f77ce239436fd4c46b3cfbf5 Oct 29 23:28:36.614438 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:36.614392 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:28:36.840620 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:36.840525 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:28:37.184717 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:37.183164 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-59eefa/rate-test-model-59eefa-kserve-5c6dcf6849-6wq8t" event={"ID":"4ae48191-676a-43fa-9ffd-a872f474e461","Type":"ContainerStarted","Data":"59f1fedccc4f40fb83438854918fcfcb89a24a8243973e1038659a602782a9e1"} Oct 29 23:28:37.184717 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:37.183222 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-59eefa/rate-test-model-59eefa-kserve-5c6dcf6849-6wq8t" event={"ID":"4ae48191-676a-43fa-9ffd-a872f474e461","Type":"ContainerStarted","Data":"c8387bd6be7f0c8c65dd8efacb609fa6350c51d2f77ce239436fd4c46b3cfbf5"} Oct 29 23:28:37.225791 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:37.225738 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w1-59eefa/rate-test-model-59eefa-kserve-5c6dcf6849-6wq8t" podStartSLOduration=2.225724253 podStartE2EDuration="2.225724253s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:28:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:28:37.22395273 +0000 UTC m=+3298.007166158" watchObservedRunningTime="2026-10-29 23:28:37.225724253 +0000 UTC m=+3298.008937678" Oct 29 23:28:37.369202 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:37.369167 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:28:37.488113 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:37.488074 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:28:37.965113 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:37.965081 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w1-59eefa/rate-test-model-59eefa-kserve-5c6dcf6849-6wq8t" Oct 29 23:28:38.136640 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:38.136605 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:28:38.303010 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:38.302920 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:28:38.506362 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:38.506289 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:28:39.005242 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:39.005204 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:28:39.292875 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:39.292794 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:28:40.087362 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:40.087314 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-vj6md_aa6a190e-3283-458a-9dfd-2de7cb193fc5/console-operator/2.log" Oct 29 23:28:40.100020 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:40.099997 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-vj6md_aa6a190e-3283-458a-9dfd-2de7cb193fc5/console-operator/2.log" Oct 29 23:28:40.809770 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:40.809725 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:28:41.462147 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:41.462110 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:28:41.627281 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:41.627229 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:28:41.736382 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:41.736271 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:28:42.146226 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:42.146137 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:28:42.390977 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:42.390942 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-17394a/distinct-2-w3-17394a-kserve-6545c766c4-h4chf"] Oct 29 23:28:42.517412 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:42.517356 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:28:43.674263 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:43.674219 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:28:43.971995 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:43.971963 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-31226c/distinct-w6-31226c-kserve-858d95fd76-bkf7c"] Oct 29 23:28:44.037219 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:44.037187 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e587514-z6mqr"] Oct 29 23:28:44.041120 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:44.041094 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e587514-z6mqr"] Oct 29 23:28:44.094311 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:44.094275 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:28:45.313413 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:45.313370 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:28:45.331195 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:45.331161 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2061553-dc6a-44aa-bfa1-02390bd63b8c" path="/var/lib/kubelet/pods/b2061553-dc6a-44aa-bfa1-02390bd63b8c/volumes" Oct 29 23:28:45.964640 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:45.964605 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w1-59eefa/rate-test-model-59eefa-kserve-5c6dcf6849-6wq8t" Oct 29 23:28:45.977255 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:45.977228 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w1-59eefa/rate-test-model-59eefa-kserve-5c6dcf6849-6wq8t" Oct 29 23:28:46.316499 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:46.316420 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w1-59eefa/rate-test-model-59eefa-kserve-5c6dcf6849-6wq8t" Oct 29 23:28:46.885091 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:46.885052 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:28:48.133819 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:48.133783 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-17394a-77794c47ff-djggj"] Oct 29 23:28:48.258097 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:48.258035 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:28:48.395415 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:48.382127 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:28:48.491591 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:48.491502 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:28:49.030940 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:28:49.030906 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d2dnn" containerName="extractor" Oct 29 23:28:49.072428 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:49.072388 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:28:49.184058 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:49.184013 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:28:49.430905 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:49.430823 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:28:49.742155 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:49.742123 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-31226c-84fb67d984-mpgbh"] Oct 29 23:28:49.778912 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:49.778876 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-31226c-8475b558c9-2v6n6"] Oct 29 23:28:49.893777 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:49.893743 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:28:50.135118 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:50.135026 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:28:50.254277 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:50.254237 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:28:51.423910 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:51.423850 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w1-e170ed/rate-test-model-e170ed-kserve-544667c877-hc2sg"] Oct 29 23:28:51.430248 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:51.430221 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-e170ed/rate-test-model-e170ed-kserve-544667c877-hc2sg" Oct 29 23:28:51.444851 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:51.444819 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-e170ed/rate-test-model-e170ed-kserve-544667c877-hc2sg"] Oct 29 23:28:51.451517 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:51.451147 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-e170ed\"/\"rate-test-model-e170ed-kserve-self-signed-certs\"" Oct 29 23:28:51.451517 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:51.451376 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-e170ed\"/\"default-dockercfg-24dcs\"" Oct 29 23:28:51.461379 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:51.457527 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-e170ed\"/\"openshift-service-ca.crt\"" Oct 29 23:28:51.461379 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:51.457540 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-e170ed\"/\"kube-root-ca.crt\"" Oct 29 23:28:51.472435 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:51.472393 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:28:51.561312 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:51.561281 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c19bfd1-9bdf-48e2-9033-29a2f02ef5b2-home\") pod \"rate-test-model-e170ed-kserve-544667c877-hc2sg\" (UID: \"2c19bfd1-9bdf-48e2-9033-29a2f02ef5b2\") " pod="ai-tenant-e2e-shared-b-w1-e170ed/rate-test-model-e170ed-kserve-544667c877-hc2sg" Oct 29 23:28:51.561513 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:51.561462 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pcqrc\" (UniqueName: \"kubernetes.io/projected/2c19bfd1-9bdf-48e2-9033-29a2f02ef5b2-kube-api-access-pcqrc\") pod \"rate-test-model-e170ed-kserve-544667c877-hc2sg\" (UID: \"2c19bfd1-9bdf-48e2-9033-29a2f02ef5b2\") " pod="ai-tenant-e2e-shared-b-w1-e170ed/rate-test-model-e170ed-kserve-544667c877-hc2sg" Oct 29 23:28:51.561580 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:51.561511 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c19bfd1-9bdf-48e2-9033-29a2f02ef5b2-model-cache\") pod \"rate-test-model-e170ed-kserve-544667c877-hc2sg\" (UID: \"2c19bfd1-9bdf-48e2-9033-29a2f02ef5b2\") " pod="ai-tenant-e2e-shared-b-w1-e170ed/rate-test-model-e170ed-kserve-544667c877-hc2sg" Oct 29 23:28:51.561580 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:51.561537 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c19bfd1-9bdf-48e2-9033-29a2f02ef5b2-tls-certs\") pod \"rate-test-model-e170ed-kserve-544667c877-hc2sg\" (UID: \"2c19bfd1-9bdf-48e2-9033-29a2f02ef5b2\") " pod="ai-tenant-e2e-shared-b-w1-e170ed/rate-test-model-e170ed-kserve-544667c877-hc2sg" Oct 29 23:28:51.561692 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:51.561658 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c19bfd1-9bdf-48e2-9033-29a2f02ef5b2-tmp-dir\") pod \"rate-test-model-e170ed-kserve-544667c877-hc2sg\" (UID: \"2c19bfd1-9bdf-48e2-9033-29a2f02ef5b2\") " pod="ai-tenant-e2e-shared-b-w1-e170ed/rate-test-model-e170ed-kserve-544667c877-hc2sg" Oct 29 23:28:51.561757 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:51.561728 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c19bfd1-9bdf-48e2-9033-29a2f02ef5b2-dshm\") pod \"rate-test-model-e170ed-kserve-544667c877-hc2sg\" (UID: \"2c19bfd1-9bdf-48e2-9033-29a2f02ef5b2\") " pod="ai-tenant-e2e-shared-b-w1-e170ed/rate-test-model-e170ed-kserve-544667c877-hc2sg" Oct 29 23:28:51.673361 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:51.663492 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c19bfd1-9bdf-48e2-9033-29a2f02ef5b2-home\") pod \"rate-test-model-e170ed-kserve-544667c877-hc2sg\" (UID: \"2c19bfd1-9bdf-48e2-9033-29a2f02ef5b2\") " pod="ai-tenant-e2e-shared-b-w1-e170ed/rate-test-model-e170ed-kserve-544667c877-hc2sg" Oct 29 23:28:51.673361 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:51.663672 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pcqrc\" (UniqueName: \"kubernetes.io/projected/2c19bfd1-9bdf-48e2-9033-29a2f02ef5b2-kube-api-access-pcqrc\") pod \"rate-test-model-e170ed-kserve-544667c877-hc2sg\" (UID: \"2c19bfd1-9bdf-48e2-9033-29a2f02ef5b2\") " pod="ai-tenant-e2e-shared-b-w1-e170ed/rate-test-model-e170ed-kserve-544667c877-hc2sg" Oct 29 23:28:51.673361 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:51.663737 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c19bfd1-9bdf-48e2-9033-29a2f02ef5b2-model-cache\") pod \"rate-test-model-e170ed-kserve-544667c877-hc2sg\" (UID: \"2c19bfd1-9bdf-48e2-9033-29a2f02ef5b2\") " pod="ai-tenant-e2e-shared-b-w1-e170ed/rate-test-model-e170ed-kserve-544667c877-hc2sg" Oct 29 23:28:51.673361 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:51.663762 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c19bfd1-9bdf-48e2-9033-29a2f02ef5b2-tls-certs\") pod \"rate-test-model-e170ed-kserve-544667c877-hc2sg\" (UID: \"2c19bfd1-9bdf-48e2-9033-29a2f02ef5b2\") " pod="ai-tenant-e2e-shared-b-w1-e170ed/rate-test-model-e170ed-kserve-544667c877-hc2sg" Oct 29 23:28:51.673361 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:51.663841 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c19bfd1-9bdf-48e2-9033-29a2f02ef5b2-tmp-dir\") pod \"rate-test-model-e170ed-kserve-544667c877-hc2sg\" (UID: \"2c19bfd1-9bdf-48e2-9033-29a2f02ef5b2\") " pod="ai-tenant-e2e-shared-b-w1-e170ed/rate-test-model-e170ed-kserve-544667c877-hc2sg" Oct 29 23:28:51.673361 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:51.663942 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c19bfd1-9bdf-48e2-9033-29a2f02ef5b2-dshm\") pod \"rate-test-model-e170ed-kserve-544667c877-hc2sg\" (UID: \"2c19bfd1-9bdf-48e2-9033-29a2f02ef5b2\") " pod="ai-tenant-e2e-shared-b-w1-e170ed/rate-test-model-e170ed-kserve-544667c877-hc2sg" Oct 29 23:28:51.673361 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:51.667294 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c19bfd1-9bdf-48e2-9033-29a2f02ef5b2-model-cache\") pod \"rate-test-model-e170ed-kserve-544667c877-hc2sg\" (UID: \"2c19bfd1-9bdf-48e2-9033-29a2f02ef5b2\") " pod="ai-tenant-e2e-shared-b-w1-e170ed/rate-test-model-e170ed-kserve-544667c877-hc2sg" Oct 29 23:28:51.673361 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:51.667657 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c19bfd1-9bdf-48e2-9033-29a2f02ef5b2-tmp-dir\") pod \"rate-test-model-e170ed-kserve-544667c877-hc2sg\" (UID: \"2c19bfd1-9bdf-48e2-9033-29a2f02ef5b2\") " pod="ai-tenant-e2e-shared-b-w1-e170ed/rate-test-model-e170ed-kserve-544667c877-hc2sg" Oct 29 23:28:51.673361 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:51.667948 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c19bfd1-9bdf-48e2-9033-29a2f02ef5b2-home\") pod \"rate-test-model-e170ed-kserve-544667c877-hc2sg\" (UID: \"2c19bfd1-9bdf-48e2-9033-29a2f02ef5b2\") " pod="ai-tenant-e2e-shared-b-w1-e170ed/rate-test-model-e170ed-kserve-544667c877-hc2sg" Oct 29 23:28:51.673361 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:51.672493 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c19bfd1-9bdf-48e2-9033-29a2f02ef5b2-dshm\") pod \"rate-test-model-e170ed-kserve-544667c877-hc2sg\" (UID: \"2c19bfd1-9bdf-48e2-9033-29a2f02ef5b2\") " pod="ai-tenant-e2e-shared-b-w1-e170ed/rate-test-model-e170ed-kserve-544667c877-hc2sg" Oct 29 23:28:51.686400 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:51.679605 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c19bfd1-9bdf-48e2-9033-29a2f02ef5b2-tls-certs\") pod \"rate-test-model-e170ed-kserve-544667c877-hc2sg\" (UID: \"2c19bfd1-9bdf-48e2-9033-29a2f02ef5b2\") " pod="ai-tenant-e2e-shared-b-w1-e170ed/rate-test-model-e170ed-kserve-544667c877-hc2sg" Oct 29 23:28:51.705007 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:51.698173 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pcqrc\" (UniqueName: \"kubernetes.io/projected/2c19bfd1-9bdf-48e2-9033-29a2f02ef5b2-kube-api-access-pcqrc\") pod \"rate-test-model-e170ed-kserve-544667c877-hc2sg\" (UID: \"2c19bfd1-9bdf-48e2-9033-29a2f02ef5b2\") " pod="ai-tenant-e2e-shared-b-w1-e170ed/rate-test-model-e170ed-kserve-544667c877-hc2sg" Oct 29 23:28:51.750847 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:51.750814 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-e170ed/rate-test-model-e170ed-kserve-544667c877-hc2sg" Oct 29 23:28:52.006817 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:52.006780 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:28:52.073095 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:52.073060 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-e170ed/rate-test-model-e170ed-kserve-544667c877-hc2sg"] Oct 29 23:28:52.379802 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:52.379689 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-e170ed/rate-test-model-e170ed-kserve-544667c877-hc2sg" event={"ID":"2c19bfd1-9bdf-48e2-9033-29a2f02ef5b2","Type":"ContainerStarted","Data":"a7efd1d116ad6b79adbb884f9cd34b050d1882d0d5e230c3fdae27a7272e2378"} Oct 29 23:28:52.379802 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:52.379740 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-e170ed/rate-test-model-e170ed-kserve-544667c877-hc2sg" event={"ID":"2c19bfd1-9bdf-48e2-9033-29a2f02ef5b2","Type":"ContainerStarted","Data":"d2d8e774dc4ffdb5316eeca6f521385df9258cab28ceb132a73d6f44b85ffd3e"} Oct 29 23:28:52.426877 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:52.426812 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w1-e170ed/rate-test-model-e170ed-kserve-544667c877-hc2sg" podStartSLOduration=1.426797447 podStartE2EDuration="1.426797447s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:28:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:28:52.420221301 +0000 UTC m=+3313.203434730" watchObservedRunningTime="2026-10-29 23:28:52.426797447 +0000 UTC m=+3313.210010885" Oct 29 23:28:52.750914 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:52.750884 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w1-e170ed/rate-test-model-e170ed-kserve-544667c877-hc2sg" Oct 29 23:28:53.134651 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:53.134566 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-17394a-77794c47ff-djggj" podUID="5917154d-50a3-4166-94e0-533aef08f790" containerName="payload-pre-processing" containerID="cri-o://f127c13a5f2b483e7e49b6a7ffb9d50ce77abeb89c03b2cc1d5971c9ccce63a7" gracePeriod=25 Oct 29 23:28:53.688443 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:53.688405 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:28:54.743433 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:54.743396 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w6-31226c-84fb67d984-mpgbh" podUID="3481d9c3-f557-4b9d-b0f2-b0c0bdb81bd9" containerName="payload-processing" containerID="cri-o://59beb4040f5ec60491fecb6584e0ae3b32213333c9f4f1ff0c302a0ecfdd7280" gracePeriod=25 Oct 29 23:28:54.780745 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:54.780690 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-31226c-8475b558c9-2v6n6" podUID="8d4b783f-0071-4524-804e-396c53f069eb" containerName="payload-pre-processing" containerID="cri-o://ec85848b77e7759edeb94a5140c9aca054dfe8421fd1f8d0c1be351cae3a83b1" gracePeriod=25 Oct 29 23:28:55.033020 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:55.032931 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:28:55.335487 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:55.335408 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:28:56.936878 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:56.936834 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:28:57.125873 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:57.125833 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-17394a-77794c47ff-djggj" podUID="5917154d-50a3-4166-94e0-533aef08f790" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:28:57.414774 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:57.414730 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-17394a/distinct-2-w3-17394a-kserve-6545c766c4-h4chf" podUID="f29e8b65-b7dd-45d4-86b0-f13a29ef8e66" containerName="main" containerID="cri-o://534283814de5a5f1bc34e8b2c5b3f11ed7f463f3e605be39cc7734627e01f942" gracePeriod=45 Oct 29 23:28:57.718766 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:57.718737 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-17394a/distinct-2-w3-17394a-kserve-6545c766c4-h4chf" Oct 29 23:28:57.857120 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:57.857081 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f29e8b65-b7dd-45d4-86b0-f13a29ef8e66-kube-api-access-86s6t\" (UniqueName: \"kubernetes.io/projected/f29e8b65-b7dd-45d4-86b0-f13a29ef8e66-kube-api-access-86s6t\") pod \"f29e8b65-b7dd-45d4-86b0-f13a29ef8e66\" (UID: \"f29e8b65-b7dd-45d4-86b0-f13a29ef8e66\") " Oct 29 23:28:57.857401 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:57.857375 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f29e8b65-b7dd-45d4-86b0-f13a29ef8e66-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f29e8b65-b7dd-45d4-86b0-f13a29ef8e66-model-cache\") pod \"f29e8b65-b7dd-45d4-86b0-f13a29ef8e66\" (UID: \"f29e8b65-b7dd-45d4-86b0-f13a29ef8e66\") " Oct 29 23:28:57.857538 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:57.857472 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f29e8b65-b7dd-45d4-86b0-f13a29ef8e66-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f29e8b65-b7dd-45d4-86b0-f13a29ef8e66-dshm\") pod \"f29e8b65-b7dd-45d4-86b0-f13a29ef8e66\" (UID: \"f29e8b65-b7dd-45d4-86b0-f13a29ef8e66\") " Oct 29 23:28:57.857601 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:57.857536 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f29e8b65-b7dd-45d4-86b0-f13a29ef8e66-home\" (UniqueName: \"kubernetes.io/empty-dir/f29e8b65-b7dd-45d4-86b0-f13a29ef8e66-home\") pod \"f29e8b65-b7dd-45d4-86b0-f13a29ef8e66\" (UID: \"f29e8b65-b7dd-45d4-86b0-f13a29ef8e66\") " Oct 29 23:28:57.857601 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:57.857579 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f29e8b65-b7dd-45d4-86b0-f13a29ef8e66-tls-certs\" (UniqueName: \"kubernetes.io/secret/f29e8b65-b7dd-45d4-86b0-f13a29ef8e66-tls-certs\") pod \"f29e8b65-b7dd-45d4-86b0-f13a29ef8e66\" (UID: \"f29e8b65-b7dd-45d4-86b0-f13a29ef8e66\") " Oct 29 23:28:57.857716 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:57.857644 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f29e8b65-b7dd-45d4-86b0-f13a29ef8e66-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f29e8b65-b7dd-45d4-86b0-f13a29ef8e66-tmp-dir\") pod \"f29e8b65-b7dd-45d4-86b0-f13a29ef8e66\" (UID: \"f29e8b65-b7dd-45d4-86b0-f13a29ef8e66\") " Oct 29 23:28:57.858773 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:57.858601 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f29e8b65-b7dd-45d4-86b0-f13a29ef8e66-home" pod "f29e8b65-b7dd-45d4-86b0-f13a29ef8e66" (UID: "f29e8b65-b7dd-45d4-86b0-f13a29ef8e66"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:28:57.858773 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:57.858702 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f29e8b65-b7dd-45d4-86b0-f13a29ef8e66-model-cache" pod "f29e8b65-b7dd-45d4-86b0-f13a29ef8e66" (UID: "f29e8b65-b7dd-45d4-86b0-f13a29ef8e66"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:28:57.858773 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:57.858761 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f29e8b65-b7dd-45d4-86b0-f13a29ef8e66-tmp-dir" pod "f29e8b65-b7dd-45d4-86b0-f13a29ef8e66" (UID: "f29e8b65-b7dd-45d4-86b0-f13a29ef8e66"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:28:57.860105 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:57.860076 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f29e8b65-b7dd-45d4-86b0-f13a29ef8e66-kube-api-access-86s6t" pod "f29e8b65-b7dd-45d4-86b0-f13a29ef8e66" (UID: "f29e8b65-b7dd-45d4-86b0-f13a29ef8e66"). InnerVolumeSpecName "kube-api-access-86s6t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:28:57.860585 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:57.860558 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f29e8b65-b7dd-45d4-86b0-f13a29ef8e66-dshm" pod "f29e8b65-b7dd-45d4-86b0-f13a29ef8e66" (UID: "f29e8b65-b7dd-45d4-86b0-f13a29ef8e66"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:28:57.862497 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:57.862471 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f29e8b65-b7dd-45d4-86b0-f13a29ef8e66-tls-certs" pod "f29e8b65-b7dd-45d4-86b0-f13a29ef8e66" (UID: "f29e8b65-b7dd-45d4-86b0-f13a29ef8e66"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:28:57.959825 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:57.959784 5057 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f29e8b65-b7dd-45d4-86b0-f13a29ef8e66-model-cache\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:28:57.959825 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:57.959821 5057 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f29e8b65-b7dd-45d4-86b0-f13a29ef8e66-dshm\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:28:57.959825 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:57.959833 5057 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f29e8b65-b7dd-45d4-86b0-f13a29ef8e66-home\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:28:57.960294 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:57.959844 5057 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f29e8b65-b7dd-45d4-86b0-f13a29ef8e66-tls-certs\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:28:57.960294 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:57.959855 5057 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f29e8b65-b7dd-45d4-86b0-f13a29ef8e66-tmp-dir\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:28:57.960294 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:57.959866 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-86s6t\" (UniqueName: \"kubernetes.io/projected/f29e8b65-b7dd-45d4-86b0-f13a29ef8e66-kube-api-access-86s6t\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:28:58.036937 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:58.036900 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:28:58.441898 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:58.441805 5057 generic.go:356] "Generic (PLEG): container finished" podID="f29e8b65-b7dd-45d4-86b0-f13a29ef8e66" containerID="534283814de5a5f1bc34e8b2c5b3f11ed7f463f3e605be39cc7734627e01f942" exitCode=0 Oct 29 23:28:58.441898 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:58.441851 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-17394a/distinct-2-w3-17394a-kserve-6545c766c4-h4chf" Oct 29 23:28:58.441898 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:58.441886 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-17394a/distinct-2-w3-17394a-kserve-6545c766c4-h4chf" event={"ID":"f29e8b65-b7dd-45d4-86b0-f13a29ef8e66","Type":"ContainerDied","Data":"534283814de5a5f1bc34e8b2c5b3f11ed7f463f3e605be39cc7734627e01f942"} Oct 29 23:28:58.442186 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:58.441931 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-17394a/distinct-2-w3-17394a-kserve-6545c766c4-h4chf" event={"ID":"f29e8b65-b7dd-45d4-86b0-f13a29ef8e66","Type":"ContainerDied","Data":"11f5dd07350fdc84aaa620b23f82eb4f62c3264b38de76fad9c0a79fd9e4207c"} Oct 29 23:28:58.442186 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:58.441951 5057 scope.go:122] "RemoveContainer" containerID="534283814de5a5f1bc34e8b2c5b3f11ed7f463f3e605be39cc7734627e01f942" Oct 29 23:28:58.455050 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:58.455031 5057 scope.go:122] "RemoveContainer" containerID="534283814de5a5f1bc34e8b2c5b3f11ed7f463f3e605be39cc7734627e01f942" Oct 29 23:28:58.455308 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:28:58.455286 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"534283814de5a5f1bc34e8b2c5b3f11ed7f463f3e605be39cc7734627e01f942\": container with ID starting with 534283814de5a5f1bc34e8b2c5b3f11ed7f463f3e605be39cc7734627e01f942 not found: ID does not exist" containerID="534283814de5a5f1bc34e8b2c5b3f11ed7f463f3e605be39cc7734627e01f942" Oct 29 23:28:58.455381 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:58.455316 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"534283814de5a5f1bc34e8b2c5b3f11ed7f463f3e605be39cc7734627e01f942"} err="failed to get container status \"534283814de5a5f1bc34e8b2c5b3f11ed7f463f3e605be39cc7734627e01f942\": rpc error: code = NotFound desc = could not find container \"534283814de5a5f1bc34e8b2c5b3f11ed7f463f3e605be39cc7734627e01f942\": container with ID starting with 534283814de5a5f1bc34e8b2c5b3f11ed7f463f3e605be39cc7734627e01f942 not found: ID does not exist" Oct 29 23:28:58.498124 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:58.498098 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-17394a/distinct-2-w3-17394a-kserve-6545c766c4-h4chf"] Oct 29 23:28:58.507221 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:58.507199 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-17394a/distinct-2-w3-17394a-kserve-6545c766c4-h4chf"] Oct 29 23:28:58.618349 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:58.618301 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:28:58.995702 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:58.995665 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-31226c/distinct-w6-31226c-kserve-858d95fd76-bkf7c" podUID="f3fc11c6-03c9-4314-99c3-f857b149415c" containerName="main" containerID="cri-o://ba7af072d51cf9d892a2aff973911964540cba2c84ebbb4916b5d0d2bd97e8f7" gracePeriod=45 Oct 29 23:28:59.031299 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:59.031274 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb829d96-9l8hz"] Oct 29 23:28:59.042109 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:59.042083 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb829d96-9l8hz"] Oct 29 23:28:59.289896 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:59.289870 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-31226c/distinct-w6-31226c-kserve-858d95fd76-bkf7c" Oct 29 23:28:59.331890 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:59.331856 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7cff5a32-f779-4aa6-98c2-7b8ff21991c5" path="/var/lib/kubelet/pods/7cff5a32-f779-4aa6-98c2-7b8ff21991c5/volumes" Oct 29 23:28:59.332614 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:59.332593 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f29e8b65-b7dd-45d4-86b0-f13a29ef8e66" path="/var/lib/kubelet/pods/f29e8b65-b7dd-45d4-86b0-f13a29ef8e66/volumes" Oct 29 23:28:59.386894 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:59.386844 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3fc11c6-03c9-4314-99c3-f857b149415c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3fc11c6-03c9-4314-99c3-f857b149415c-tmp-dir\") pod \"f3fc11c6-03c9-4314-99c3-f857b149415c\" (UID: \"f3fc11c6-03c9-4314-99c3-f857b149415c\") " Oct 29 23:28:59.387135 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:59.386916 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f3fc11c6-03c9-4314-99c3-f857b149415c-tls-certs\" (UniqueName: \"kubernetes.io/secret/f3fc11c6-03c9-4314-99c3-f857b149415c-tls-certs\") pod \"f3fc11c6-03c9-4314-99c3-f857b149415c\" (UID: \"f3fc11c6-03c9-4314-99c3-f857b149415c\") " Oct 29 23:28:59.387135 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:59.387025 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3fc11c6-03c9-4314-99c3-f857b149415c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3fc11c6-03c9-4314-99c3-f857b149415c-dshm\") pod \"f3fc11c6-03c9-4314-99c3-f857b149415c\" (UID: \"f3fc11c6-03c9-4314-99c3-f857b149415c\") " Oct 29 23:28:59.387135 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:59.387055 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3fc11c6-03c9-4314-99c3-f857b149415c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3fc11c6-03c9-4314-99c3-f857b149415c-model-cache\") pod \"f3fc11c6-03c9-4314-99c3-f857b149415c\" (UID: \"f3fc11c6-03c9-4314-99c3-f857b149415c\") " Oct 29 23:28:59.387135 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:59.387130 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3fc11c6-03c9-4314-99c3-f857b149415c-home\" (UniqueName: \"kubernetes.io/empty-dir/f3fc11c6-03c9-4314-99c3-f857b149415c-home\") pod \"f3fc11c6-03c9-4314-99c3-f857b149415c\" (UID: \"f3fc11c6-03c9-4314-99c3-f857b149415c\") " Oct 29 23:28:59.387696 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:59.387282 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3fc11c6-03c9-4314-99c3-f857b149415c-kube-api-access-xfn5w\" (UniqueName: \"kubernetes.io/projected/f3fc11c6-03c9-4314-99c3-f857b149415c-kube-api-access-xfn5w\") pod \"f3fc11c6-03c9-4314-99c3-f857b149415c\" (UID: \"f3fc11c6-03c9-4314-99c3-f857b149415c\") " Oct 29 23:28:59.390633 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:59.390320 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3fc11c6-03c9-4314-99c3-f857b149415c-tmp-dir" pod "f3fc11c6-03c9-4314-99c3-f857b149415c" (UID: "f3fc11c6-03c9-4314-99c3-f857b149415c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:28:59.392485 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:59.392262 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3fc11c6-03c9-4314-99c3-f857b149415c-kube-api-access-xfn5w" pod "f3fc11c6-03c9-4314-99c3-f857b149415c" (UID: "f3fc11c6-03c9-4314-99c3-f857b149415c"). InnerVolumeSpecName "kube-api-access-xfn5w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:28:59.393216 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:59.392647 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3fc11c6-03c9-4314-99c3-f857b149415c-model-cache" pod "f3fc11c6-03c9-4314-99c3-f857b149415c" (UID: "f3fc11c6-03c9-4314-99c3-f857b149415c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:28:59.394092 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:59.394070 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3fc11c6-03c9-4314-99c3-f857b149415c-home" pod "f3fc11c6-03c9-4314-99c3-f857b149415c" (UID: "f3fc11c6-03c9-4314-99c3-f857b149415c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:28:59.397525 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:59.397483 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3fc11c6-03c9-4314-99c3-f857b149415c-tls-certs" pod "f3fc11c6-03c9-4314-99c3-f857b149415c" (UID: "f3fc11c6-03c9-4314-99c3-f857b149415c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:28:59.397870 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:59.397845 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3fc11c6-03c9-4314-99c3-f857b149415c-dshm" pod "f3fc11c6-03c9-4314-99c3-f857b149415c" (UID: "f3fc11c6-03c9-4314-99c3-f857b149415c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:28:59.456400 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:59.456365 5057 generic.go:356] "Generic (PLEG): container finished" podID="f3fc11c6-03c9-4314-99c3-f857b149415c" containerID="ba7af072d51cf9d892a2aff973911964540cba2c84ebbb4916b5d0d2bd97e8f7" exitCode=0 Oct 29 23:28:59.456400 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:59.456395 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-31226c/distinct-w6-31226c-kserve-858d95fd76-bkf7c" Oct 29 23:28:59.456651 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:59.456431 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-31226c/distinct-w6-31226c-kserve-858d95fd76-bkf7c" event={"ID":"f3fc11c6-03c9-4314-99c3-f857b149415c","Type":"ContainerDied","Data":"ba7af072d51cf9d892a2aff973911964540cba2c84ebbb4916b5d0d2bd97e8f7"} Oct 29 23:28:59.456651 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:59.456463 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-31226c/distinct-w6-31226c-kserve-858d95fd76-bkf7c" event={"ID":"f3fc11c6-03c9-4314-99c3-f857b149415c","Type":"ContainerDied","Data":"54d0af1252aa238c5e891f1c8dd85f8deb17d528e68f7710a3942f0999a4be22"} Oct 29 23:28:59.456651 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:59.456494 5057 scope.go:122] "RemoveContainer" containerID="ba7af072d51cf9d892a2aff973911964540cba2c84ebbb4916b5d0d2bd97e8f7" Oct 29 23:28:59.472965 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:59.472942 5057 scope.go:122] "RemoveContainer" containerID="ba7af072d51cf9d892a2aff973911964540cba2c84ebbb4916b5d0d2bd97e8f7" Oct 29 23:28:59.473282 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:28:59.473260 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba7af072d51cf9d892a2aff973911964540cba2c84ebbb4916b5d0d2bd97e8f7\": container with ID starting with ba7af072d51cf9d892a2aff973911964540cba2c84ebbb4916b5d0d2bd97e8f7 not found: ID does not exist" containerID="ba7af072d51cf9d892a2aff973911964540cba2c84ebbb4916b5d0d2bd97e8f7" Oct 29 23:28:59.473449 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:59.473294 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba7af072d51cf9d892a2aff973911964540cba2c84ebbb4916b5d0d2bd97e8f7"} err="failed to get container status \"ba7af072d51cf9d892a2aff973911964540cba2c84ebbb4916b5d0d2bd97e8f7\": rpc error: code = NotFound desc = could not find container \"ba7af072d51cf9d892a2aff973911964540cba2c84ebbb4916b5d0d2bd97e8f7\": container with ID starting with ba7af072d51cf9d892a2aff973911964540cba2c84ebbb4916b5d0d2bd97e8f7 not found: ID does not exist" Oct 29 23:28:59.490563 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:59.490537 5057 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3fc11c6-03c9-4314-99c3-f857b149415c-home\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:28:59.490690 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:59.490566 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xfn5w\" (UniqueName: \"kubernetes.io/projected/f3fc11c6-03c9-4314-99c3-f857b149415c-kube-api-access-xfn5w\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:28:59.490690 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:59.490580 5057 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3fc11c6-03c9-4314-99c3-f857b149415c-tmp-dir\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:28:59.490690 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:59.490593 5057 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3fc11c6-03c9-4314-99c3-f857b149415c-tls-certs\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:28:59.490690 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:59.490605 5057 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3fc11c6-03c9-4314-99c3-f857b149415c-dshm\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:28:59.490690 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:59.490617 5057 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3fc11c6-03c9-4314-99c3-f857b149415c-model-cache\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:28:59.517040 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:59.516949 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-31226c/distinct-w6-31226c-kserve-858d95fd76-bkf7c"] Oct 29 23:28:59.518739 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:59.518716 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-31226c/distinct-w6-31226c-kserve-858d95fd76-bkf7c"] Oct 29 23:28:59.594791 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:59.594758 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v2tfm"] Oct 29 23:28:59.733975 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:59.733938 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w6-31226c-84fb67d984-mpgbh" podUID="3481d9c3-f557-4b9d-b0f2-b0c0bdb81bd9" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:28:59.734145 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:28:59.733972 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-31226c-8475b558c9-2v6n6" podUID="8d4b783f-0071-4524-804e-396c53f069eb" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:29:00.316073 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:00.316035 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:29:00.845994 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:00.845957 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:29:00.957014 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:00.956970 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:29:01.331003 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:01.330972 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3fc11c6-03c9-4314-99c3-f857b149415c" path="/var/lib/kubelet/pods/f3fc11c6-03c9-4314-99c3-f857b149415c/volumes" Oct 29 23:29:01.751007 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:01.750968 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w1-e170ed/rate-test-model-e170ed-kserve-544667c877-hc2sg" Oct 29 23:29:01.763860 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:01.763831 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w1-e170ed/rate-test-model-e170ed-kserve-544667c877-hc2sg" Oct 29 23:29:02.180107 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:02.177710 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:29:02.287359 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:02.287313 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:29:02.397605 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:02.397568 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:29:02.500698 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:02.500669 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w1-e170ed/rate-test-model-e170ed-kserve-544667c877-hc2sg" Oct 29 23:29:03.915268 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:03.915211 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:29:05.590679 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:05.590644 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:29:07.126073 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:07.126028 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-17394a-77794c47ff-djggj" podUID="5917154d-50a3-4166-94e0-533aef08f790" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:29:07.190857 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:07.190803 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:29:08.485605 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:08.485559 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:29:08.663988 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:08.663953 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:29:09.733681 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:09.733636 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w6-31226c-84fb67d984-mpgbh" podUID="3481d9c3-f557-4b9d-b0f2-b0c0bdb81bd9" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:29:09.734210 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:09.733729 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-31226c-8475b558c9-2v6n6" podUID="8d4b783f-0071-4524-804e-396c53f069eb" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:29:10.307318 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:10.307277 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:29:11.889620 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:11.889583 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:29:13.439504 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:13.439482 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-17394a-77794c47ff-djggj" Oct 29 23:29:13.518788 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:13.518731 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:29:13.600777 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:13.600742 5057 generic.go:356] "Generic (PLEG): container finished" podID="5917154d-50a3-4166-94e0-533aef08f790" containerID="f127c13a5f2b483e7e49b6a7ffb9d50ce77abeb89c03b2cc1d5971c9ccce63a7" exitCode=0 Oct 29 23:29:13.600979 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:13.600805 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-17394a-77794c47ff-djggj" Oct 29 23:29:13.600979 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:13.600826 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-17394a-77794c47ff-djggj" event={"ID":"5917154d-50a3-4166-94e0-533aef08f790","Type":"ContainerDied","Data":"f127c13a5f2b483e7e49b6a7ffb9d50ce77abeb89c03b2cc1d5971c9ccce63a7"} Oct 29 23:29:13.600979 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:13.600893 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-17394a-77794c47ff-djggj" event={"ID":"5917154d-50a3-4166-94e0-533aef08f790","Type":"ContainerDied","Data":"b4881508a0a3662a5d410605c3395bd491fd630daf1c35ec2835f00c41e53fbc"} Oct 29 23:29:13.600979 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:13.600922 5057 scope.go:122] "RemoveContainer" containerID="f127c13a5f2b483e7e49b6a7ffb9d50ce77abeb89c03b2cc1d5971c9ccce63a7" Oct 29 23:29:13.602097 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:13.601994 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5917154d-50a3-4166-94e0-533aef08f790-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5917154d-50a3-4166-94e0-533aef08f790-plugins-config-volume\") pod \"5917154d-50a3-4166-94e0-533aef08f790\" (UID: \"5917154d-50a3-4166-94e0-533aef08f790\") " Oct 29 23:29:13.602231 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:13.602164 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5917154d-50a3-4166-94e0-533aef08f790-kube-api-access-nxnz8\" (UniqueName: \"kubernetes.io/projected/5917154d-50a3-4166-94e0-533aef08f790-kube-api-access-nxnz8\") pod \"5917154d-50a3-4166-94e0-533aef08f790\" (UID: \"5917154d-50a3-4166-94e0-533aef08f790\") " Oct 29 23:29:13.602487 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:13.602460 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5917154d-50a3-4166-94e0-533aef08f790-plugins-config-volume" pod "5917154d-50a3-4166-94e0-533aef08f790" (UID: "5917154d-50a3-4166-94e0-533aef08f790"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:29:13.602901 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:13.602877 5057 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5917154d-50a3-4166-94e0-533aef08f790-plugins-config-volume\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:29:13.604481 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:13.604458 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5917154d-50a3-4166-94e0-533aef08f790-kube-api-access-nxnz8" pod "5917154d-50a3-4166-94e0-533aef08f790" (UID: "5917154d-50a3-4166-94e0-533aef08f790"). InnerVolumeSpecName "kube-api-access-nxnz8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:29:13.625278 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:13.625256 5057 scope.go:122] "RemoveContainer" containerID="f127c13a5f2b483e7e49b6a7ffb9d50ce77abeb89c03b2cc1d5971c9ccce63a7" Oct 29 23:29:13.625596 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:29:13.625569 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f127c13a5f2b483e7e49b6a7ffb9d50ce77abeb89c03b2cc1d5971c9ccce63a7\": container with ID starting with f127c13a5f2b483e7e49b6a7ffb9d50ce77abeb89c03b2cc1d5971c9ccce63a7 not found: ID does not exist" containerID="f127c13a5f2b483e7e49b6a7ffb9d50ce77abeb89c03b2cc1d5971c9ccce63a7" Oct 29 23:29:13.625656 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:13.625607 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f127c13a5f2b483e7e49b6a7ffb9d50ce77abeb89c03b2cc1d5971c9ccce63a7"} err="failed to get container status \"f127c13a5f2b483e7e49b6a7ffb9d50ce77abeb89c03b2cc1d5971c9ccce63a7\": rpc error: code = NotFound desc = could not find container \"f127c13a5f2b483e7e49b6a7ffb9d50ce77abeb89c03b2cc1d5971c9ccce63a7\": container with ID starting with f127c13a5f2b483e7e49b6a7ffb9d50ce77abeb89c03b2cc1d5971c9ccce63a7 not found: ID does not exist" Oct 29 23:29:13.705054 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:13.705016 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nxnz8\" (UniqueName: \"kubernetes.io/projected/5917154d-50a3-4166-94e0-533aef08f790-kube-api-access-nxnz8\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:29:13.951260 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:13.951212 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-17394a-77794c47ff-djggj"] Oct 29 23:29:13.962561 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:13.962489 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-17394a-77794c47ff-djggj"] Oct 29 23:29:15.123759 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:15.123203 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:29:15.132483 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:15.132457 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-31226c-84fb67d984-mpgbh" Oct 29 23:29:15.158474 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:15.158454 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-31226c-8475b558c9-2v6n6" Oct 29 23:29:15.233576 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:15.233527 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3481d9c3-f557-4b9d-b0f2-b0c0bdb81bd9-kube-api-access-4jwjl\" (UniqueName: \"kubernetes.io/projected/3481d9c3-f557-4b9d-b0f2-b0c0bdb81bd9-kube-api-access-4jwjl\") pod \"3481d9c3-f557-4b9d-b0f2-b0c0bdb81bd9\" (UID: \"3481d9c3-f557-4b9d-b0f2-b0c0bdb81bd9\") " Oct 29 23:29:15.233913 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:15.233874 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3481d9c3-f557-4b9d-b0f2-b0c0bdb81bd9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3481d9c3-f557-4b9d-b0f2-b0c0bdb81bd9-plugins-config-volume\") pod \"3481d9c3-f557-4b9d-b0f2-b0c0bdb81bd9\" (UID: \"3481d9c3-f557-4b9d-b0f2-b0c0bdb81bd9\") " Oct 29 23:29:15.235204 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:15.235178 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3481d9c3-f557-4b9d-b0f2-b0c0bdb81bd9-plugins-config-volume" pod "3481d9c3-f557-4b9d-b0f2-b0c0bdb81bd9" (UID: "3481d9c3-f557-4b9d-b0f2-b0c0bdb81bd9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:29:15.238146 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:15.238108 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3481d9c3-f557-4b9d-b0f2-b0c0bdb81bd9-kube-api-access-4jwjl" pod "3481d9c3-f557-4b9d-b0f2-b0c0bdb81bd9" (UID: "3481d9c3-f557-4b9d-b0f2-b0c0bdb81bd9"). InnerVolumeSpecName "kube-api-access-4jwjl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:29:15.333870 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:15.333784 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5917154d-50a3-4166-94e0-533aef08f790" path="/var/lib/kubelet/pods/5917154d-50a3-4166-94e0-533aef08f790/volumes" Oct 29 23:29:15.335684 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:15.335649 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d4b783f-0071-4524-804e-396c53f069eb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d4b783f-0071-4524-804e-396c53f069eb-plugins-config-volume\") pod \"8d4b783f-0071-4524-804e-396c53f069eb\" (UID: \"8d4b783f-0071-4524-804e-396c53f069eb\") " Oct 29 23:29:15.335843 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:15.335741 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d4b783f-0071-4524-804e-396c53f069eb-kube-api-access-7pkvj\" (UniqueName: \"kubernetes.io/projected/8d4b783f-0071-4524-804e-396c53f069eb-kube-api-access-7pkvj\") pod \"8d4b783f-0071-4524-804e-396c53f069eb\" (UID: \"8d4b783f-0071-4524-804e-396c53f069eb\") " Oct 29 23:29:15.336085 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:15.336056 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d4b783f-0071-4524-804e-396c53f069eb-plugins-config-volume" pod "8d4b783f-0071-4524-804e-396c53f069eb" (UID: "8d4b783f-0071-4524-804e-396c53f069eb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:29:15.336530 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:15.336506 5057 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3481d9c3-f557-4b9d-b0f2-b0c0bdb81bd9-plugins-config-volume\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:29:15.336607 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:15.336537 5057 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d4b783f-0071-4524-804e-396c53f069eb-plugins-config-volume\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:29:15.336607 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:15.336552 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4jwjl\" (UniqueName: \"kubernetes.io/projected/3481d9c3-f557-4b9d-b0f2-b0c0bdb81bd9-kube-api-access-4jwjl\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:29:15.338523 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:15.338495 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d4b783f-0071-4524-804e-396c53f069eb-kube-api-access-7pkvj" pod "8d4b783f-0071-4524-804e-396c53f069eb" (UID: "8d4b783f-0071-4524-804e-396c53f069eb"). InnerVolumeSpecName "kube-api-access-7pkvj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:29:15.439149 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:15.439105 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7pkvj\" (UniqueName: \"kubernetes.io/projected/8d4b783f-0071-4524-804e-396c53f069eb-kube-api-access-7pkvj\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:29:15.622310 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:15.622213 5057 generic.go:356] "Generic (PLEG): container finished" podID="8d4b783f-0071-4524-804e-396c53f069eb" containerID="ec85848b77e7759edeb94a5140c9aca054dfe8421fd1f8d0c1be351cae3a83b1" exitCode=0 Oct 29 23:29:15.622310 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:15.622272 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-31226c-8475b558c9-2v6n6" Oct 29 23:29:15.622582 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:15.622308 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-31226c-8475b558c9-2v6n6" event={"ID":"8d4b783f-0071-4524-804e-396c53f069eb","Type":"ContainerDied","Data":"ec85848b77e7759edeb94a5140c9aca054dfe8421fd1f8d0c1be351cae3a83b1"} Oct 29 23:29:15.622582 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:15.622407 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-31226c-8475b558c9-2v6n6" event={"ID":"8d4b783f-0071-4524-804e-396c53f069eb","Type":"ContainerDied","Data":"15451b41b864963ec6930664d81d7c40a21edc62aa77711cf60fee7ded5b8839"} Oct 29 23:29:15.622582 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:15.622438 5057 scope.go:122] "RemoveContainer" containerID="ec85848b77e7759edeb94a5140c9aca054dfe8421fd1f8d0c1be351cae3a83b1" Oct 29 23:29:15.624159 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:15.624135 5057 generic.go:356] "Generic (PLEG): container finished" podID="3481d9c3-f557-4b9d-b0f2-b0c0bdb81bd9" containerID="59beb4040f5ec60491fecb6584e0ae3b32213333c9f4f1ff0c302a0ecfdd7280" exitCode=0 Oct 29 23:29:15.624285 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:15.624231 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-31226c-84fb67d984-mpgbh" Oct 29 23:29:15.624433 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:15.624410 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-31226c-84fb67d984-mpgbh" event={"ID":"3481d9c3-f557-4b9d-b0f2-b0c0bdb81bd9","Type":"ContainerDied","Data":"59beb4040f5ec60491fecb6584e0ae3b32213333c9f4f1ff0c302a0ecfdd7280"} Oct 29 23:29:15.624488 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:15.624461 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-31226c-84fb67d984-mpgbh" event={"ID":"3481d9c3-f557-4b9d-b0f2-b0c0bdb81bd9","Type":"ContainerDied","Data":"0ec98b129f183eb1cb94e00beac2a9f6186480e0633af9bdc7ddb032904f4a59"} Oct 29 23:29:15.635869 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:15.635852 5057 scope.go:122] "RemoveContainer" containerID="ec85848b77e7759edeb94a5140c9aca054dfe8421fd1f8d0c1be351cae3a83b1" Oct 29 23:29:15.636166 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:29:15.636140 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec85848b77e7759edeb94a5140c9aca054dfe8421fd1f8d0c1be351cae3a83b1\": container with ID starting with ec85848b77e7759edeb94a5140c9aca054dfe8421fd1f8d0c1be351cae3a83b1 not found: ID does not exist" containerID="ec85848b77e7759edeb94a5140c9aca054dfe8421fd1f8d0c1be351cae3a83b1" Oct 29 23:29:15.636264 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:15.636175 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec85848b77e7759edeb94a5140c9aca054dfe8421fd1f8d0c1be351cae3a83b1"} err="failed to get container status \"ec85848b77e7759edeb94a5140c9aca054dfe8421fd1f8d0c1be351cae3a83b1\": rpc error: code = NotFound desc = could not find container \"ec85848b77e7759edeb94a5140c9aca054dfe8421fd1f8d0c1be351cae3a83b1\": container with ID starting with ec85848b77e7759edeb94a5140c9aca054dfe8421fd1f8d0c1be351cae3a83b1 not found: ID does not exist" Oct 29 23:29:15.636264 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:15.636199 5057 scope.go:122] "RemoveContainer" containerID="59beb4040f5ec60491fecb6584e0ae3b32213333c9f4f1ff0c302a0ecfdd7280" Oct 29 23:29:15.648411 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:15.648390 5057 scope.go:122] "RemoveContainer" containerID="59beb4040f5ec60491fecb6584e0ae3b32213333c9f4f1ff0c302a0ecfdd7280" Oct 29 23:29:15.648699 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:29:15.648675 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59beb4040f5ec60491fecb6584e0ae3b32213333c9f4f1ff0c302a0ecfdd7280\": container with ID starting with 59beb4040f5ec60491fecb6584e0ae3b32213333c9f4f1ff0c302a0ecfdd7280 not found: ID does not exist" containerID="59beb4040f5ec60491fecb6584e0ae3b32213333c9f4f1ff0c302a0ecfdd7280" Oct 29 23:29:15.648762 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:15.648706 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59beb4040f5ec60491fecb6584e0ae3b32213333c9f4f1ff0c302a0ecfdd7280"} err="failed to get container status \"59beb4040f5ec60491fecb6584e0ae3b32213333c9f4f1ff0c302a0ecfdd7280\": rpc error: code = NotFound desc = could not find container \"59beb4040f5ec60491fecb6584e0ae3b32213333c9f4f1ff0c302a0ecfdd7280\": container with ID starting with 59beb4040f5ec60491fecb6584e0ae3b32213333c9f4f1ff0c302a0ecfdd7280 not found: ID does not exist" Oct 29 23:29:15.674388 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:15.674363 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-31226c-84fb67d984-mpgbh"] Oct 29 23:29:15.686656 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:15.686631 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-31226c-84fb67d984-mpgbh"] Oct 29 23:29:15.719993 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:15.719955 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-31226c-8475b558c9-2v6n6"] Oct 29 23:29:15.726985 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:15.726959 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-31226c-8475b558c9-2v6n6"] Oct 29 23:29:16.498129 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:16.498092 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:29:16.792542 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:16.792449 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:29:17.335175 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:17.335141 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3481d9c3-f557-4b9d-b0f2-b0c0bdb81bd9" path="/var/lib/kubelet/pods/3481d9c3-f557-4b9d-b0f2-b0c0bdb81bd9/volumes" Oct 29 23:29:17.335775 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:17.335751 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d4b783f-0071-4524-804e-396c53f069eb" path="/var/lib/kubelet/pods/8d4b783f-0071-4524-804e-396c53f069eb/volumes" Oct 29 23:29:18.621094 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:18.621052 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:29:19.029021 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:29:19.028983 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d2dnn" containerName="extractor" Oct 29 23:29:20.219977 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:20.219934 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:29:22.017469 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:22.017432 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:29:23.259248 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:23.259211 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-17394a-f64cd8977-xhjpk"] Oct 29 23:29:23.259737 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:23.259493 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-17394a-f64cd8977-xhjpk" podUID="21c72ab9-0742-4f8a-aee1-8f820d967c94" containerName="maas-api" containerID="cri-o://2c0666f47fdff3cddd2313a8765c7664b08dec0ba9817811ef1d341c9571d9c1" gracePeriod=30 Oct 29 23:29:23.363831 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:23.363791 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:29:23.582189 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:23.582159 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-17394a-f64cd8977-xhjpk" Oct 29 23:29:23.688355 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:23.665656 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/21c72ab9-0742-4f8a-aee1-8f820d967c94-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21c72ab9-0742-4f8a-aee1-8f820d967c94-maas-api-tls\") pod \"21c72ab9-0742-4f8a-aee1-8f820d967c94\" (UID: \"21c72ab9-0742-4f8a-aee1-8f820d967c94\") " Oct 29 23:29:23.688355 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:23.665786 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21c72ab9-0742-4f8a-aee1-8f820d967c94-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/21c72ab9-0742-4f8a-aee1-8f820d967c94-openshift-service-ca\") pod \"21c72ab9-0742-4f8a-aee1-8f820d967c94\" (UID: \"21c72ab9-0742-4f8a-aee1-8f820d967c94\") " Oct 29 23:29:23.688355 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:23.665879 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/21c72ab9-0742-4f8a-aee1-8f820d967c94-metrics-certs\" (UniqueName: \"kubernetes.io/secret/21c72ab9-0742-4f8a-aee1-8f820d967c94-metrics-certs\") pod \"21c72ab9-0742-4f8a-aee1-8f820d967c94\" (UID: \"21c72ab9-0742-4f8a-aee1-8f820d967c94\") " Oct 29 23:29:23.688355 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:23.665945 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21c72ab9-0742-4f8a-aee1-8f820d967c94-kube-api-access-57z5j\" (UniqueName: \"kubernetes.io/projected/21c72ab9-0742-4f8a-aee1-8f820d967c94-kube-api-access-57z5j\") pod \"21c72ab9-0742-4f8a-aee1-8f820d967c94\" (UID: \"21c72ab9-0742-4f8a-aee1-8f820d967c94\") " Oct 29 23:29:23.711727 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:23.708793 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21c72ab9-0742-4f8a-aee1-8f820d967c94-openshift-service-ca" pod "21c72ab9-0742-4f8a-aee1-8f820d967c94" (UID: "21c72ab9-0742-4f8a-aee1-8f820d967c94"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:29:23.718532 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:23.718493 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21c72ab9-0742-4f8a-aee1-8f820d967c94-maas-api-tls" pod "21c72ab9-0742-4f8a-aee1-8f820d967c94" (UID: "21c72ab9-0742-4f8a-aee1-8f820d967c94"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:29:23.718705 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:23.718513 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21c72ab9-0742-4f8a-aee1-8f820d967c94-metrics-certs" pod "21c72ab9-0742-4f8a-aee1-8f820d967c94" (UID: "21c72ab9-0742-4f8a-aee1-8f820d967c94"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:29:23.718705 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:23.718570 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21c72ab9-0742-4f8a-aee1-8f820d967c94-kube-api-access-57z5j" pod "21c72ab9-0742-4f8a-aee1-8f820d967c94" (UID: "21c72ab9-0742-4f8a-aee1-8f820d967c94"). InnerVolumeSpecName "kube-api-access-57z5j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:29:23.729155 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:23.728661 5057 generic.go:356] "Generic (PLEG): container finished" podID="21c72ab9-0742-4f8a-aee1-8f820d967c94" containerID="2c0666f47fdff3cddd2313a8765c7664b08dec0ba9817811ef1d341c9571d9c1" exitCode=0 Oct 29 23:29:23.729155 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:23.728709 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-17394a-f64cd8977-xhjpk" Oct 29 23:29:23.729155 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:23.728949 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-17394a-f64cd8977-xhjpk" event={"ID":"21c72ab9-0742-4f8a-aee1-8f820d967c94","Type":"ContainerDied","Data":"2c0666f47fdff3cddd2313a8765c7664b08dec0ba9817811ef1d341c9571d9c1"} Oct 29 23:29:23.729155 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:23.728991 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-17394a-f64cd8977-xhjpk" event={"ID":"21c72ab9-0742-4f8a-aee1-8f820d967c94","Type":"ContainerDied","Data":"ae37a6d5fc513a559ad0a4cde9b6d50e0b1bb1bc09ceb1bb6b03c76edeae082f"} Oct 29 23:29:23.729155 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:23.729022 5057 scope.go:122] "RemoveContainer" containerID="2c0666f47fdff3cddd2313a8765c7664b08dec0ba9817811ef1d341c9571d9c1" Oct 29 23:29:23.753774 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:23.753751 5057 scope.go:122] "RemoveContainer" containerID="2c0666f47fdff3cddd2313a8765c7664b08dec0ba9817811ef1d341c9571d9c1" Oct 29 23:29:23.754102 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:29:23.754084 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c0666f47fdff3cddd2313a8765c7664b08dec0ba9817811ef1d341c9571d9c1\": container with ID starting with 2c0666f47fdff3cddd2313a8765c7664b08dec0ba9817811ef1d341c9571d9c1 not found: ID does not exist" containerID="2c0666f47fdff3cddd2313a8765c7664b08dec0ba9817811ef1d341c9571d9c1" Oct 29 23:29:23.754171 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:23.754111 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c0666f47fdff3cddd2313a8765c7664b08dec0ba9817811ef1d341c9571d9c1"} err="failed to get container status \"2c0666f47fdff3cddd2313a8765c7664b08dec0ba9817811ef1d341c9571d9c1\": rpc error: code = NotFound desc = could not find container \"2c0666f47fdff3cddd2313a8765c7664b08dec0ba9817811ef1d341c9571d9c1\": container with ID starting with 2c0666f47fdff3cddd2313a8765c7664b08dec0ba9817811ef1d341c9571d9c1 not found: ID does not exist" Oct 29 23:29:23.768083 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:23.768058 5057 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21c72ab9-0742-4f8a-aee1-8f820d967c94-maas-api-tls\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:29:23.768083 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:23.768084 5057 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/21c72ab9-0742-4f8a-aee1-8f820d967c94-openshift-service-ca\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:29:23.768245 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:23.768095 5057 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/21c72ab9-0742-4f8a-aee1-8f820d967c94-metrics-certs\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:29:23.768245 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:23.768104 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-57z5j\" (UniqueName: \"kubernetes.io/projected/21c72ab9-0742-4f8a-aee1-8f820d967c94-kube-api-access-57z5j\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:29:23.786139 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:23.786111 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-17394a-f64cd8977-xhjpk"] Oct 29 23:29:23.791811 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:23.791781 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-17394a-f64cd8977-xhjpk"] Oct 29 23:29:23.867679 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:23.867594 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:29:23.999390 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:23.999320 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:29:24.117550 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:24.117272 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:29:24.480077 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:24.480035 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:29:25.329674 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:25.329642 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21c72ab9-0742-4f8a-aee1-8f820d967c94" path="/var/lib/kubelet/pods/21c72ab9-0742-4f8a-aee1-8f820d967c94/volumes" Oct 29 23:29:25.363042 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:25.363011 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:29:25.485096 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:25.485036 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:29:25.712760 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:25.712725 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:29:27.398737 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:27.398699 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:29:28.648583 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:28.648551 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f3d24e75-b57596bdd-tvq82"] Oct 29 23:29:28.683493 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:28.683458 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-f3d24e75-58f4755655-nsl26"] Oct 29 23:29:28.806374 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:28.797607 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:29:28.918201 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:28.918114 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:29:29.111661 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:29.111513 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:29:30.607427 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:30.607394 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:29:31.277984 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:31.277946 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:29:31.406911 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:31.406869 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:29:31.705184 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:31.705149 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:29:31.819459 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:31.819410 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:29:32.192839 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:32.192759 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:29:32.474492 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:32.474455 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:29:32.601739 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:32.601695 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:29:32.798678 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:32.798582 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:29:33.649436 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:33.649394 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-f3d24e75-b57596bdd-tvq82" podUID="268baf9a-3c0a-4ac5-a170-e69a7f3d1cfc" containerName="payload-processing" containerID="cri-o://d8b88aeb5bbc5de07c873491bf590b2a6bf6a803d45584ceb105830a4de88a3c" gracePeriod=25 Oct 29 23:29:33.739671 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:33.739618 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:29:33.780069 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:33.780036 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-f3d24e75-58f4755655-nsl26" podUID="e0c5087d-e510-4841-a25c-dd6908304d61" containerName="payload-pre-processing" containerID="cri-o://99c27e53364f12b11d95e1846f068b9065620c840d8c0dc435b3d27a3905e14f" gracePeriod=25 Oct 29 23:29:33.865571 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:33.865541 5057 generic.go:356] "Generic (PLEG): container finished" podID="268baf9a-3c0a-4ac5-a170-e69a7f3d1cfc" containerID="d8b88aeb5bbc5de07c873491bf590b2a6bf6a803d45584ceb105830a4de88a3c" exitCode=0 Oct 29 23:29:33.866000 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:33.865830 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-f3d24e75-b57596bdd-tvq82" event={"ID":"268baf9a-3c0a-4ac5-a170-e69a7f3d1cfc","Type":"ContainerDied","Data":"d8b88aeb5bbc5de07c873491bf590b2a6bf6a803d45584ceb105830a4de88a3c"} Oct 29 23:29:33.970872 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:33.970843 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-f3d24e75-b57596bdd-tvq82" Oct 29 23:29:34.051775 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:34.051747 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/268baf9a-3c0a-4ac5-a170-e69a7f3d1cfc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/268baf9a-3c0a-4ac5-a170-e69a7f3d1cfc-plugins-config-volume\") pod \"268baf9a-3c0a-4ac5-a170-e69a7f3d1cfc\" (UID: \"268baf9a-3c0a-4ac5-a170-e69a7f3d1cfc\") " Oct 29 23:29:34.051978 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:34.051964 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/268baf9a-3c0a-4ac5-a170-e69a7f3d1cfc-kube-api-access-hc7dd\" (UniqueName: \"kubernetes.io/projected/268baf9a-3c0a-4ac5-a170-e69a7f3d1cfc-kube-api-access-hc7dd\") pod \"268baf9a-3c0a-4ac5-a170-e69a7f3d1cfc\" (UID: \"268baf9a-3c0a-4ac5-a170-e69a7f3d1cfc\") " Oct 29 23:29:34.052163 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:34.052136 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/268baf9a-3c0a-4ac5-a170-e69a7f3d1cfc-plugins-config-volume" pod "268baf9a-3c0a-4ac5-a170-e69a7f3d1cfc" (UID: "268baf9a-3c0a-4ac5-a170-e69a7f3d1cfc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:29:34.052557 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:34.052532 5057 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/268baf9a-3c0a-4ac5-a170-e69a7f3d1cfc-plugins-config-volume\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:29:34.054230 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:34.054206 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/268baf9a-3c0a-4ac5-a170-e69a7f3d1cfc-kube-api-access-hc7dd" pod "268baf9a-3c0a-4ac5-a170-e69a7f3d1cfc" (UID: "268baf9a-3c0a-4ac5-a170-e69a7f3d1cfc"). InnerVolumeSpecName "kube-api-access-hc7dd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:29:34.087378 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:34.087313 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-f3d24e75-58f4755655-nsl26" Oct 29 23:29:34.096162 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:34.096130 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:29:34.168313 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:34.168213 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0c5087d-e510-4841-a25c-dd6908304d61-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0c5087d-e510-4841-a25c-dd6908304d61-plugins-config-volume\") pod \"e0c5087d-e510-4841-a25c-dd6908304d61\" (UID: \"e0c5087d-e510-4841-a25c-dd6908304d61\") " Oct 29 23:29:34.168313 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:34.168300 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0c5087d-e510-4841-a25c-dd6908304d61-kube-api-access-j2hzw\" (UniqueName: \"kubernetes.io/projected/e0c5087d-e510-4841-a25c-dd6908304d61-kube-api-access-j2hzw\") pod \"e0c5087d-e510-4841-a25c-dd6908304d61\" (UID: \"e0c5087d-e510-4841-a25c-dd6908304d61\") " Oct 29 23:29:34.168964 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:34.168940 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hc7dd\" (UniqueName: \"kubernetes.io/projected/268baf9a-3c0a-4ac5-a170-e69a7f3d1cfc-kube-api-access-hc7dd\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:29:34.170108 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:34.169821 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0c5087d-e510-4841-a25c-dd6908304d61-plugins-config-volume" pod "e0c5087d-e510-4841-a25c-dd6908304d61" (UID: "e0c5087d-e510-4841-a25c-dd6908304d61"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:29:34.171371 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:34.171345 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0c5087d-e510-4841-a25c-dd6908304d61-kube-api-access-j2hzw" pod "e0c5087d-e510-4841-a25c-dd6908304d61" (UID: "e0c5087d-e510-4841-a25c-dd6908304d61"). InnerVolumeSpecName "kube-api-access-j2hzw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:29:34.269831 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:34.269800 5057 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0c5087d-e510-4841-a25c-dd6908304d61-plugins-config-volume\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:29:34.269831 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:34.269827 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j2hzw\" (UniqueName: \"kubernetes.io/projected/e0c5087d-e510-4841-a25c-dd6908304d61-kube-api-access-j2hzw\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:29:34.875476 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:34.875440 5057 generic.go:356] "Generic (PLEG): container finished" podID="e0c5087d-e510-4841-a25c-dd6908304d61" containerID="99c27e53364f12b11d95e1846f068b9065620c840d8c0dc435b3d27a3905e14f" exitCode=0 Oct 29 23:29:34.875965 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:34.875490 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-f3d24e75-58f4755655-nsl26" Oct 29 23:29:34.875965 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:34.875521 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-f3d24e75-58f4755655-nsl26" event={"ID":"e0c5087d-e510-4841-a25c-dd6908304d61","Type":"ContainerDied","Data":"99c27e53364f12b11d95e1846f068b9065620c840d8c0dc435b3d27a3905e14f"} Oct 29 23:29:34.875965 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:34.875580 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-f3d24e75-58f4755655-nsl26" event={"ID":"e0c5087d-e510-4841-a25c-dd6908304d61","Type":"ContainerDied","Data":"adbabbc86a40a5939f68830b9e881a6711fab2d9569041b91850243f6f0b628b"} Oct 29 23:29:34.875965 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:34.875614 5057 scope.go:122] "RemoveContainer" containerID="99c27e53364f12b11d95e1846f068b9065620c840d8c0dc435b3d27a3905e14f" Oct 29 23:29:34.877374 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:34.877312 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-f3d24e75-b57596bdd-tvq82" event={"ID":"268baf9a-3c0a-4ac5-a170-e69a7f3d1cfc","Type":"ContainerDied","Data":"d5a2a9ee7130e42a5ed0b2ffc7a20971aa2ad18328bcd35de9a8b43d9baa22bf"} Oct 29 23:29:34.877475 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:34.877382 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-f3d24e75-b57596bdd-tvq82" Oct 29 23:29:34.889172 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:34.889145 5057 scope.go:122] "RemoveContainer" containerID="99c27e53364f12b11d95e1846f068b9065620c840d8c0dc435b3d27a3905e14f" Oct 29 23:29:34.890031 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:29:34.889998 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99c27e53364f12b11d95e1846f068b9065620c840d8c0dc435b3d27a3905e14f\": container with ID starting with 99c27e53364f12b11d95e1846f068b9065620c840d8c0dc435b3d27a3905e14f not found: ID does not exist" containerID="99c27e53364f12b11d95e1846f068b9065620c840d8c0dc435b3d27a3905e14f" Oct 29 23:29:34.890147 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:34.890040 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99c27e53364f12b11d95e1846f068b9065620c840d8c0dc435b3d27a3905e14f"} err="failed to get container status \"99c27e53364f12b11d95e1846f068b9065620c840d8c0dc435b3d27a3905e14f\": rpc error: code = NotFound desc = could not find container \"99c27e53364f12b11d95e1846f068b9065620c840d8c0dc435b3d27a3905e14f\": container with ID starting with 99c27e53364f12b11d95e1846f068b9065620c840d8c0dc435b3d27a3905e14f not found: ID does not exist" Oct 29 23:29:34.890147 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:34.890065 5057 scope.go:122] "RemoveContainer" containerID="d8b88aeb5bbc5de07c873491bf590b2a6bf6a803d45584ceb105830a4de88a3c" Oct 29 23:29:34.929940 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:34.929883 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-f3d24e75-58f4755655-nsl26"] Oct 29 23:29:34.941159 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:34.941133 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-f3d24e75-58f4755655-nsl26"] Oct 29 23:29:34.973669 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:34.973641 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f3d24e75-b57596bdd-tvq82"] Oct 29 23:29:34.979533 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:34.979504 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f3d24e75-b57596bdd-tvq82"] Oct 29 23:29:35.332765 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:35.332730 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="268baf9a-3c0a-4ac5-a170-e69a7f3d1cfc" path="/var/lib/kubelet/pods/268baf9a-3c0a-4ac5-a170-e69a7f3d1cfc/volumes" Oct 29 23:29:35.333302 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:35.333286 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0c5087d-e510-4841-a25c-dd6908304d61" path="/var/lib/kubelet/pods/e0c5087d-e510-4841-a25c-dd6908304d61/volumes" Oct 29 23:29:35.602931 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:35.602849 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:29:36.512792 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:36.512758 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:29:37.175959 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:37.175914 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:29:38.571423 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:38.571381 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-31226c-6c695fb468-fkr4w"] Oct 29 23:29:38.571924 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:38.571633 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-31226c-6c695fb468-fkr4w" podUID="d3d4a1e8-5e7a-4894-811f-5bb731d83b0d" containerName="maas-api" containerID="cri-o://9e2f1e464f82fed716d204ec0226fb417ca8ff769d8081a6a16e43421deb38db" gracePeriod=30 Oct 29 23:29:38.645151 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:38.645119 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:29:38.908038 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:38.908010 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-31226c-6c695fb468-fkr4w" Oct 29 23:29:38.922401 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:38.922365 5057 generic.go:356] "Generic (PLEG): container finished" podID="d3d4a1e8-5e7a-4894-811f-5bb731d83b0d" containerID="9e2f1e464f82fed716d204ec0226fb417ca8ff769d8081a6a16e43421deb38db" exitCode=0 Oct 29 23:29:38.922551 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:38.922401 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-31226c-6c695fb468-fkr4w" event={"ID":"d3d4a1e8-5e7a-4894-811f-5bb731d83b0d","Type":"ContainerDied","Data":"9e2f1e464f82fed716d204ec0226fb417ca8ff769d8081a6a16e43421deb38db"} Oct 29 23:29:38.922551 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:38.922424 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-31226c-6c695fb468-fkr4w" Oct 29 23:29:38.922551 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:38.922457 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-31226c-6c695fb468-fkr4w" event={"ID":"d3d4a1e8-5e7a-4894-811f-5bb731d83b0d","Type":"ContainerDied","Data":"b89407f91ac437375b417532c954cfcf74202208c51170efa846841416afbed8"} Oct 29 23:29:38.922551 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:38.922482 5057 scope.go:122] "RemoveContainer" containerID="9e2f1e464f82fed716d204ec0226fb417ca8ff769d8081a6a16e43421deb38db" Oct 29 23:29:38.935993 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:38.935974 5057 scope.go:122] "RemoveContainer" containerID="9e2f1e464f82fed716d204ec0226fb417ca8ff769d8081a6a16e43421deb38db" Oct 29 23:29:38.936259 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:29:38.936233 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e2f1e464f82fed716d204ec0226fb417ca8ff769d8081a6a16e43421deb38db\": container with ID starting with 9e2f1e464f82fed716d204ec0226fb417ca8ff769d8081a6a16e43421deb38db not found: ID does not exist" containerID="9e2f1e464f82fed716d204ec0226fb417ca8ff769d8081a6a16e43421deb38db" Oct 29 23:29:38.936373 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:38.936273 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e2f1e464f82fed716d204ec0226fb417ca8ff769d8081a6a16e43421deb38db"} err="failed to get container status \"9e2f1e464f82fed716d204ec0226fb417ca8ff769d8081a6a16e43421deb38db\": rpc error: code = NotFound desc = could not find container \"9e2f1e464f82fed716d204ec0226fb417ca8ff769d8081a6a16e43421deb38db\": container with ID starting with 9e2f1e464f82fed716d204ec0226fb417ca8ff769d8081a6a16e43421deb38db not found: ID does not exist" Oct 29 23:29:38.962810 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:38.962782 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3d4a1e8-5e7a-4894-811f-5bb731d83b0d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3d4a1e8-5e7a-4894-811f-5bb731d83b0d-openshift-service-ca\") pod \"d3d4a1e8-5e7a-4894-811f-5bb731d83b0d\" (UID: \"d3d4a1e8-5e7a-4894-811f-5bb731d83b0d\") " Oct 29 23:29:38.962960 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:38.962942 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d3d4a1e8-5e7a-4894-811f-5bb731d83b0d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3d4a1e8-5e7a-4894-811f-5bb731d83b0d-maas-api-tls\") pod \"d3d4a1e8-5e7a-4894-811f-5bb731d83b0d\" (UID: \"d3d4a1e8-5e7a-4894-811f-5bb731d83b0d\") " Oct 29 23:29:38.963050 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:38.963034 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3d4a1e8-5e7a-4894-811f-5bb731d83b0d-kube-api-access-cdrcq\" (UniqueName: \"kubernetes.io/projected/d3d4a1e8-5e7a-4894-811f-5bb731d83b0d-kube-api-access-cdrcq\") pod \"d3d4a1e8-5e7a-4894-811f-5bb731d83b0d\" (UID: \"d3d4a1e8-5e7a-4894-811f-5bb731d83b0d\") " Oct 29 23:29:38.963114 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:38.963093 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d3d4a1e8-5e7a-4894-811f-5bb731d83b0d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d3d4a1e8-5e7a-4894-811f-5bb731d83b0d-metrics-certs\") pod \"d3d4a1e8-5e7a-4894-811f-5bb731d83b0d\" (UID: \"d3d4a1e8-5e7a-4894-811f-5bb731d83b0d\") " Oct 29 23:29:38.963175 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:38.963137 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3d4a1e8-5e7a-4894-811f-5bb731d83b0d-openshift-service-ca" pod "d3d4a1e8-5e7a-4894-811f-5bb731d83b0d" (UID: "d3d4a1e8-5e7a-4894-811f-5bb731d83b0d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:29:38.963729 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:38.963620 5057 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3d4a1e8-5e7a-4894-811f-5bb731d83b0d-openshift-service-ca\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:29:38.965812 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:38.965785 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3d4a1e8-5e7a-4894-811f-5bb731d83b0d-metrics-certs" pod "d3d4a1e8-5e7a-4894-811f-5bb731d83b0d" (UID: "d3d4a1e8-5e7a-4894-811f-5bb731d83b0d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:29:38.965925 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:38.965820 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3d4a1e8-5e7a-4894-811f-5bb731d83b0d-kube-api-access-cdrcq" pod "d3d4a1e8-5e7a-4894-811f-5bb731d83b0d" (UID: "d3d4a1e8-5e7a-4894-811f-5bb731d83b0d"). InnerVolumeSpecName "kube-api-access-cdrcq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:29:38.967282 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:38.967253 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3d4a1e8-5e7a-4894-811f-5bb731d83b0d-maas-api-tls" pod "d3d4a1e8-5e7a-4894-811f-5bb731d83b0d" (UID: "d3d4a1e8-5e7a-4894-811f-5bb731d83b0d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:29:39.047104 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:39.047063 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ab5c06a-lkxql"] Oct 29 23:29:39.053529 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:39.053501 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ab5c06a-lkxql"] Oct 29 23:29:39.066650 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:39.066621 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cdrcq\" (UniqueName: \"kubernetes.io/projected/d3d4a1e8-5e7a-4894-811f-5bb731d83b0d-kube-api-access-cdrcq\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:29:39.066749 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:39.066653 5057 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d3d4a1e8-5e7a-4894-811f-5bb731d83b0d-metrics-certs\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:29:39.066749 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:39.066669 5057 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3d4a1e8-5e7a-4894-811f-5bb731d83b0d-maas-api-tls\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:29:39.220240 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:39.220198 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:29:39.278579 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:39.277614 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-31226c-6c695fb468-fkr4w"] Oct 29 23:29:39.286133 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:39.286102 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-31226c-6c695fb468-fkr4w"] Oct 29 23:29:39.331560 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:39.331527 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79428e20-0cad-4a0a-8529-d1677be60c68" path="/var/lib/kubelet/pods/79428e20-0cad-4a0a-8529-d1677be60c68/volumes" Oct 29 23:29:39.332183 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:39.332160 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3d4a1e8-5e7a-4894-811f-5bb731d83b0d" path="/var/lib/kubelet/pods/d3d4a1e8-5e7a-4894-811f-5bb731d83b0d/volumes" Oct 29 23:29:39.332768 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:39.332747 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v2tfm"] Oct 29 23:29:39.333905 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:39.333863 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:29:39.445544 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:39.445509 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:29:39.647130 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:39.645626 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:29:39.760423 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:39.760379 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:29:39.884655 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:39.884601 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:29:41.350878 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:41.350843 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:29:42.009022 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:42.008978 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:29:42.121314 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:42.121276 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:29:42.244850 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:42.244808 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:29:42.729923 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:42.729887 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:29:43.887047 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:43.887007 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:29:44.219610 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:44.219577 5057 scope.go:122] "RemoveContainer" containerID="30961bbe0bfe08fd7af701cc7e8ca63e2cfe05ef2ca534e53704101b18fc37cf" Oct 29 23:29:44.230817 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:44.230796 5057 scope.go:122] "RemoveContainer" containerID="15df27fea49b6e79c662c9f1b6a1cf7db5244546ea096e7c0602723233d0c503" Oct 29 23:29:44.244259 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:44.242571 5057 scope.go:122] "RemoveContainer" containerID="ef290954fc8f72696e0cd88454e6ba0373fe7038a44d7a39bc145b3b42a90cb6" Oct 29 23:29:44.254131 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:44.254112 5057 scope.go:122] "RemoveContainer" containerID="81e3ca73979b9dcd5daef0ec7d2f3ac632c25c49434e19b8afe04dfa20fe307e" Oct 29 23:29:44.263502 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:44.263476 5057 scope.go:122] "RemoveContainer" containerID="646ae53e0347d12b4f917965dd68d7ac502e13417b3fa374db8b92de7af9ec22" Oct 29 23:29:44.273405 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:44.273386 5057 scope.go:122] "RemoveContainer" containerID="328cd3316a84326fd27c3b4cf8da90944a69f676f7485700daf2198325cb050e" Oct 29 23:29:45.113515 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:45.113480 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-b2921b-6599749ddd-kbw8v"] Oct 29 23:29:45.114144 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:45.114107 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-b2921b-6599749ddd-kbw8v" podUID="02be284e-3e49-4fa7-8475-a5a79f8997b1" containerName="maas-api" containerID="cri-o://6277901ee055d20006326dce028a7d0607d9325974928333951b57d526290e85" gracePeriod=30 Oct 29 23:29:45.180370 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:45.178527 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:29:45.451795 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:45.450554 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-b2921b-6599749ddd-kbw8v" Oct 29 23:29:45.571644 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:45.571612 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02be284e-3e49-4fa7-8475-a5a79f8997b1-kube-api-access-g98km\" (UniqueName: \"kubernetes.io/projected/02be284e-3e49-4fa7-8475-a5a79f8997b1-kube-api-access-g98km\") pod \"02be284e-3e49-4fa7-8475-a5a79f8997b1\" (UID: \"02be284e-3e49-4fa7-8475-a5a79f8997b1\") " Oct 29 23:29:45.571841 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:45.571715 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/02be284e-3e49-4fa7-8475-a5a79f8997b1-metrics-certs\" (UniqueName: \"kubernetes.io/secret/02be284e-3e49-4fa7-8475-a5a79f8997b1-metrics-certs\") pod \"02be284e-3e49-4fa7-8475-a5a79f8997b1\" (UID: \"02be284e-3e49-4fa7-8475-a5a79f8997b1\") " Oct 29 23:29:45.571841 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:45.571764 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/02be284e-3e49-4fa7-8475-a5a79f8997b1-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/02be284e-3e49-4fa7-8475-a5a79f8997b1-maas-api-tls\") pod \"02be284e-3e49-4fa7-8475-a5a79f8997b1\" (UID: \"02be284e-3e49-4fa7-8475-a5a79f8997b1\") " Oct 29 23:29:45.572018 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:45.572004 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/02be284e-3e49-4fa7-8475-a5a79f8997b1-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/02be284e-3e49-4fa7-8475-a5a79f8997b1-openshift-service-ca\") pod \"02be284e-3e49-4fa7-8475-a5a79f8997b1\" (UID: \"02be284e-3e49-4fa7-8475-a5a79f8997b1\") " Oct 29 23:29:45.572322 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:45.572295 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02be284e-3e49-4fa7-8475-a5a79f8997b1-openshift-service-ca" pod "02be284e-3e49-4fa7-8475-a5a79f8997b1" (UID: "02be284e-3e49-4fa7-8475-a5a79f8997b1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:29:45.572633 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:45.572615 5057 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/02be284e-3e49-4fa7-8475-a5a79f8997b1-openshift-service-ca\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:29:45.573731 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:45.573703 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02be284e-3e49-4fa7-8475-a5a79f8997b1-kube-api-access-g98km" pod "02be284e-3e49-4fa7-8475-a5a79f8997b1" (UID: "02be284e-3e49-4fa7-8475-a5a79f8997b1"). InnerVolumeSpecName "kube-api-access-g98km". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:29:45.573841 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:45.573754 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02be284e-3e49-4fa7-8475-a5a79f8997b1-maas-api-tls" pod "02be284e-3e49-4fa7-8475-a5a79f8997b1" (UID: "02be284e-3e49-4fa7-8475-a5a79f8997b1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:29:45.573841 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:45.573804 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02be284e-3e49-4fa7-8475-a5a79f8997b1-metrics-certs" pod "02be284e-3e49-4fa7-8475-a5a79f8997b1" (UID: "02be284e-3e49-4fa7-8475-a5a79f8997b1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:29:45.675522 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:45.675182 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g98km\" (UniqueName: \"kubernetes.io/projected/02be284e-3e49-4fa7-8475-a5a79f8997b1-kube-api-access-g98km\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:29:45.675522 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:45.675224 5057 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/02be284e-3e49-4fa7-8475-a5a79f8997b1-metrics-certs\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:29:45.675522 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:45.675241 5057 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/02be284e-3e49-4fa7-8475-a5a79f8997b1-maas-api-tls\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:29:45.755113 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:45.755074 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:29:45.864609 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:45.864572 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:29:45.998610 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:45.998569 5057 generic.go:356] "Generic (PLEG): container finished" podID="02be284e-3e49-4fa7-8475-a5a79f8997b1" containerID="6277901ee055d20006326dce028a7d0607d9325974928333951b57d526290e85" exitCode=0 Oct 29 23:29:45.998797 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:45.998635 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-b2921b-6599749ddd-kbw8v" Oct 29 23:29:45.998797 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:45.998627 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-b2921b-6599749ddd-kbw8v" event={"ID":"02be284e-3e49-4fa7-8475-a5a79f8997b1","Type":"ContainerDied","Data":"6277901ee055d20006326dce028a7d0607d9325974928333951b57d526290e85"} Oct 29 23:29:45.998797 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:45.998707 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-b2921b-6599749ddd-kbw8v" event={"ID":"02be284e-3e49-4fa7-8475-a5a79f8997b1","Type":"ContainerDied","Data":"6246f3109c9c5fa654478bf6806e90fc848528247cd96cec81ce573e05cca639"} Oct 29 23:29:45.998797 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:45.998739 5057 scope.go:122] "RemoveContainer" containerID="6277901ee055d20006326dce028a7d0607d9325974928333951b57d526290e85" Oct 29 23:29:46.012097 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:46.012036 5057 scope.go:122] "RemoveContainer" containerID="6277901ee055d20006326dce028a7d0607d9325974928333951b57d526290e85" Oct 29 23:29:46.012478 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:29:46.012452 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6277901ee055d20006326dce028a7d0607d9325974928333951b57d526290e85\": container with ID starting with 6277901ee055d20006326dce028a7d0607d9325974928333951b57d526290e85 not found: ID does not exist" containerID="6277901ee055d20006326dce028a7d0607d9325974928333951b57d526290e85" Oct 29 23:29:46.012588 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:46.012486 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6277901ee055d20006326dce028a7d0607d9325974928333951b57d526290e85"} err="failed to get container status \"6277901ee055d20006326dce028a7d0607d9325974928333951b57d526290e85\": rpc error: code = NotFound desc = could not find container \"6277901ee055d20006326dce028a7d0607d9325974928333951b57d526290e85\": container with ID starting with 6277901ee055d20006326dce028a7d0607d9325974928333951b57d526290e85 not found: ID does not exist" Oct 29 23:29:46.051742 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:46.051714 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-b2921b-6599749ddd-kbw8v"] Oct 29 23:29:46.062914 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:46.062889 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-b2921b-6599749ddd-kbw8v"] Oct 29 23:29:46.399793 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:46.399701 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:29:47.193678 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:47.193615 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b18b56d5-jmkc5"] Oct 29 23:29:47.197151 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:47.195319 5057 cpu_manager.go:414] "containerMap: removing container" podUID="e0c5087d-e510-4841-a25c-dd6908304d61" containerName="payload-pre-processing" Oct 29 23:29:47.197151 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:47.195360 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0c5087d-e510-4841-a25c-dd6908304d61" containerName="payload-pre-processing" Oct 29 23:29:47.197151 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:47.195378 5057 cpu_manager.go:414] "containerMap: removing container" podUID="3481d9c3-f557-4b9d-b0f2-b0c0bdb81bd9" containerName="payload-processing" Oct 29 23:29:47.197151 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:47.195392 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3481d9c3-f557-4b9d-b0f2-b0c0bdb81bd9" containerName="payload-processing" Oct 29 23:29:47.197151 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:47.195405 5057 cpu_manager.go:414] "containerMap: removing container" podUID="02be284e-3e49-4fa7-8475-a5a79f8997b1" containerName="maas-api" Oct 29 23:29:47.197151 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:47.195412 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02be284e-3e49-4fa7-8475-a5a79f8997b1" containerName="maas-api" Oct 29 23:29:47.197151 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:47.195458 5057 cpu_manager.go:414] "containerMap: removing container" podUID="5917154d-50a3-4166-94e0-533aef08f790" containerName="payload-pre-processing" Oct 29 23:29:47.197151 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:47.195466 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5917154d-50a3-4166-94e0-533aef08f790" containerName="payload-pre-processing" Oct 29 23:29:47.197151 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:47.195501 5057 cpu_manager.go:414] "containerMap: removing container" podUID="268baf9a-3c0a-4ac5-a170-e69a7f3d1cfc" containerName="payload-processing" Oct 29 23:29:47.197151 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:47.195508 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="268baf9a-3c0a-4ac5-a170-e69a7f3d1cfc" containerName="payload-processing" Oct 29 23:29:47.197151 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:47.195523 5057 cpu_manager.go:414] "containerMap: removing container" podUID="f3fc11c6-03c9-4314-99c3-f857b149415c" containerName="main" Oct 29 23:29:47.197151 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:47.195529 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3fc11c6-03c9-4314-99c3-f857b149415c" containerName="main" Oct 29 23:29:47.197151 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:47.195549 5057 cpu_manager.go:414] "containerMap: removing container" podUID="d3d4a1e8-5e7a-4894-811f-5bb731d83b0d" containerName="maas-api" Oct 29 23:29:47.197151 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:47.195556 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3d4a1e8-5e7a-4894-811f-5bb731d83b0d" containerName="maas-api" Oct 29 23:29:47.197151 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:47.195584 5057 cpu_manager.go:414] "containerMap: removing container" podUID="8d4b783f-0071-4524-804e-396c53f069eb" containerName="payload-pre-processing" Oct 29 23:29:47.197151 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:47.195591 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d4b783f-0071-4524-804e-396c53f069eb" containerName="payload-pre-processing" Oct 29 23:29:47.197151 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:47.195670 5057 cpu_manager.go:414] "containerMap: removing container" podUID="f29e8b65-b7dd-45d4-86b0-f13a29ef8e66" containerName="main" Oct 29 23:29:47.197151 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:47.195679 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f29e8b65-b7dd-45d4-86b0-f13a29ef8e66" containerName="main" Oct 29 23:29:47.197151 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:47.195716 5057 cpu_manager.go:414] "containerMap: removing container" podUID="21c72ab9-0742-4f8a-aee1-8f820d967c94" containerName="maas-api" Oct 29 23:29:47.197151 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:47.195723 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21c72ab9-0742-4f8a-aee1-8f820d967c94" containerName="maas-api" Oct 29 23:29:47.197151 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:47.196063 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="f29e8b65-b7dd-45d4-86b0-f13a29ef8e66" containerName="main" Oct 29 23:29:47.197151 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:47.196093 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="21c72ab9-0742-4f8a-aee1-8f820d967c94" containerName="maas-api" Oct 29 23:29:47.197151 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:47.196120 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0c5087d-e510-4841-a25c-dd6908304d61" containerName="payload-pre-processing" Oct 29 23:29:47.197151 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:47.196139 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="3481d9c3-f557-4b9d-b0f2-b0c0bdb81bd9" containerName="payload-processing" Oct 29 23:29:47.197151 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:47.196147 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="02be284e-3e49-4fa7-8475-a5a79f8997b1" containerName="maas-api" Oct 29 23:29:47.197151 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:47.196165 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="5917154d-50a3-4166-94e0-533aef08f790" containerName="payload-pre-processing" Oct 29 23:29:47.197151 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:47.196182 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="268baf9a-3c0a-4ac5-a170-e69a7f3d1cfc" containerName="payload-processing" Oct 29 23:29:47.197151 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:47.196194 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3fc11c6-03c9-4314-99c3-f857b149415c" containerName="main" Oct 29 23:29:47.197151 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:47.196208 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3d4a1e8-5e7a-4894-811f-5bb731d83b0d" containerName="maas-api" Oct 29 23:29:47.197151 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:47.196227 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d4b783f-0071-4524-804e-396c53f069eb" containerName="payload-pre-processing" Oct 29 23:29:47.208461 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:47.208418 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:29:47.209907 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:47.209877 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b18b56d5-jmkc5" Oct 29 23:29:47.216403 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:47.216378 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b18b56d5-jmkc5"] Oct 29 23:29:47.224707 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:47.224681 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:29:47.224961 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:47.224935 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dmx57\"" Oct 29 23:29:47.303397 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:47.303361 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce4fe136-400e-4285-9af1-bfcd51109ae2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b18b56d5-jmkc5\" (UID: \"ce4fe136-400e-4285-9af1-bfcd51109ae2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b18b56d5-jmkc5" Oct 29 23:29:47.303608 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:47.303524 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce4fe136-400e-4285-9af1-bfcd51109ae2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b18b56d5-jmkc5\" (UID: \"ce4fe136-400e-4285-9af1-bfcd51109ae2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b18b56d5-jmkc5" Oct 29 23:29:47.303608 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:47.303547 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mq6j2\" (UniqueName: \"kubernetes.io/projected/ce4fe136-400e-4285-9af1-bfcd51109ae2-kube-api-access-mq6j2\") pod \"maas-api-revoke-sub-b18b56d5-jmkc5\" (UID: \"ce4fe136-400e-4285-9af1-bfcd51109ae2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b18b56d5-jmkc5" Oct 29 23:29:47.321465 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:47.321153 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:29:47.332906 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:47.332873 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="02be284e-3e49-4fa7-8475-a5a79f8997b1" path="/var/lib/kubelet/pods/02be284e-3e49-4fa7-8475-a5a79f8997b1/volumes" Oct 29 23:29:47.405381 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:47.405316 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce4fe136-400e-4285-9af1-bfcd51109ae2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b18b56d5-jmkc5\" (UID: \"ce4fe136-400e-4285-9af1-bfcd51109ae2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b18b56d5-jmkc5" Oct 29 23:29:47.405832 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:47.405498 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce4fe136-400e-4285-9af1-bfcd51109ae2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b18b56d5-jmkc5\" (UID: \"ce4fe136-400e-4285-9af1-bfcd51109ae2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b18b56d5-jmkc5" Oct 29 23:29:47.405832 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:47.405530 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mq6j2\" (UniqueName: \"kubernetes.io/projected/ce4fe136-400e-4285-9af1-bfcd51109ae2-kube-api-access-mq6j2\") pod \"maas-api-revoke-sub-b18b56d5-jmkc5\" (UID: \"ce4fe136-400e-4285-9af1-bfcd51109ae2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b18b56d5-jmkc5" Oct 29 23:29:47.406205 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:47.406180 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce4fe136-400e-4285-9af1-bfcd51109ae2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b18b56d5-jmkc5\" (UID: \"ce4fe136-400e-4285-9af1-bfcd51109ae2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b18b56d5-jmkc5" Oct 29 23:29:47.407821 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:47.407798 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce4fe136-400e-4285-9af1-bfcd51109ae2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b18b56d5-jmkc5\" (UID: \"ce4fe136-400e-4285-9af1-bfcd51109ae2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b18b56d5-jmkc5" Oct 29 23:29:47.436936 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:47.436906 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mq6j2\" (UniqueName: \"kubernetes.io/projected/ce4fe136-400e-4285-9af1-bfcd51109ae2-kube-api-access-mq6j2\") pod \"maas-api-revoke-sub-b18b56d5-jmkc5\" (UID: \"ce4fe136-400e-4285-9af1-bfcd51109ae2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b18b56d5-jmkc5" Oct 29 23:29:47.538660 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:47.537971 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b18b56d5-jmkc5" Oct 29 23:29:47.545527 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:47.545006 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-bclh8"] Oct 29 23:29:47.545527 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:47.545252 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-bclh8" podUID="b510ea66-642c-49ae-a600-fab375005cb6" containerName="istio-proxy" containerID="cri-o://cdd38dbe12d687c21c3991a8a66a66291152d9ca48c363c07b83c4c1846ac9b1" gracePeriod=30 Oct 29 23:29:47.549755 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:47.549723 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:29:47.714734 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:47.714045 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:29:47.877677 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:47.877633 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b18b56d5-jmkc5"] Oct 29 23:29:47.879189 ip-10-0-140-74 kubenswrapper[5057]: W1029 23:29:47.879156 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce4fe136_400e_4285_9af1_bfcd51109ae2.slice/crio-816b6c37a26e255f5fb135b92b8da3e1ce0f45f90af0b0fef7aaeecc0c629f29 WatchSource:0}: Error finding container 816b6c37a26e255f5fb135b92b8da3e1ce0f45f90af0b0fef7aaeecc0c629f29: Status 404 returned error can't find the container with id 816b6c37a26e255f5fb135b92b8da3e1ce0f45f90af0b0fef7aaeecc0c629f29 Oct 29 23:29:48.023360 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:48.023316 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b18b56d5-jmkc5" event={"ID":"ce4fe136-400e-4285-9af1-bfcd51109ae2","Type":"ContainerStarted","Data":"816b6c37a26e255f5fb135b92b8da3e1ce0f45f90af0b0fef7aaeecc0c629f29"} Oct 29 23:29:49.029625 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:29:49.029585 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d2dnn" containerName="extractor" Oct 29 23:29:49.035792 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:49.035765 5057 generic.go:356] "Generic (PLEG): container finished" podID="ce4fe136-400e-4285-9af1-bfcd51109ae2" containerID="7f5b3796c81b9190c2bb45cb2c5d2c79ce4a53bfc0433ff8412f05232d42359f" exitCode=0 Oct 29 23:29:49.035925 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:49.035848 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b18b56d5-jmkc5" event={"ID":"ce4fe136-400e-4285-9af1-bfcd51109ae2","Type":"ContainerDied","Data":"7f5b3796c81b9190c2bb45cb2c5d2c79ce4a53bfc0433ff8412f05232d42359f"} Oct 29 23:29:49.200634 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:49.200602 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:29:49.332966 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:49.332933 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:29:49.441861 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:49.441816 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:29:49.655621 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:49.655532 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-f3d24e75-openshift-default-7b4d9c69f7-rd4pc"] Oct 29 23:29:49.655815 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:49.655748 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-f3d24e75-openshift-default-7b4d9c69f7-rd4pc" podUID="9c02e87f-b552-45f4-97d2-d54429170748" containerName="istio-proxy" containerID="cri-o://6d483f81071b9b331fcb3bb241eac4037d0ef484096e892adccddc5d082273df" gracePeriod=30 Oct 29 23:29:49.688527 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:49.688493 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:29:50.219505 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:50.219468 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b18b56d5-jmkc5" Oct 29 23:29:50.351812 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:50.351738 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce4fe136-400e-4285-9af1-bfcd51109ae2-kube-api-access-mq6j2\" (UniqueName: \"kubernetes.io/projected/ce4fe136-400e-4285-9af1-bfcd51109ae2-kube-api-access-mq6j2\") pod \"ce4fe136-400e-4285-9af1-bfcd51109ae2\" (UID: \"ce4fe136-400e-4285-9af1-bfcd51109ae2\") " Oct 29 23:29:50.351812 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:50.351782 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ce4fe136-400e-4285-9af1-bfcd51109ae2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce4fe136-400e-4285-9af1-bfcd51109ae2-maas-api-xks-ca\") pod \"ce4fe136-400e-4285-9af1-bfcd51109ae2\" (UID: \"ce4fe136-400e-4285-9af1-bfcd51109ae2\") " Oct 29 23:29:50.352041 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:50.351889 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce4fe136-400e-4285-9af1-bfcd51109ae2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce4fe136-400e-4285-9af1-bfcd51109ae2-maas-api-openshift-ca\") pod \"ce4fe136-400e-4285-9af1-bfcd51109ae2\" (UID: \"ce4fe136-400e-4285-9af1-bfcd51109ae2\") " Oct 29 23:29:50.352286 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:50.352258 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce4fe136-400e-4285-9af1-bfcd51109ae2-maas-api-openshift-ca" pod "ce4fe136-400e-4285-9af1-bfcd51109ae2" (UID: "ce4fe136-400e-4285-9af1-bfcd51109ae2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:29:50.352470 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:50.352449 5057 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce4fe136-400e-4285-9af1-bfcd51109ae2-maas-api-openshift-ca\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:29:50.354051 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:50.354029 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce4fe136-400e-4285-9af1-bfcd51109ae2-maas-api-xks-ca" pod "ce4fe136-400e-4285-9af1-bfcd51109ae2" (UID: "ce4fe136-400e-4285-9af1-bfcd51109ae2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:29:50.354185 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:50.354164 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce4fe136-400e-4285-9af1-bfcd51109ae2-kube-api-access-mq6j2" pod "ce4fe136-400e-4285-9af1-bfcd51109ae2" (UID: "ce4fe136-400e-4285-9af1-bfcd51109ae2"). InnerVolumeSpecName "kube-api-access-mq6j2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:29:50.454125 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:50.454090 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mq6j2\" (UniqueName: \"kubernetes.io/projected/ce4fe136-400e-4285-9af1-bfcd51109ae2-kube-api-access-mq6j2\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:29:50.454125 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:50.454127 5057 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce4fe136-400e-4285-9af1-bfcd51109ae2-maas-api-xks-ca\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:29:51.029975 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:51.029940 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:29:51.054266 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:51.054236 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b18b56d5-jmkc5" Oct 29 23:29:51.054266 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:51.054260 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b18b56d5-jmkc5" event={"ID":"ce4fe136-400e-4285-9af1-bfcd51109ae2","Type":"ContainerDied","Data":"816b6c37a26e255f5fb135b92b8da3e1ce0f45f90af0b0fef7aaeecc0c629f29"} Oct 29 23:29:51.054534 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:51.054307 5057 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="816b6c37a26e255f5fb135b92b8da3e1ce0f45f90af0b0fef7aaeecc0c629f29" Oct 29 23:29:52.603020 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:52.602987 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:29:52.844725 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:52.844700 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-bclh8" Oct 29 23:29:52.988298 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:52.988266 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b510ea66-642c-49ae-a600-fab375005cb6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b510ea66-642c-49ae-a600-fab375005cb6-istio-envoy\") pod \"b510ea66-642c-49ae-a600-fab375005cb6\" (UID: \"b510ea66-642c-49ae-a600-fab375005cb6\") " Oct 29 23:29:52.988513 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:52.988311 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b510ea66-642c-49ae-a600-fab375005cb6-kube-api-access-9xr25\" (UniqueName: \"kubernetes.io/projected/b510ea66-642c-49ae-a600-fab375005cb6-kube-api-access-9xr25\") pod \"b510ea66-642c-49ae-a600-fab375005cb6\" (UID: \"b510ea66-642c-49ae-a600-fab375005cb6\") " Oct 29 23:29:52.988513 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:52.988367 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b510ea66-642c-49ae-a600-fab375005cb6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b510ea66-642c-49ae-a600-fab375005cb6-istio-podinfo\") pod \"b510ea66-642c-49ae-a600-fab375005cb6\" (UID: \"b510ea66-642c-49ae-a600-fab375005cb6\") " Oct 29 23:29:52.988513 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:52.988436 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b510ea66-642c-49ae-a600-fab375005cb6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b510ea66-642c-49ae-a600-fab375005cb6-workload-certs\") pod \"b510ea66-642c-49ae-a600-fab375005cb6\" (UID: \"b510ea66-642c-49ae-a600-fab375005cb6\") " Oct 29 23:29:52.988795 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:52.988757 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b510ea66-642c-49ae-a600-fab375005cb6-workload-certs" pod "b510ea66-642c-49ae-a600-fab375005cb6" (UID: "b510ea66-642c-49ae-a600-fab375005cb6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:29:52.988864 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:52.988820 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b510ea66-642c-49ae-a600-fab375005cb6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b510ea66-642c-49ae-a600-fab375005cb6-workload-socket\") pod \"b510ea66-642c-49ae-a600-fab375005cb6\" (UID: \"b510ea66-642c-49ae-a600-fab375005cb6\") " Oct 29 23:29:52.988927 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:52.988900 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b510ea66-642c-49ae-a600-fab375005cb6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b510ea66-642c-49ae-a600-fab375005cb6-credential-socket\") pod \"b510ea66-642c-49ae-a600-fab375005cb6\" (UID: \"b510ea66-642c-49ae-a600-fab375005cb6\") " Oct 29 23:29:52.989044 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:52.989026 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b510ea66-642c-49ae-a600-fab375005cb6-istio-token\" (UniqueName: \"kubernetes.io/projected/b510ea66-642c-49ae-a600-fab375005cb6-istio-token\") pod \"b510ea66-642c-49ae-a600-fab375005cb6\" (UID: \"b510ea66-642c-49ae-a600-fab375005cb6\") " Oct 29 23:29:52.989132 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:52.989060 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b510ea66-642c-49ae-a600-fab375005cb6-workload-socket" pod "b510ea66-642c-49ae-a600-fab375005cb6" (UID: "b510ea66-642c-49ae-a600-fab375005cb6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:29:52.989132 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:52.989121 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b510ea66-642c-49ae-a600-fab375005cb6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b510ea66-642c-49ae-a600-fab375005cb6-istiod-ca-cert\") pod \"b510ea66-642c-49ae-a600-fab375005cb6\" (UID: \"b510ea66-642c-49ae-a600-fab375005cb6\") " Oct 29 23:29:52.989236 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:52.989185 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b510ea66-642c-49ae-a600-fab375005cb6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b510ea66-642c-49ae-a600-fab375005cb6-istio-data\") pod \"b510ea66-642c-49ae-a600-fab375005cb6\" (UID: \"b510ea66-642c-49ae-a600-fab375005cb6\") " Oct 29 23:29:52.989236 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:52.989186 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b510ea66-642c-49ae-a600-fab375005cb6-credential-socket" pod "b510ea66-642c-49ae-a600-fab375005cb6" (UID: "b510ea66-642c-49ae-a600-fab375005cb6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:29:52.989896 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:52.989666 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b510ea66-642c-49ae-a600-fab375005cb6-istio-data" pod "b510ea66-642c-49ae-a600-fab375005cb6" (UID: "b510ea66-642c-49ae-a600-fab375005cb6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:29:52.989896 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:52.989672 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b510ea66-642c-49ae-a600-fab375005cb6-istiod-ca-cert" pod "b510ea66-642c-49ae-a600-fab375005cb6" (UID: "b510ea66-642c-49ae-a600-fab375005cb6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:29:52.989896 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:52.989818 5057 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b510ea66-642c-49ae-a600-fab375005cb6-workload-certs\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:29:52.989896 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:52.989835 5057 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b510ea66-642c-49ae-a600-fab375005cb6-workload-socket\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:29:52.989896 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:52.989848 5057 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b510ea66-642c-49ae-a600-fab375005cb6-credential-socket\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:29:52.989896 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:52.989863 5057 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b510ea66-642c-49ae-a600-fab375005cb6-istiod-ca-cert\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:29:52.989896 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:52.989876 5057 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b510ea66-642c-49ae-a600-fab375005cb6-istio-data\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:29:52.990772 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:52.990748 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b510ea66-642c-49ae-a600-fab375005cb6-istio-podinfo" pod "b510ea66-642c-49ae-a600-fab375005cb6" (UID: "b510ea66-642c-49ae-a600-fab375005cb6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:29:52.991523 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:52.991500 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b510ea66-642c-49ae-a600-fab375005cb6-istio-envoy" pod "b510ea66-642c-49ae-a600-fab375005cb6" (UID: "b510ea66-642c-49ae-a600-fab375005cb6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:29:52.991610 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:52.991536 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b510ea66-642c-49ae-a600-fab375005cb6-kube-api-access-9xr25" pod "b510ea66-642c-49ae-a600-fab375005cb6" (UID: "b510ea66-642c-49ae-a600-fab375005cb6"). InnerVolumeSpecName "kube-api-access-9xr25". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:29:52.991610 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:52.991553 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b510ea66-642c-49ae-a600-fab375005cb6-istio-token" pod "b510ea66-642c-49ae-a600-fab375005cb6" (UID: "b510ea66-642c-49ae-a600-fab375005cb6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:29:53.074189 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:53.074122 5057 generic.go:356] "Generic (PLEG): container finished" podID="b510ea66-642c-49ae-a600-fab375005cb6" containerID="cdd38dbe12d687c21c3991a8a66a66291152d9ca48c363c07b83c4c1846ac9b1" exitCode=0 Oct 29 23:29:53.074189 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:53.074160 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-bclh8" event={"ID":"b510ea66-642c-49ae-a600-fab375005cb6","Type":"ContainerDied","Data":"cdd38dbe12d687c21c3991a8a66a66291152d9ca48c363c07b83c4c1846ac9b1"} Oct 29 23:29:53.074189 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:53.074168 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-bclh8" Oct 29 23:29:53.074427 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:53.074200 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-bclh8" event={"ID":"b510ea66-642c-49ae-a600-fab375005cb6","Type":"ContainerDied","Data":"6c5a7f8eaa61c80ec547b019fc877bd764fea3599280aae13062b245b82c87e0"} Oct 29 23:29:53.074427 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:53.074222 5057 scope.go:122] "RemoveContainer" containerID="cdd38dbe12d687c21c3991a8a66a66291152d9ca48c363c07b83c4c1846ac9b1" Oct 29 23:29:53.086305 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:53.086280 5057 scope.go:122] "RemoveContainer" containerID="cdd38dbe12d687c21c3991a8a66a66291152d9ca48c363c07b83c4c1846ac9b1" Oct 29 23:29:53.086674 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:29:53.086649 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cdd38dbe12d687c21c3991a8a66a66291152d9ca48c363c07b83c4c1846ac9b1\": container with ID starting with cdd38dbe12d687c21c3991a8a66a66291152d9ca48c363c07b83c4c1846ac9b1 not found: ID does not exist" containerID="cdd38dbe12d687c21c3991a8a66a66291152d9ca48c363c07b83c4c1846ac9b1" Oct 29 23:29:53.086805 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:53.086683 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cdd38dbe12d687c21c3991a8a66a66291152d9ca48c363c07b83c4c1846ac9b1"} err="failed to get container status \"cdd38dbe12d687c21c3991a8a66a66291152d9ca48c363c07b83c4c1846ac9b1\": rpc error: code = NotFound desc = could not find container \"cdd38dbe12d687c21c3991a8a66a66291152d9ca48c363c07b83c4c1846ac9b1\": container with ID starting with cdd38dbe12d687c21c3991a8a66a66291152d9ca48c363c07b83c4c1846ac9b1 not found: ID does not exist" Oct 29 23:29:53.090977 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:53.090936 5057 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b510ea66-642c-49ae-a600-fab375005cb6-istio-token\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:29:53.090977 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:53.090965 5057 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b510ea66-642c-49ae-a600-fab375005cb6-istio-envoy\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:29:53.091154 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:53.090981 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9xr25\" (UniqueName: \"kubernetes.io/projected/b510ea66-642c-49ae-a600-fab375005cb6-kube-api-access-9xr25\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:29:53.091154 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:53.090997 5057 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b510ea66-642c-49ae-a600-fab375005cb6-istio-podinfo\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:29:53.132217 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:53.132177 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-bclh8"] Oct 29 23:29:53.136576 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:53.136551 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-bclh8"] Oct 29 23:29:53.234516 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:53.234480 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:29:53.331544 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:53.331464 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b510ea66-642c-49ae-a600-fab375005cb6" path="/var/lib/kubelet/pods/b510ea66-642c-49ae-a600-fab375005cb6/volumes" Oct 29 23:29:53.361923 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:53.361889 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:29:53.558358 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:53.554806 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-6bggl"] Oct 29 23:29:53.558358 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:53.555037 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-6bggl" podUID="bb648622-516b-4da4-90c9-8716c09bf068" containerName="istio-proxy" containerID="cri-o://528ce9db4e0b868b1b70bae2ab0f7de031eb494ae772b6dfdf93d60db8ac29b8" gracePeriod=30 Oct 29 23:29:53.602247 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:53.602100 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:29:54.196225 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:54.196190 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:29:54.954893 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:54.954864 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f3d24e75-openshift-default-7b4d9c69f7-rd4pc" Oct 29 23:29:55.018279 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:55.017536 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c02e87f-b552-45f4-97d2-d54429170748-istio-token\" (UniqueName: \"kubernetes.io/projected/9c02e87f-b552-45f4-97d2-d54429170748-istio-token\") pod \"9c02e87f-b552-45f4-97d2-d54429170748\" (UID: \"9c02e87f-b552-45f4-97d2-d54429170748\") " Oct 29 23:29:55.018279 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:55.017666 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c02e87f-b552-45f4-97d2-d54429170748-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c02e87f-b552-45f4-97d2-d54429170748-workload-certs\") pod \"9c02e87f-b552-45f4-97d2-d54429170748\" (UID: \"9c02e87f-b552-45f4-97d2-d54429170748\") " Oct 29 23:29:55.018279 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:55.017723 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c02e87f-b552-45f4-97d2-d54429170748-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c02e87f-b552-45f4-97d2-d54429170748-credential-socket\") pod \"9c02e87f-b552-45f4-97d2-d54429170748\" (UID: \"9c02e87f-b552-45f4-97d2-d54429170748\") " Oct 29 23:29:55.018279 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:55.017764 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c02e87f-b552-45f4-97d2-d54429170748-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c02e87f-b552-45f4-97d2-d54429170748-istiod-ca-cert\") pod \"9c02e87f-b552-45f4-97d2-d54429170748\" (UID: \"9c02e87f-b552-45f4-97d2-d54429170748\") " Oct 29 23:29:55.018279 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:55.017819 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c02e87f-b552-45f4-97d2-d54429170748-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c02e87f-b552-45f4-97d2-d54429170748-workload-socket\") pod \"9c02e87f-b552-45f4-97d2-d54429170748\" (UID: \"9c02e87f-b552-45f4-97d2-d54429170748\") " Oct 29 23:29:55.018279 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:55.017903 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c02e87f-b552-45f4-97d2-d54429170748-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c02e87f-b552-45f4-97d2-d54429170748-istio-envoy\") pod \"9c02e87f-b552-45f4-97d2-d54429170748\" (UID: \"9c02e87f-b552-45f4-97d2-d54429170748\") " Oct 29 23:29:55.018279 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:55.017938 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c02e87f-b552-45f4-97d2-d54429170748-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c02e87f-b552-45f4-97d2-d54429170748-istio-data\") pod \"9c02e87f-b552-45f4-97d2-d54429170748\" (UID: \"9c02e87f-b552-45f4-97d2-d54429170748\") " Oct 29 23:29:55.018279 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:55.018003 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9c02e87f-b552-45f4-97d2-d54429170748-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c02e87f-b552-45f4-97d2-d54429170748-istio-podinfo\") pod \"9c02e87f-b552-45f4-97d2-d54429170748\" (UID: \"9c02e87f-b552-45f4-97d2-d54429170748\") " Oct 29 23:29:55.018279 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:55.018058 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c02e87f-b552-45f4-97d2-d54429170748-kube-api-access-n7rw9\" (UniqueName: \"kubernetes.io/projected/9c02e87f-b552-45f4-97d2-d54429170748-kube-api-access-n7rw9\") pod \"9c02e87f-b552-45f4-97d2-d54429170748\" (UID: \"9c02e87f-b552-45f4-97d2-d54429170748\") " Oct 29 23:29:55.019263 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:55.019234 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c02e87f-b552-45f4-97d2-d54429170748-credential-socket" pod "9c02e87f-b552-45f4-97d2-d54429170748" (UID: "9c02e87f-b552-45f4-97d2-d54429170748"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:29:55.019850 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:55.019821 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c02e87f-b552-45f4-97d2-d54429170748-istio-data" pod "9c02e87f-b552-45f4-97d2-d54429170748" (UID: "9c02e87f-b552-45f4-97d2-d54429170748"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:29:55.019968 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:55.019890 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c02e87f-b552-45f4-97d2-d54429170748-workload-certs" pod "9c02e87f-b552-45f4-97d2-d54429170748" (UID: "9c02e87f-b552-45f4-97d2-d54429170748"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:29:55.019968 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:55.019937 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c02e87f-b552-45f4-97d2-d54429170748-workload-socket" pod "9c02e87f-b552-45f4-97d2-d54429170748" (UID: "9c02e87f-b552-45f4-97d2-d54429170748"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:29:55.020088 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:55.020072 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c02e87f-b552-45f4-97d2-d54429170748-istiod-ca-cert" pod "9c02e87f-b552-45f4-97d2-d54429170748" (UID: "9c02e87f-b552-45f4-97d2-d54429170748"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:29:55.021803 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:55.021779 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9c02e87f-b552-45f4-97d2-d54429170748-istio-podinfo" pod "9c02e87f-b552-45f4-97d2-d54429170748" (UID: "9c02e87f-b552-45f4-97d2-d54429170748"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:29:55.022190 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:55.021962 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c02e87f-b552-45f4-97d2-d54429170748-istio-token" pod "9c02e87f-b552-45f4-97d2-d54429170748" (UID: "9c02e87f-b552-45f4-97d2-d54429170748"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:29:55.022552 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:55.022526 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c02e87f-b552-45f4-97d2-d54429170748-kube-api-access-n7rw9" pod "9c02e87f-b552-45f4-97d2-d54429170748" (UID: "9c02e87f-b552-45f4-97d2-d54429170748"). InnerVolumeSpecName "kube-api-access-n7rw9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:29:55.023320 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:55.023296 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c02e87f-b552-45f4-97d2-d54429170748-istio-envoy" pod "9c02e87f-b552-45f4-97d2-d54429170748" (UID: "9c02e87f-b552-45f4-97d2-d54429170748"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:29:55.095614 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:55.095578 5057 generic.go:356] "Generic (PLEG): container finished" podID="9c02e87f-b552-45f4-97d2-d54429170748" containerID="6d483f81071b9b331fcb3bb241eac4037d0ef484096e892adccddc5d082273df" exitCode=0 Oct 29 23:29:55.095797 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:55.095640 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f3d24e75-openshift-default-7b4d9c69f7-rd4pc" Oct 29 23:29:55.095797 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:55.095639 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f3d24e75-openshift-default-7b4d9c69f7-rd4pc" event={"ID":"9c02e87f-b552-45f4-97d2-d54429170748","Type":"ContainerDied","Data":"6d483f81071b9b331fcb3bb241eac4037d0ef484096e892adccddc5d082273df"} Oct 29 23:29:55.095797 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:55.095783 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f3d24e75-openshift-default-7b4d9c69f7-rd4pc" event={"ID":"9c02e87f-b552-45f4-97d2-d54429170748","Type":"ContainerDied","Data":"cac90193515f3f2f4c933e310428ad76b4c1b17cfb09d3c7488c517c8e08f545"} Oct 29 23:29:55.095961 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:55.095808 5057 scope.go:122] "RemoveContainer" containerID="6d483f81071b9b331fcb3bb241eac4037d0ef484096e892adccddc5d082273df" Oct 29 23:29:55.110382 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:55.110356 5057 scope.go:122] "RemoveContainer" containerID="ffe99f5d1703980ea3c268cb83c005ab697c8dacbb0e060ef257ad2e96a03df2" Oct 29 23:29:55.120501 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:55.120472 5057 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c02e87f-b552-45f4-97d2-d54429170748-credential-socket\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:29:55.120501 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:55.120499 5057 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c02e87f-b552-45f4-97d2-d54429170748-istiod-ca-cert\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:29:55.120666 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:55.120516 5057 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c02e87f-b552-45f4-97d2-d54429170748-workload-socket\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:29:55.120666 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:55.120527 5057 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c02e87f-b552-45f4-97d2-d54429170748-istio-envoy\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:29:55.120666 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:55.120539 5057 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c02e87f-b552-45f4-97d2-d54429170748-istio-data\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:29:55.120666 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:55.120551 5057 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c02e87f-b552-45f4-97d2-d54429170748-istio-podinfo\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:29:55.120666 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:55.120563 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n7rw9\" (UniqueName: \"kubernetes.io/projected/9c02e87f-b552-45f4-97d2-d54429170748-kube-api-access-n7rw9\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:29:55.120666 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:55.120577 5057 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c02e87f-b552-45f4-97d2-d54429170748-istio-token\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:29:55.120666 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:55.120590 5057 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c02e87f-b552-45f4-97d2-d54429170748-workload-certs\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:29:55.121829 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:55.121808 5057 scope.go:122] "RemoveContainer" containerID="6d483f81071b9b331fcb3bb241eac4037d0ef484096e892adccddc5d082273df" Oct 29 23:29:55.122141 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:29:55.122123 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d483f81071b9b331fcb3bb241eac4037d0ef484096e892adccddc5d082273df\": container with ID starting with 6d483f81071b9b331fcb3bb241eac4037d0ef484096e892adccddc5d082273df not found: ID does not exist" containerID="6d483f81071b9b331fcb3bb241eac4037d0ef484096e892adccddc5d082273df" Oct 29 23:29:55.122207 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:55.122157 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d483f81071b9b331fcb3bb241eac4037d0ef484096e892adccddc5d082273df"} err="failed to get container status \"6d483f81071b9b331fcb3bb241eac4037d0ef484096e892adccddc5d082273df\": rpc error: code = NotFound desc = could not find container \"6d483f81071b9b331fcb3bb241eac4037d0ef484096e892adccddc5d082273df\": container with ID starting with 6d483f81071b9b331fcb3bb241eac4037d0ef484096e892adccddc5d082273df not found: ID does not exist" Oct 29 23:29:55.122207 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:55.122175 5057 scope.go:122] "RemoveContainer" containerID="ffe99f5d1703980ea3c268cb83c005ab697c8dacbb0e060ef257ad2e96a03df2" Oct 29 23:29:55.122442 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:29:55.122420 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ffe99f5d1703980ea3c268cb83c005ab697c8dacbb0e060ef257ad2e96a03df2\": container with ID starting with ffe99f5d1703980ea3c268cb83c005ab697c8dacbb0e060ef257ad2e96a03df2 not found: ID does not exist" containerID="ffe99f5d1703980ea3c268cb83c005ab697c8dacbb0e060ef257ad2e96a03df2" Oct 29 23:29:55.122516 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:55.122452 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ffe99f5d1703980ea3c268cb83c005ab697c8dacbb0e060ef257ad2e96a03df2"} err="failed to get container status \"ffe99f5d1703980ea3c268cb83c005ab697c8dacbb0e060ef257ad2e96a03df2\": rpc error: code = NotFound desc = could not find container \"ffe99f5d1703980ea3c268cb83c005ab697c8dacbb0e060ef257ad2e96a03df2\": container with ID starting with ffe99f5d1703980ea3c268cb83c005ab697c8dacbb0e060ef257ad2e96a03df2 not found: ID does not exist" Oct 29 23:29:55.151089 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:55.151056 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-f3d24e75-openshift-default-7b4d9c69f7-rd4pc"] Oct 29 23:29:55.154782 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:55.154756 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-f3d24e75-openshift-default-7b4d9c69f7-rd4pc"] Oct 29 23:29:55.342388 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:55.342354 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c02e87f-b552-45f4-97d2-d54429170748" path="/var/lib/kubelet/pods/9c02e87f-b552-45f4-97d2-d54429170748/volumes" Oct 29 23:29:55.846277 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:55.846243 5057 patch_prober.go:29] interesting pod/e2e-mt-f3d24e75-openshift-default-7b4d9c69f7-rd4pc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.250:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 23:29:55.846457 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:55.846307 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-f3d24e75-openshift-default-7b4d9c69f7-rd4pc" podUID="9c02e87f-b552-45f4-97d2-d54429170748" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.250:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 23:29:55.870169 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:55.870127 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:29:57.506818 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:57.506777 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:29:57.565525 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:57.565489 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1ce6b81-pdp5g"] Oct 29 23:29:57.567023 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:57.566555 5057 cpu_manager.go:414] "containerMap: removing container" podUID="9c02e87f-b552-45f4-97d2-d54429170748" containerName="istio-proxy" Oct 29 23:29:57.567023 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:57.566580 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c02e87f-b552-45f4-97d2-d54429170748" containerName="istio-proxy" Oct 29 23:29:57.567023 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:57.566609 5057 cpu_manager.go:414] "containerMap: removing container" podUID="b510ea66-642c-49ae-a600-fab375005cb6" containerName="istio-proxy" Oct 29 23:29:57.567023 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:57.566619 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b510ea66-642c-49ae-a600-fab375005cb6" containerName="istio-proxy" Oct 29 23:29:57.567023 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:57.566663 5057 cpu_manager.go:414] "containerMap: removing container" podUID="ce4fe136-400e-4285-9af1-bfcd51109ae2" containerName="revoke-keys" Oct 29 23:29:57.567023 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:57.566672 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce4fe136-400e-4285-9af1-bfcd51109ae2" containerName="revoke-keys" Oct 29 23:29:57.567023 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:57.566700 5057 cpu_manager.go:414] "containerMap: removing container" podUID="9c02e87f-b552-45f4-97d2-d54429170748" containerName="istio-proxy" Oct 29 23:29:57.567023 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:57.566708 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c02e87f-b552-45f4-97d2-d54429170748" containerName="istio-proxy" Oct 29 23:29:57.567023 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:57.566871 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce4fe136-400e-4285-9af1-bfcd51109ae2" containerName="revoke-keys" Oct 29 23:29:57.567023 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:57.566888 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c02e87f-b552-45f4-97d2-d54429170748" containerName="istio-proxy" Oct 29 23:29:57.567023 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:57.566901 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c02e87f-b552-45f4-97d2-d54429170748" containerName="istio-proxy" Oct 29 23:29:57.567023 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:57.566910 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="b510ea66-642c-49ae-a600-fab375005cb6" containerName="istio-proxy" Oct 29 23:29:57.572589 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:57.572568 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1ce6b81-pdp5g" Oct 29 23:29:57.587853 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:57.587828 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:29:57.592013 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:57.591983 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1ce6b81-pdp5g"] Oct 29 23:29:57.595629 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:57.595606 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dmx57\"" Oct 29 23:29:57.666296 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:57.666262 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwrll\" (UniqueName: \"kubernetes.io/projected/36aa03cd-9d75-41dd-9e2d-401a3792be1b-kube-api-access-pwrll\") pod \"maas-api-revoke-sub-c1ce6b81-pdp5g\" (UID: \"36aa03cd-9d75-41dd-9e2d-401a3792be1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1ce6b81-pdp5g" Oct 29 23:29:57.666521 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:57.666317 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36aa03cd-9d75-41dd-9e2d-401a3792be1b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1ce6b81-pdp5g\" (UID: \"36aa03cd-9d75-41dd-9e2d-401a3792be1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1ce6b81-pdp5g" Oct 29 23:29:57.666605 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:57.666583 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36aa03cd-9d75-41dd-9e2d-401a3792be1b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1ce6b81-pdp5g\" (UID: \"36aa03cd-9d75-41dd-9e2d-401a3792be1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1ce6b81-pdp5g" Oct 29 23:29:57.768072 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:57.768001 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36aa03cd-9d75-41dd-9e2d-401a3792be1b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1ce6b81-pdp5g\" (UID: \"36aa03cd-9d75-41dd-9e2d-401a3792be1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1ce6b81-pdp5g" Oct 29 23:29:57.768072 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:57.768045 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwrll\" (UniqueName: \"kubernetes.io/projected/36aa03cd-9d75-41dd-9e2d-401a3792be1b-kube-api-access-pwrll\") pod \"maas-api-revoke-sub-c1ce6b81-pdp5g\" (UID: \"36aa03cd-9d75-41dd-9e2d-401a3792be1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1ce6b81-pdp5g" Oct 29 23:29:57.768295 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:57.768075 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36aa03cd-9d75-41dd-9e2d-401a3792be1b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1ce6b81-pdp5g\" (UID: \"36aa03cd-9d75-41dd-9e2d-401a3792be1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1ce6b81-pdp5g" Oct 29 23:29:57.768703 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:57.768682 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36aa03cd-9d75-41dd-9e2d-401a3792be1b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1ce6b81-pdp5g\" (UID: \"36aa03cd-9d75-41dd-9e2d-401a3792be1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1ce6b81-pdp5g" Oct 29 23:29:57.770368 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:57.770350 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36aa03cd-9d75-41dd-9e2d-401a3792be1b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1ce6b81-pdp5g\" (UID: \"36aa03cd-9d75-41dd-9e2d-401a3792be1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1ce6b81-pdp5g" Oct 29 23:29:57.791018 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:57.790995 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwrll\" (UniqueName: \"kubernetes.io/projected/36aa03cd-9d75-41dd-9e2d-401a3792be1b-kube-api-access-pwrll\") pod \"maas-api-revoke-sub-c1ce6b81-pdp5g\" (UID: \"36aa03cd-9d75-41dd-9e2d-401a3792be1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1ce6b81-pdp5g" Oct 29 23:29:57.888397 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:57.888362 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1ce6b81-pdp5g" Oct 29 23:29:58.196492 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:58.196458 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1ce6b81-pdp5g"] Oct 29 23:29:58.197749 ip-10-0-140-74 kubenswrapper[5057]: W1029 23:29:58.197712 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36aa03cd_9d75_41dd_9e2d_401a3792be1b.slice/crio-a9fd5386b5ca6ffe8771937f5b7416834b11404389f287b1f0b07dd957276387 WatchSource:0}: Error finding container a9fd5386b5ca6ffe8771937f5b7416834b11404389f287b1f0b07dd957276387: Status 404 returned error can't find the container with id a9fd5386b5ca6ffe8771937f5b7416834b11404389f287b1f0b07dd957276387 Oct 29 23:29:58.852505 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:58.852481 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-6bggl" Oct 29 23:29:58.884086 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:58.884055 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb648622-516b-4da4-90c9-8716c09bf068-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb648622-516b-4da4-90c9-8716c09bf068-workload-socket\") pod \"bb648622-516b-4da4-90c9-8716c09bf068\" (UID: \"bb648622-516b-4da4-90c9-8716c09bf068\") " Oct 29 23:29:58.884246 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:58.884194 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb648622-516b-4da4-90c9-8716c09bf068-kube-api-access-m75v6\" (UniqueName: \"kubernetes.io/projected/bb648622-516b-4da4-90c9-8716c09bf068-kube-api-access-m75v6\") pod \"bb648622-516b-4da4-90c9-8716c09bf068\" (UID: \"bb648622-516b-4da4-90c9-8716c09bf068\") " Oct 29 23:29:58.884360 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:58.884278 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb648622-516b-4da4-90c9-8716c09bf068-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb648622-516b-4da4-90c9-8716c09bf068-credential-socket\") pod \"bb648622-516b-4da4-90c9-8716c09bf068\" (UID: \"bb648622-516b-4da4-90c9-8716c09bf068\") " Oct 29 23:29:58.884360 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:58.884343 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb648622-516b-4da4-90c9-8716c09bf068-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb648622-516b-4da4-90c9-8716c09bf068-istio-envoy\") pod \"bb648622-516b-4da4-90c9-8716c09bf068\" (UID: \"bb648622-516b-4da4-90c9-8716c09bf068\") " Oct 29 23:29:58.884497 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:58.884352 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb648622-516b-4da4-90c9-8716c09bf068-workload-socket" pod "bb648622-516b-4da4-90c9-8716c09bf068" (UID: "bb648622-516b-4da4-90c9-8716c09bf068"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:29:58.884497 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:58.884371 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb648622-516b-4da4-90c9-8716c09bf068-istio-token\" (UniqueName: \"kubernetes.io/projected/bb648622-516b-4da4-90c9-8716c09bf068-istio-token\") pod \"bb648622-516b-4da4-90c9-8716c09bf068\" (UID: \"bb648622-516b-4da4-90c9-8716c09bf068\") " Oct 29 23:29:58.884497 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:58.884413 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bb648622-516b-4da4-90c9-8716c09bf068-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb648622-516b-4da4-90c9-8716c09bf068-istio-podinfo\") pod \"bb648622-516b-4da4-90c9-8716c09bf068\" (UID: \"bb648622-516b-4da4-90c9-8716c09bf068\") " Oct 29 23:29:58.884664 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:58.884496 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb648622-516b-4da4-90c9-8716c09bf068-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb648622-516b-4da4-90c9-8716c09bf068-istio-data\") pod \"bb648622-516b-4da4-90c9-8716c09bf068\" (UID: \"bb648622-516b-4da4-90c9-8716c09bf068\") " Oct 29 23:29:58.884664 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:58.884578 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb648622-516b-4da4-90c9-8716c09bf068-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb648622-516b-4da4-90c9-8716c09bf068-workload-certs\") pod \"bb648622-516b-4da4-90c9-8716c09bf068\" (UID: \"bb648622-516b-4da4-90c9-8716c09bf068\") " Oct 29 23:29:58.884664 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:58.884582 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb648622-516b-4da4-90c9-8716c09bf068-credential-socket" pod "bb648622-516b-4da4-90c9-8716c09bf068" (UID: "bb648622-516b-4da4-90c9-8716c09bf068"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:29:58.884664 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:58.884628 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb648622-516b-4da4-90c9-8716c09bf068-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb648622-516b-4da4-90c9-8716c09bf068-istiod-ca-cert\") pod \"bb648622-516b-4da4-90c9-8716c09bf068\" (UID: \"bb648622-516b-4da4-90c9-8716c09bf068\") " Oct 29 23:29:58.884891 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:58.884806 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb648622-516b-4da4-90c9-8716c09bf068-workload-certs" pod "bb648622-516b-4da4-90c9-8716c09bf068" (UID: "bb648622-516b-4da4-90c9-8716c09bf068"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:29:58.884891 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:58.884819 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb648622-516b-4da4-90c9-8716c09bf068-istio-data" pod "bb648622-516b-4da4-90c9-8716c09bf068" (UID: "bb648622-516b-4da4-90c9-8716c09bf068"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:29:58.885152 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:58.885129 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb648622-516b-4da4-90c9-8716c09bf068-istiod-ca-cert" pod "bb648622-516b-4da4-90c9-8716c09bf068" (UID: "bb648622-516b-4da4-90c9-8716c09bf068"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:29:58.885298 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:58.885264 5057 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb648622-516b-4da4-90c9-8716c09bf068-istiod-ca-cert\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:29:58.885298 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:58.885289 5057 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb648622-516b-4da4-90c9-8716c09bf068-workload-socket\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:29:58.885484 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:58.885302 5057 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb648622-516b-4da4-90c9-8716c09bf068-credential-socket\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:29:58.885484 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:58.885317 5057 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb648622-516b-4da4-90c9-8716c09bf068-istio-data\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:29:58.885484 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:58.885346 5057 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb648622-516b-4da4-90c9-8716c09bf068-workload-certs\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:29:58.887577 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:58.887268 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb648622-516b-4da4-90c9-8716c09bf068-istio-token" pod "bb648622-516b-4da4-90c9-8716c09bf068" (UID: "bb648622-516b-4da4-90c9-8716c09bf068"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:29:58.887797 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:58.887762 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bb648622-516b-4da4-90c9-8716c09bf068-istio-podinfo" pod "bb648622-516b-4da4-90c9-8716c09bf068" (UID: "bb648622-516b-4da4-90c9-8716c09bf068"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:29:58.887924 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:58.887883 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb648622-516b-4da4-90c9-8716c09bf068-istio-envoy" pod "bb648622-516b-4da4-90c9-8716c09bf068" (UID: "bb648622-516b-4da4-90c9-8716c09bf068"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:29:58.900370 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:58.891600 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb648622-516b-4da4-90c9-8716c09bf068-kube-api-access-m75v6" pod "bb648622-516b-4da4-90c9-8716c09bf068" (UID: "bb648622-516b-4da4-90c9-8716c09bf068"). InnerVolumeSpecName "kube-api-access-m75v6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:29:58.987073 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:58.987036 5057 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb648622-516b-4da4-90c9-8716c09bf068-istio-envoy\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:29:58.987073 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:58.987067 5057 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb648622-516b-4da4-90c9-8716c09bf068-istio-token\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:29:58.987073 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:58.987079 5057 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb648622-516b-4da4-90c9-8716c09bf068-istio-podinfo\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:29:58.987308 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:58.987091 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m75v6\" (UniqueName: \"kubernetes.io/projected/bb648622-516b-4da4-90c9-8716c09bf068-kube-api-access-m75v6\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:29:59.098224 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:59.098133 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:29:59.136757 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:59.136716 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1ce6b81-pdp5g" event={"ID":"36aa03cd-9d75-41dd-9e2d-401a3792be1b","Type":"ContainerStarted","Data":"91bbc94e017db3e8f921ef54c84ff472cc86b33071c359d997885bdd14e215da"} Oct 29 23:29:59.136914 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:59.136786 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1ce6b81-pdp5g" event={"ID":"36aa03cd-9d75-41dd-9e2d-401a3792be1b","Type":"ContainerStarted","Data":"a9fd5386b5ca6ffe8771937f5b7416834b11404389f287b1f0b07dd957276387"} Oct 29 23:29:59.138365 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:59.138306 5057 generic.go:356] "Generic (PLEG): container finished" podID="bb648622-516b-4da4-90c9-8716c09bf068" containerID="528ce9db4e0b868b1b70bae2ab0f7de031eb494ae772b6dfdf93d60db8ac29b8" exitCode=0 Oct 29 23:29:59.138505 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:59.138364 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-6bggl" Oct 29 23:29:59.138505 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:59.138374 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-6bggl" event={"ID":"bb648622-516b-4da4-90c9-8716c09bf068","Type":"ContainerDied","Data":"528ce9db4e0b868b1b70bae2ab0f7de031eb494ae772b6dfdf93d60db8ac29b8"} Oct 29 23:29:59.138505 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:59.138423 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-6bggl" event={"ID":"bb648622-516b-4da4-90c9-8716c09bf068","Type":"ContainerDied","Data":"4e9169f2e12ab4050ddcd6cc73f9be384cb5b84ee773ede792de2900275b7c0a"} Oct 29 23:29:59.138505 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:59.138450 5057 scope.go:122] "RemoveContainer" containerID="528ce9db4e0b868b1b70bae2ab0f7de031eb494ae772b6dfdf93d60db8ac29b8" Oct 29 23:29:59.155480 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:59.155374 5057 scope.go:122] "RemoveContainer" containerID="528ce9db4e0b868b1b70bae2ab0f7de031eb494ae772b6dfdf93d60db8ac29b8" Oct 29 23:29:59.155792 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:29:59.155758 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"528ce9db4e0b868b1b70bae2ab0f7de031eb494ae772b6dfdf93d60db8ac29b8\": container with ID starting with 528ce9db4e0b868b1b70bae2ab0f7de031eb494ae772b6dfdf93d60db8ac29b8 not found: ID does not exist" containerID="528ce9db4e0b868b1b70bae2ab0f7de031eb494ae772b6dfdf93d60db8ac29b8" Oct 29 23:29:59.155876 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:59.155802 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"528ce9db4e0b868b1b70bae2ab0f7de031eb494ae772b6dfdf93d60db8ac29b8"} err="failed to get container status \"528ce9db4e0b868b1b70bae2ab0f7de031eb494ae772b6dfdf93d60db8ac29b8\": rpc error: code = NotFound desc = could not find container \"528ce9db4e0b868b1b70bae2ab0f7de031eb494ae772b6dfdf93d60db8ac29b8\": container with ID starting with 528ce9db4e0b868b1b70bae2ab0f7de031eb494ae772b6dfdf93d60db8ac29b8 not found: ID does not exist" Oct 29 23:29:59.180901 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:59.180857 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1ce6b81-pdp5g" podStartSLOduration=2.180842316 podStartE2EDuration="2.180842316s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:29:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:29:59.17557436 +0000 UTC m=+3379.958787786" watchObservedRunningTime="2026-10-29 23:29:59.180842316 +0000 UTC m=+3379.964055742" Oct 29 23:29:59.213967 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:59.213887 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-6bggl"] Oct 29 23:29:59.218322 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:59.218296 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-6bggl"] Oct 29 23:29:59.331202 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:29:59.331171 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb648622-516b-4da4-90c9-8716c09bf068" path="/var/lib/kubelet/pods/bb648622-516b-4da4-90c9-8716c09bf068/volumes" Oct 29 23:30:00.145316 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:00.145218 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-af237d-29888610-6hd66"] Oct 29 23:30:00.146442 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:00.146411 5057 cpu_manager.go:414] "containerMap: removing container" podUID="bb648622-516b-4da4-90c9-8716c09bf068" containerName="istio-proxy" Oct 29 23:30:00.146442 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:00.146438 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb648622-516b-4da4-90c9-8716c09bf068" containerName="istio-proxy" Oct 29 23:30:00.146686 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:00.146673 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb648622-516b-4da4-90c9-8716c09bf068" containerName="istio-proxy" Oct 29 23:30:00.156201 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:00.155651 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-67363b-29888610-mfd4n"] Oct 29 23:30:00.159008 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:00.158982 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-af237d-29888610-6hd66" Oct 29 23:30:00.162804 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:00.162775 5057 generic.go:356] "Generic (PLEG): container finished" podID="36aa03cd-9d75-41dd-9e2d-401a3792be1b" containerID="91bbc94e017db3e8f921ef54c84ff472cc86b33071c359d997885bdd14e215da" exitCode=0 Oct 29 23:30:00.165092 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:00.165069 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1ce6b81-pdp5g" event={"ID":"36aa03cd-9d75-41dd-9e2d-401a3792be1b","Type":"ContainerDied","Data":"91bbc94e017db3e8f921ef54c84ff472cc86b33071c359d997885bdd14e215da"} Oct 29 23:30:00.165219 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:00.165165 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-67363b-29888610-mfd4n" Oct 29 23:30:00.172886 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:00.172861 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e131af-29888610-mp6s5"] Oct 29 23:30:00.186220 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:00.186194 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-063b46-29888610-pbkwf"] Oct 29 23:30:00.186398 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:00.186319 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e131af-29888610-mp6s5" Oct 29 23:30:00.200020 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:00.198209 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-af237d-29888610-6hd66"] Oct 29 23:30:00.200020 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:00.198307 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-67363b-29888610-mfd4n"] Oct 29 23:30:00.200020 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:00.198477 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-063b46-29888610-pbkwf" Oct 29 23:30:00.204315 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:00.204272 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e131af-29888610-mp6s5"] Oct 29 23:30:00.208392 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:00.206984 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4caf8e1b-81e4-4cd0-a870-fe55dccdbe6d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-af237d-29888610-6hd66\" (UID: \"4caf8e1b-81e4-4cd0-a870-fe55dccdbe6d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-af237d-29888610-6hd66" Oct 29 23:30:00.208392 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:00.207084 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdnps\" (UniqueName: \"kubernetes.io/projected/4caf8e1b-81e4-4cd0-a870-fe55dccdbe6d-kube-api-access-xdnps\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-af237d-29888610-6hd66\" (UID: \"4caf8e1b-81e4-4cd0-a870-fe55dccdbe6d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-af237d-29888610-6hd66" Oct 29 23:30:00.215355 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:00.215094 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-063b46-29888610-pbkwf"] Oct 29 23:30:00.308564 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:00.308527 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dkv99\" (UniqueName: \"kubernetes.io/projected/73ffc0bf-5804-444b-b081-9f68d4168890-kube-api-access-dkv99\") pod \"maas-api-key-cleanup-e2e-worker-w1-e131af-29888610-mp6s5\" (UID: \"73ffc0bf-5804-444b-b081-9f68d4168890\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e131af-29888610-mp6s5" Oct 29 23:30:00.308758 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:00.308650 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vsl9k\" (UniqueName: \"kubernetes.io/projected/f4d887af-8034-47ad-834c-c0097b7b48a6-kube-api-access-vsl9k\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-67363b-29888610-mfd4n\" (UID: \"f4d887af-8034-47ad-834c-c0097b7b48a6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-67363b-29888610-mfd4n" Oct 29 23:30:00.308843 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:00.308764 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwmpr\" (UniqueName: \"kubernetes.io/projected/1d9a1f4c-5134-4b26-8292-bf3d588ba960-kube-api-access-dwmpr\") pod \"maas-api-key-cleanup-e2e-worker-w4-063b46-29888610-pbkwf\" (UID: \"1d9a1f4c-5134-4b26-8292-bf3d588ba960\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-063b46-29888610-pbkwf" Oct 29 23:30:00.308843 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:00.308800 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/73ffc0bf-5804-444b-b081-9f68d4168890-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-e131af-29888610-mp6s5\" (UID: \"73ffc0bf-5804-444b-b081-9f68d4168890\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e131af-29888610-mp6s5" Oct 29 23:30:00.308953 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:00.308877 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4caf8e1b-81e4-4cd0-a870-fe55dccdbe6d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-af237d-29888610-6hd66\" (UID: \"4caf8e1b-81e4-4cd0-a870-fe55dccdbe6d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-af237d-29888610-6hd66" Oct 29 23:30:00.308953 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:00.308942 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4d887af-8034-47ad-834c-c0097b7b48a6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-67363b-29888610-mfd4n\" (UID: \"f4d887af-8034-47ad-834c-c0097b7b48a6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-67363b-29888610-mfd4n" Oct 29 23:30:00.309063 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:00.309017 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdnps\" (UniqueName: \"kubernetes.io/projected/4caf8e1b-81e4-4cd0-a870-fe55dccdbe6d-kube-api-access-xdnps\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-af237d-29888610-6hd66\" (UID: \"4caf8e1b-81e4-4cd0-a870-fe55dccdbe6d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-af237d-29888610-6hd66" Oct 29 23:30:00.309120 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:00.309069 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d9a1f4c-5134-4b26-8292-bf3d588ba960-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-063b46-29888610-pbkwf\" (UID: \"1d9a1f4c-5134-4b26-8292-bf3d588ba960\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-063b46-29888610-pbkwf" Oct 29 23:30:00.309753 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:00.309728 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4caf8e1b-81e4-4cd0-a870-fe55dccdbe6d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-af237d-29888610-6hd66\" (UID: \"4caf8e1b-81e4-4cd0-a870-fe55dccdbe6d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-af237d-29888610-6hd66" Oct 29 23:30:00.331608 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:00.331578 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdnps\" (UniqueName: \"kubernetes.io/projected/4caf8e1b-81e4-4cd0-a870-fe55dccdbe6d-kube-api-access-xdnps\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-af237d-29888610-6hd66\" (UID: \"4caf8e1b-81e4-4cd0-a870-fe55dccdbe6d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-af237d-29888610-6hd66" Oct 29 23:30:00.410826 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:00.410740 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkv99\" (UniqueName: \"kubernetes.io/projected/73ffc0bf-5804-444b-b081-9f68d4168890-kube-api-access-dkv99\") pod \"maas-api-key-cleanup-e2e-worker-w1-e131af-29888610-mp6s5\" (UID: \"73ffc0bf-5804-444b-b081-9f68d4168890\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e131af-29888610-mp6s5" Oct 29 23:30:00.410989 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:00.410864 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vsl9k\" (UniqueName: \"kubernetes.io/projected/f4d887af-8034-47ad-834c-c0097b7b48a6-kube-api-access-vsl9k\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-67363b-29888610-mfd4n\" (UID: \"f4d887af-8034-47ad-834c-c0097b7b48a6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-67363b-29888610-mfd4n" Oct 29 23:30:00.410989 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:00.410949 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwmpr\" (UniqueName: \"kubernetes.io/projected/1d9a1f4c-5134-4b26-8292-bf3d588ba960-kube-api-access-dwmpr\") pod \"maas-api-key-cleanup-e2e-worker-w4-063b46-29888610-pbkwf\" (UID: \"1d9a1f4c-5134-4b26-8292-bf3d588ba960\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-063b46-29888610-pbkwf" Oct 29 23:30:00.410989 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:00.410977 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/73ffc0bf-5804-444b-b081-9f68d4168890-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-e131af-29888610-mp6s5\" (UID: \"73ffc0bf-5804-444b-b081-9f68d4168890\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e131af-29888610-mp6s5" Oct 29 23:30:00.411281 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:00.411066 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4d887af-8034-47ad-834c-c0097b7b48a6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-67363b-29888610-mfd4n\" (UID: \"f4d887af-8034-47ad-834c-c0097b7b48a6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-67363b-29888610-mfd4n" Oct 29 23:30:00.411281 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:00.411154 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d9a1f4c-5134-4b26-8292-bf3d588ba960-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-063b46-29888610-pbkwf\" (UID: \"1d9a1f4c-5134-4b26-8292-bf3d588ba960\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-063b46-29888610-pbkwf" Oct 29 23:30:00.411895 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:00.411870 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/73ffc0bf-5804-444b-b081-9f68d4168890-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-e131af-29888610-mp6s5\" (UID: \"73ffc0bf-5804-444b-b081-9f68d4168890\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e131af-29888610-mp6s5" Oct 29 23:30:00.411963 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:00.411938 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4d887af-8034-47ad-834c-c0097b7b48a6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-67363b-29888610-mfd4n\" (UID: \"f4d887af-8034-47ad-834c-c0097b7b48a6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-67363b-29888610-mfd4n" Oct 29 23:30:00.412005 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:00.411955 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d9a1f4c-5134-4b26-8292-bf3d588ba960-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-063b46-29888610-pbkwf\" (UID: \"1d9a1f4c-5134-4b26-8292-bf3d588ba960\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-063b46-29888610-pbkwf" Oct 29 23:30:00.433638 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:00.433613 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkv99\" (UniqueName: \"kubernetes.io/projected/73ffc0bf-5804-444b-b081-9f68d4168890-kube-api-access-dkv99\") pod \"maas-api-key-cleanup-e2e-worker-w1-e131af-29888610-mp6s5\" (UID: \"73ffc0bf-5804-444b-b081-9f68d4168890\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e131af-29888610-mp6s5" Oct 29 23:30:00.433820 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:00.433803 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwmpr\" (UniqueName: \"kubernetes.io/projected/1d9a1f4c-5134-4b26-8292-bf3d588ba960-kube-api-access-dwmpr\") pod \"maas-api-key-cleanup-e2e-worker-w4-063b46-29888610-pbkwf\" (UID: \"1d9a1f4c-5134-4b26-8292-bf3d588ba960\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-063b46-29888610-pbkwf" Oct 29 23:30:00.434142 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:00.434109 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vsl9k\" (UniqueName: \"kubernetes.io/projected/f4d887af-8034-47ad-834c-c0097b7b48a6-kube-api-access-vsl9k\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-67363b-29888610-mfd4n\" (UID: \"f4d887af-8034-47ad-834c-c0097b7b48a6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-67363b-29888610-mfd4n" Oct 29 23:30:00.486472 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:00.486440 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-af237d-29888610-6hd66" Oct 29 23:30:00.506689 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:00.506666 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-67363b-29888610-mfd4n" Oct 29 23:30:00.525890 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:00.525863 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e131af-29888610-mp6s5" Oct 29 23:30:00.537222 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:00.537194 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-063b46-29888610-pbkwf" Oct 29 23:30:00.811531 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:00.811500 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-af237d-29888610-6hd66"] Oct 29 23:30:00.811938 ip-10-0-140-74 kubenswrapper[5057]: W1029 23:30:00.811909 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4caf8e1b_81e4_4cd0_a870_fe55dccdbe6d.slice/crio-b427c1f5f08c7fbbe7295e75332f5384aa25bb375584e47e93feff75b7915b78 WatchSource:0}: Error finding container b427c1f5f08c7fbbe7295e75332f5384aa25bb375584e47e93feff75b7915b78: Status 404 returned error can't find the container with id b427c1f5f08c7fbbe7295e75332f5384aa25bb375584e47e93feff75b7915b78 Oct 29 23:30:00.814728 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:00.814697 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:30:00.837761 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:00.837733 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-67363b-29888610-mfd4n"] Oct 29 23:30:00.837987 ip-10-0-140-74 kubenswrapper[5057]: W1029 23:30:00.837959 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4d887af_8034_47ad_834c_c0097b7b48a6.slice/crio-3bf95f70309bfb3e6bc8e76ab110949a814eedb1ff920c24589a395fafe44c9e WatchSource:0}: Error finding container 3bf95f70309bfb3e6bc8e76ab110949a814eedb1ff920c24589a395fafe44c9e: Status 404 returned error can't find the container with id 3bf95f70309bfb3e6bc8e76ab110949a814eedb1ff920c24589a395fafe44c9e Oct 29 23:30:00.874700 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:00.874651 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e131af-29888610-mp6s5"] Oct 29 23:30:00.877982 ip-10-0-140-74 kubenswrapper[5057]: W1029 23:30:00.877950 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73ffc0bf_5804_444b_b081_9f68d4168890.slice/crio-313050aa892d698c139de3a625162a18ce6915713ae503bf487ff07591436c5d WatchSource:0}: Error finding container 313050aa892d698c139de3a625162a18ce6915713ae503bf487ff07591436c5d: Status 404 returned error can't find the container with id 313050aa892d698c139de3a625162a18ce6915713ae503bf487ff07591436c5d Oct 29 23:30:00.919956 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:00.919929 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-063b46-29888610-pbkwf"] Oct 29 23:30:00.920477 ip-10-0-140-74 kubenswrapper[5057]: W1029 23:30:00.920448 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d9a1f4c_5134_4b26_8292_bf3d588ba960.slice/crio-65ee8ca1a47bd4150387f2efe5c66667a22b3b7f1a5d441c71939eb0259f694d WatchSource:0}: Error finding container 65ee8ca1a47bd4150387f2efe5c66667a22b3b7f1a5d441c71939eb0259f694d: Status 404 returned error can't find the container with id 65ee8ca1a47bd4150387f2efe5c66667a22b3b7f1a5d441c71939eb0259f694d Oct 29 23:30:01.031234 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:01.031186 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-93b5809f-cxw6q"] Oct 29 23:30:01.034878 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:01.034846 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-93b5809f-cxw6q"] Oct 29 23:30:01.174590 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:01.174552 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-063b46-29888610-pbkwf" event={"ID":"1d9a1f4c-5134-4b26-8292-bf3d588ba960","Type":"ContainerStarted","Data":"65ee8ca1a47bd4150387f2efe5c66667a22b3b7f1a5d441c71939eb0259f694d"} Oct 29 23:30:01.176098 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:01.176062 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-67363b-29888610-mfd4n" event={"ID":"f4d887af-8034-47ad-834c-c0097b7b48a6","Type":"ContainerStarted","Data":"3bf95f70309bfb3e6bc8e76ab110949a814eedb1ff920c24589a395fafe44c9e"} Oct 29 23:30:01.178194 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:01.178168 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e131af-29888610-mp6s5" event={"ID":"73ffc0bf-5804-444b-b081-9f68d4168890","Type":"ContainerStarted","Data":"313050aa892d698c139de3a625162a18ce6915713ae503bf487ff07591436c5d"} Oct 29 23:30:01.180002 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:01.179974 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-af237d-29888610-6hd66" event={"ID":"4caf8e1b-81e4-4cd0-a870-fe55dccdbe6d","Type":"ContainerStarted","Data":"b427c1f5f08c7fbbe7295e75332f5384aa25bb375584e47e93feff75b7915b78"} Oct 29 23:30:01.332781 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:01.332751 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="480effeb-d77e-407d-95c7-d242ab181afe" path="/var/lib/kubelet/pods/480effeb-d77e-407d-95c7-d242ab181afe/volumes" Oct 29 23:30:01.333969 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:01.333946 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1ce6b81-pdp5g" Oct 29 23:30:01.426790 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:01.426755 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/36aa03cd-9d75-41dd-9e2d-401a3792be1b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36aa03cd-9d75-41dd-9e2d-401a3792be1b-maas-api-openshift-ca\") pod \"36aa03cd-9d75-41dd-9e2d-401a3792be1b\" (UID: \"36aa03cd-9d75-41dd-9e2d-401a3792be1b\") " Oct 29 23:30:01.426982 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:01.426959 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36aa03cd-9d75-41dd-9e2d-401a3792be1b-kube-api-access-pwrll\" (UniqueName: \"kubernetes.io/projected/36aa03cd-9d75-41dd-9e2d-401a3792be1b-kube-api-access-pwrll\") pod \"36aa03cd-9d75-41dd-9e2d-401a3792be1b\" (UID: \"36aa03cd-9d75-41dd-9e2d-401a3792be1b\") " Oct 29 23:30:01.427111 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:01.427094 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/36aa03cd-9d75-41dd-9e2d-401a3792be1b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36aa03cd-9d75-41dd-9e2d-401a3792be1b-maas-api-xks-ca\") pod \"36aa03cd-9d75-41dd-9e2d-401a3792be1b\" (UID: \"36aa03cd-9d75-41dd-9e2d-401a3792be1b\") " Oct 29 23:30:01.427556 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:01.427473 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36aa03cd-9d75-41dd-9e2d-401a3792be1b-maas-api-openshift-ca" pod "36aa03cd-9d75-41dd-9e2d-401a3792be1b" (UID: "36aa03cd-9d75-41dd-9e2d-401a3792be1b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:30:01.427835 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:01.427814 5057 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36aa03cd-9d75-41dd-9e2d-401a3792be1b-maas-api-openshift-ca\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:30:01.430192 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:01.430169 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36aa03cd-9d75-41dd-9e2d-401a3792be1b-kube-api-access-pwrll" pod "36aa03cd-9d75-41dd-9e2d-401a3792be1b" (UID: "36aa03cd-9d75-41dd-9e2d-401a3792be1b"). InnerVolumeSpecName "kube-api-access-pwrll". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:30:01.430381 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:01.430325 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/36aa03cd-9d75-41dd-9e2d-401a3792be1b-maas-api-xks-ca" pod "36aa03cd-9d75-41dd-9e2d-401a3792be1b" (UID: "36aa03cd-9d75-41dd-9e2d-401a3792be1b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:30:01.529533 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:01.529501 5057 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36aa03cd-9d75-41dd-9e2d-401a3792be1b-maas-api-xks-ca\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:30:01.529647 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:01.529536 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pwrll\" (UniqueName: \"kubernetes.io/projected/36aa03cd-9d75-41dd-9e2d-401a3792be1b-kube-api-access-pwrll\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:30:01.868853 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:01.868819 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w4-65e2d3/sub-test-model-65e2d3-kserve-554d948579-9m65w"] Oct 29 23:30:01.869683 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:01.869659 5057 cpu_manager.go:414] "containerMap: removing container" podUID="36aa03cd-9d75-41dd-9e2d-401a3792be1b" containerName="revoke-keys" Oct 29 23:30:01.869683 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:01.869679 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36aa03cd-9d75-41dd-9e2d-401a3792be1b" containerName="revoke-keys" Oct 29 23:30:01.869887 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:01.869829 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="36aa03cd-9d75-41dd-9e2d-401a3792be1b" containerName="revoke-keys" Oct 29 23:30:01.873716 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:01.873693 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-65e2d3/sub-test-model-65e2d3-kserve-554d948579-9m65w" Oct 29 23:30:01.890208 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:01.889661 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-65e2d3\"/\"openshift-service-ca.crt\"" Oct 29 23:30:01.896107 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:01.896071 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-65e2d3/sub-test-model-65e2d3-kserve-554d948579-9m65w"] Oct 29 23:30:01.900491 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:01.900463 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-65e2d3\"/\"kube-root-ca.crt\"" Oct 29 23:30:01.901486 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:01.901464 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-65e2d3\"/\"sub-test-model-65e2d3-kserve-self-signed-certs\"" Oct 29 23:30:01.901616 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:01.901580 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-65e2d3\"/\"default-dockercfg-m687x\"" Oct 29 23:30:01.934273 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:01.934242 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa71d8ce-5a91-4d79-bd0a-8f9997a3c9c4-tls-certs\") pod \"sub-test-model-65e2d3-kserve-554d948579-9m65w\" (UID: \"aa71d8ce-5a91-4d79-bd0a-8f9997a3c9c4\") " pod="ai-tenant-e2e-shared-b-w4-65e2d3/sub-test-model-65e2d3-kserve-554d948579-9m65w" Oct 29 23:30:01.934495 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:01.934285 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa71d8ce-5a91-4d79-bd0a-8f9997a3c9c4-tmp-dir\") pod \"sub-test-model-65e2d3-kserve-554d948579-9m65w\" (UID: \"aa71d8ce-5a91-4d79-bd0a-8f9997a3c9c4\") " pod="ai-tenant-e2e-shared-b-w4-65e2d3/sub-test-model-65e2d3-kserve-554d948579-9m65w" Oct 29 23:30:01.934495 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:01.934453 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa71d8ce-5a91-4d79-bd0a-8f9997a3c9c4-dshm\") pod \"sub-test-model-65e2d3-kserve-554d948579-9m65w\" (UID: \"aa71d8ce-5a91-4d79-bd0a-8f9997a3c9c4\") " pod="ai-tenant-e2e-shared-b-w4-65e2d3/sub-test-model-65e2d3-kserve-554d948579-9m65w" Oct 29 23:30:01.934632 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:01.934530 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rqd7\" (UniqueName: \"kubernetes.io/projected/aa71d8ce-5a91-4d79-bd0a-8f9997a3c9c4-kube-api-access-5rqd7\") pod \"sub-test-model-65e2d3-kserve-554d948579-9m65w\" (UID: \"aa71d8ce-5a91-4d79-bd0a-8f9997a3c9c4\") " pod="ai-tenant-e2e-shared-b-w4-65e2d3/sub-test-model-65e2d3-kserve-554d948579-9m65w" Oct 29 23:30:01.934632 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:01.934575 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa71d8ce-5a91-4d79-bd0a-8f9997a3c9c4-model-cache\") pod \"sub-test-model-65e2d3-kserve-554d948579-9m65w\" (UID: \"aa71d8ce-5a91-4d79-bd0a-8f9997a3c9c4\") " pod="ai-tenant-e2e-shared-b-w4-65e2d3/sub-test-model-65e2d3-kserve-554d948579-9m65w" Oct 29 23:30:01.934632 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:01.934610 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa71d8ce-5a91-4d79-bd0a-8f9997a3c9c4-home\") pod \"sub-test-model-65e2d3-kserve-554d948579-9m65w\" (UID: \"aa71d8ce-5a91-4d79-bd0a-8f9997a3c9c4\") " pod="ai-tenant-e2e-shared-b-w4-65e2d3/sub-test-model-65e2d3-kserve-554d948579-9m65w" Oct 29 23:30:01.974738 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:01.974707 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:30:02.035806 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:02.035775 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa71d8ce-5a91-4d79-bd0a-8f9997a3c9c4-dshm\") pod \"sub-test-model-65e2d3-kserve-554d948579-9m65w\" (UID: \"aa71d8ce-5a91-4d79-bd0a-8f9997a3c9c4\") " pod="ai-tenant-e2e-shared-b-w4-65e2d3/sub-test-model-65e2d3-kserve-554d948579-9m65w" Oct 29 23:30:02.035953 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:02.035888 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rqd7\" (UniqueName: \"kubernetes.io/projected/aa71d8ce-5a91-4d79-bd0a-8f9997a3c9c4-kube-api-access-5rqd7\") pod \"sub-test-model-65e2d3-kserve-554d948579-9m65w\" (UID: \"aa71d8ce-5a91-4d79-bd0a-8f9997a3c9c4\") " pod="ai-tenant-e2e-shared-b-w4-65e2d3/sub-test-model-65e2d3-kserve-554d948579-9m65w" Oct 29 23:30:02.036019 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:02.035949 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa71d8ce-5a91-4d79-bd0a-8f9997a3c9c4-model-cache\") pod \"sub-test-model-65e2d3-kserve-554d948579-9m65w\" (UID: \"aa71d8ce-5a91-4d79-bd0a-8f9997a3c9c4\") " pod="ai-tenant-e2e-shared-b-w4-65e2d3/sub-test-model-65e2d3-kserve-554d948579-9m65w" Oct 29 23:30:02.036019 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:02.036001 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa71d8ce-5a91-4d79-bd0a-8f9997a3c9c4-home\") pod \"sub-test-model-65e2d3-kserve-554d948579-9m65w\" (UID: \"aa71d8ce-5a91-4d79-bd0a-8f9997a3c9c4\") " pod="ai-tenant-e2e-shared-b-w4-65e2d3/sub-test-model-65e2d3-kserve-554d948579-9m65w" Oct 29 23:30:02.036155 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:02.036141 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa71d8ce-5a91-4d79-bd0a-8f9997a3c9c4-tls-certs\") pod \"sub-test-model-65e2d3-kserve-554d948579-9m65w\" (UID: \"aa71d8ce-5a91-4d79-bd0a-8f9997a3c9c4\") " pod="ai-tenant-e2e-shared-b-w4-65e2d3/sub-test-model-65e2d3-kserve-554d948579-9m65w" Oct 29 23:30:02.036229 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:02.036180 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa71d8ce-5a91-4d79-bd0a-8f9997a3c9c4-tmp-dir\") pod \"sub-test-model-65e2d3-kserve-554d948579-9m65w\" (UID: \"aa71d8ce-5a91-4d79-bd0a-8f9997a3c9c4\") " pod="ai-tenant-e2e-shared-b-w4-65e2d3/sub-test-model-65e2d3-kserve-554d948579-9m65w" Oct 29 23:30:02.036452 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:02.036423 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa71d8ce-5a91-4d79-bd0a-8f9997a3c9c4-model-cache\") pod \"sub-test-model-65e2d3-kserve-554d948579-9m65w\" (UID: \"aa71d8ce-5a91-4d79-bd0a-8f9997a3c9c4\") " pod="ai-tenant-e2e-shared-b-w4-65e2d3/sub-test-model-65e2d3-kserve-554d948579-9m65w" Oct 29 23:30:02.036576 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:02.036452 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa71d8ce-5a91-4d79-bd0a-8f9997a3c9c4-home\") pod \"sub-test-model-65e2d3-kserve-554d948579-9m65w\" (UID: \"aa71d8ce-5a91-4d79-bd0a-8f9997a3c9c4\") " pod="ai-tenant-e2e-shared-b-w4-65e2d3/sub-test-model-65e2d3-kserve-554d948579-9m65w" Oct 29 23:30:02.036652 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:02.036630 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa71d8ce-5a91-4d79-bd0a-8f9997a3c9c4-tmp-dir\") pod \"sub-test-model-65e2d3-kserve-554d948579-9m65w\" (UID: \"aa71d8ce-5a91-4d79-bd0a-8f9997a3c9c4\") " pod="ai-tenant-e2e-shared-b-w4-65e2d3/sub-test-model-65e2d3-kserve-554d948579-9m65w" Oct 29 23:30:02.038591 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:02.038535 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa71d8ce-5a91-4d79-bd0a-8f9997a3c9c4-dshm\") pod \"sub-test-model-65e2d3-kserve-554d948579-9m65w\" (UID: \"aa71d8ce-5a91-4d79-bd0a-8f9997a3c9c4\") " pod="ai-tenant-e2e-shared-b-w4-65e2d3/sub-test-model-65e2d3-kserve-554d948579-9m65w" Oct 29 23:30:02.038882 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:02.038863 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa71d8ce-5a91-4d79-bd0a-8f9997a3c9c4-tls-certs\") pod \"sub-test-model-65e2d3-kserve-554d948579-9m65w\" (UID: \"aa71d8ce-5a91-4d79-bd0a-8f9997a3c9c4\") " pod="ai-tenant-e2e-shared-b-w4-65e2d3/sub-test-model-65e2d3-kserve-554d948579-9m65w" Oct 29 23:30:02.059025 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:02.059002 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rqd7\" (UniqueName: \"kubernetes.io/projected/aa71d8ce-5a91-4d79-bd0a-8f9997a3c9c4-kube-api-access-5rqd7\") pod \"sub-test-model-65e2d3-kserve-554d948579-9m65w\" (UID: \"aa71d8ce-5a91-4d79-bd0a-8f9997a3c9c4\") " pod="ai-tenant-e2e-shared-b-w4-65e2d3/sub-test-model-65e2d3-kserve-554d948579-9m65w" Oct 29 23:30:02.191905 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:02.191879 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1ce6b81-pdp5g" Oct 29 23:30:02.191905 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:02.191894 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1ce6b81-pdp5g" event={"ID":"36aa03cd-9d75-41dd-9e2d-401a3792be1b","Type":"ContainerDied","Data":"a9fd5386b5ca6ffe8771937f5b7416834b11404389f287b1f0b07dd957276387"} Oct 29 23:30:02.192514 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:02.191948 5057 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a9fd5386b5ca6ffe8771937f5b7416834b11404389f287b1f0b07dd957276387" Oct 29 23:30:02.194284 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:02.194252 5057 generic.go:356] "Generic (PLEG): container finished" podID="f4d887af-8034-47ad-834c-c0097b7b48a6" containerID="267187e5547d8c0fa2e9cf67a785751116cfc719dc0d51da420e349dfdcfa200" exitCode=60 Oct 29 23:30:02.194459 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:02.194377 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-67363b-29888610-mfd4n" event={"ID":"f4d887af-8034-47ad-834c-c0097b7b48a6","Type":"ContainerDied","Data":"267187e5547d8c0fa2e9cf67a785751116cfc719dc0d51da420e349dfdcfa200"} Oct 29 23:30:02.194691 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:02.194670 5057 scope.go:122] "RemoveContainer" containerID="267187e5547d8c0fa2e9cf67a785751116cfc719dc0d51da420e349dfdcfa200" Oct 29 23:30:02.195064 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:02.194841 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-65e2d3/sub-test-model-65e2d3-kserve-554d948579-9m65w" Oct 29 23:30:02.196390 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:02.196326 5057 generic.go:356] "Generic (PLEG): container finished" podID="73ffc0bf-5804-444b-b081-9f68d4168890" containerID="4e724b27c95b1e3af500542241f3316819616d3d981015c6f35cb8cd92ac7b88" exitCode=60 Oct 29 23:30:02.196490 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:02.196373 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e131af-29888610-mp6s5" event={"ID":"73ffc0bf-5804-444b-b081-9f68d4168890","Type":"ContainerDied","Data":"4e724b27c95b1e3af500542241f3316819616d3d981015c6f35cb8cd92ac7b88"} Oct 29 23:30:02.196641 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:02.196602 5057 scope.go:122] "RemoveContainer" containerID="4e724b27c95b1e3af500542241f3316819616d3d981015c6f35cb8cd92ac7b88" Oct 29 23:30:02.200831 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:02.199657 5057 generic.go:356] "Generic (PLEG): container finished" podID="4caf8e1b-81e4-4cd0-a870-fe55dccdbe6d" containerID="8fc4f49461305d4406b68e40f8ca4e6dd341d9ad1e0b9f75cb4dc9494c5e1559" exitCode=60 Oct 29 23:30:02.200831 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:02.199759 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-af237d-29888610-6hd66" event={"ID":"4caf8e1b-81e4-4cd0-a870-fe55dccdbe6d","Type":"ContainerDied","Data":"8fc4f49461305d4406b68e40f8ca4e6dd341d9ad1e0b9f75cb4dc9494c5e1559"} Oct 29 23:30:02.200831 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:02.200012 5057 scope.go:122] "RemoveContainer" containerID="8fc4f49461305d4406b68e40f8ca4e6dd341d9ad1e0b9f75cb4dc9494c5e1559" Oct 29 23:30:02.202914 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:02.202098 5057 generic.go:356] "Generic (PLEG): container finished" podID="1d9a1f4c-5134-4b26-8292-bf3d588ba960" containerID="ee36d539ec2751f2b52d0935662e5ff69083ba1d8125a6b53e79b03c73c27d55" exitCode=60 Oct 29 23:30:02.202914 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:02.202249 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-063b46-29888610-pbkwf" event={"ID":"1d9a1f4c-5134-4b26-8292-bf3d588ba960","Type":"ContainerDied","Data":"ee36d539ec2751f2b52d0935662e5ff69083ba1d8125a6b53e79b03c73c27d55"} Oct 29 23:30:02.202914 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:02.202620 5057 scope.go:122] "RemoveContainer" containerID="ee36d539ec2751f2b52d0935662e5ff69083ba1d8125a6b53e79b03c73c27d55" Oct 29 23:30:02.536687 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:02.536655 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-65e2d3/sub-test-model-65e2d3-kserve-554d948579-9m65w"] Oct 29 23:30:02.538525 ip-10-0-140-74 kubenswrapper[5057]: W1029 23:30:02.538498 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaa71d8ce_5a91_4d79_bd0a_8f9997a3c9c4.slice/crio-3093d202840bfa750b1df20c96977d7d18ab94cac37e0b297bc102947dc66df1 WatchSource:0}: Error finding container 3093d202840bfa750b1df20c96977d7d18ab94cac37e0b297bc102947dc66df1: Status 404 returned error can't find the container with id 3093d202840bfa750b1df20c96977d7d18ab94cac37e0b297bc102947dc66df1 Oct 29 23:30:02.539548 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:02.539513 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:30:03.212046 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:03.212012 5057 generic.go:356] "Generic (PLEG): container finished" podID="4caf8e1b-81e4-4cd0-a870-fe55dccdbe6d" containerID="56851d02b4512166d7a7d71e4e882aa10bea743ed967a34761ae49729f6ef78e" exitCode=60 Oct 29 23:30:03.212394 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:03.212164 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-af237d-29888610-6hd66" event={"ID":"4caf8e1b-81e4-4cd0-a870-fe55dccdbe6d","Type":"ContainerDied","Data":"56851d02b4512166d7a7d71e4e882aa10bea743ed967a34761ae49729f6ef78e"} Oct 29 23:30:03.212394 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:03.212253 5057 scope.go:122] "RemoveContainer" containerID="8fc4f49461305d4406b68e40f8ca4e6dd341d9ad1e0b9f75cb4dc9494c5e1559" Oct 29 23:30:03.212618 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:03.212506 5057 scope.go:122] "RemoveContainer" containerID="56851d02b4512166d7a7d71e4e882aa10bea743ed967a34761ae49729f6ef78e" Oct 29 23:30:03.212951 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:30:03.212925 5057 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-af237d-29888610-6hd66_odh-ai-gateway-infra(4caf8e1b-81e4-4cd0-a870-fe55dccdbe6d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-af237d-29888610-6hd66" podUID="4caf8e1b-81e4-4cd0-a870-fe55dccdbe6d" Oct 29 23:30:03.214599 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:03.214575 5057 generic.go:356] "Generic (PLEG): container finished" podID="1d9a1f4c-5134-4b26-8292-bf3d588ba960" containerID="95f8da04f1bbf5848cf995e21621de78160f75c87cab126f85dcd962680dfc43" exitCode=60 Oct 29 23:30:03.214684 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:03.214652 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-063b46-29888610-pbkwf" event={"ID":"1d9a1f4c-5134-4b26-8292-bf3d588ba960","Type":"ContainerDied","Data":"95f8da04f1bbf5848cf995e21621de78160f75c87cab126f85dcd962680dfc43"} Oct 29 23:30:03.214892 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:03.214877 5057 scope.go:122] "RemoveContainer" containerID="95f8da04f1bbf5848cf995e21621de78160f75c87cab126f85dcd962680dfc43" Oct 29 23:30:03.215209 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:30:03.215178 5057 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-063b46-29888610-pbkwf_odh-ai-gateway-infra(1d9a1f4c-5134-4b26-8292-bf3d588ba960)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-063b46-29888610-pbkwf" podUID="1d9a1f4c-5134-4b26-8292-bf3d588ba960" Oct 29 23:30:03.216610 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:03.216589 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-67363b-29888610-mfd4n" event={"ID":"f4d887af-8034-47ad-834c-c0097b7b48a6","Type":"ContainerStarted","Data":"d6f61d474b79c513feed9c5ce9d31ed506fe4548b3bd7c3481f92284d6a4bc61"} Oct 29 23:30:03.218222 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:03.218198 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-65e2d3/sub-test-model-65e2d3-kserve-554d948579-9m65w" event={"ID":"aa71d8ce-5a91-4d79-bd0a-8f9997a3c9c4","Type":"ContainerStarted","Data":"2251a2a475e5968d2edc5ceabbe001d7c7896943fec54c739ac2df776dc22161"} Oct 29 23:30:03.218306 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:03.218236 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-65e2d3/sub-test-model-65e2d3-kserve-554d948579-9m65w" event={"ID":"aa71d8ce-5a91-4d79-bd0a-8f9997a3c9c4","Type":"ContainerStarted","Data":"3093d202840bfa750b1df20c96977d7d18ab94cac37e0b297bc102947dc66df1"} Oct 29 23:30:03.220061 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:03.220038 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e131af-29888610-mp6s5" event={"ID":"73ffc0bf-5804-444b-b081-9f68d4168890","Type":"ContainerStarted","Data":"5a231fa572e9853973ba688a094065ed409147dcc90dd9062ed8829ebe65a3a4"} Oct 29 23:30:03.290545 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:03.290498 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-67363b-29888610-mfd4n" podStartSLOduration=3.290483757 podStartE2EDuration="3.290483757s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:30:03.289166893 +0000 UTC m=+3384.072380338" watchObservedRunningTime="2026-10-29 23:30:03.290483757 +0000 UTC m=+3384.073697182" Oct 29 23:30:03.336474 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:03.336426 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e131af-29888610-mp6s5" podStartSLOduration=3.33641289 podStartE2EDuration="3.33641289s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:30:03.33371868 +0000 UTC m=+3384.116932107" watchObservedRunningTime="2026-10-29 23:30:03.33641289 +0000 UTC m=+3384.119626315" Oct 29 23:30:03.386968 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:30:03.386929 5057 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4d887af_8034_47ad_834c_c0097b7b48a6.slice/crio-conmon-d6f61d474b79c513feed9c5ce9d31ed506fe4548b3bd7c3481f92284d6a4bc61.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73ffc0bf_5804_444b_b081_9f68d4168890.slice/crio-5a231fa572e9853973ba688a094065ed409147dcc90dd9062ed8829ebe65a3a4.scope\": RecentStats: unable to find data in memory cache]" Oct 29 23:30:03.418110 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:03.418081 5057 scope.go:122] "RemoveContainer" containerID="ee36d539ec2751f2b52d0935662e5ff69083ba1d8125a6b53e79b03c73c27d55" Oct 29 23:30:03.418809 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:03.418711 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w4-65e2d3/sub-test-model-65e2d3-kserve-554d948579-9m65w" podStartSLOduration=2.41869141 podStartE2EDuration="2.41869141s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:30:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:30:03.41327901 +0000 UTC m=+3384.196492441" watchObservedRunningTime="2026-10-29 23:30:03.41869141 +0000 UTC m=+3384.201904837" Oct 29 23:30:04.177316 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:04.177278 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:30:04.195299 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:04.195269 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w4-65e2d3/sub-test-model-65e2d3-kserve-554d948579-9m65w" Oct 29 23:30:04.232088 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:04.232055 5057 generic.go:356] "Generic (PLEG): container finished" podID="73ffc0bf-5804-444b-b081-9f68d4168890" containerID="5a231fa572e9853973ba688a094065ed409147dcc90dd9062ed8829ebe65a3a4" exitCode=60 Oct 29 23:30:04.232508 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:04.232133 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e131af-29888610-mp6s5" event={"ID":"73ffc0bf-5804-444b-b081-9f68d4168890","Type":"ContainerDied","Data":"5a231fa572e9853973ba688a094065ed409147dcc90dd9062ed8829ebe65a3a4"} Oct 29 23:30:04.232508 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:04.232199 5057 scope.go:122] "RemoveContainer" containerID="4e724b27c95b1e3af500542241f3316819616d3d981015c6f35cb8cd92ac7b88" Oct 29 23:30:04.232508 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:04.232446 5057 scope.go:122] "RemoveContainer" containerID="5a231fa572e9853973ba688a094065ed409147dcc90dd9062ed8829ebe65a3a4" Oct 29 23:30:04.232808 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:30:04.232779 5057 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-e131af-29888610-mp6s5_odh-ai-gateway-infra(73ffc0bf-5804-444b-b081-9f68d4168890)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e131af-29888610-mp6s5" podUID="73ffc0bf-5804-444b-b081-9f68d4168890" Oct 29 23:30:04.234855 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:04.234838 5057 scope.go:122] "RemoveContainer" containerID="56851d02b4512166d7a7d71e4e882aa10bea743ed967a34761ae49729f6ef78e" Oct 29 23:30:04.235098 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:30:04.235077 5057 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-af237d-29888610-6hd66_odh-ai-gateway-infra(4caf8e1b-81e4-4cd0-a870-fe55dccdbe6d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-af237d-29888610-6hd66" podUID="4caf8e1b-81e4-4cd0-a870-fe55dccdbe6d" Oct 29 23:30:04.236620 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:04.236603 5057 scope.go:122] "RemoveContainer" containerID="95f8da04f1bbf5848cf995e21621de78160f75c87cab126f85dcd962680dfc43" Oct 29 23:30:04.236867 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:30:04.236851 5057 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-063b46-29888610-pbkwf_odh-ai-gateway-infra(1d9a1f4c-5134-4b26-8292-bf3d588ba960)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-063b46-29888610-pbkwf" podUID="1d9a1f4c-5134-4b26-8292-bf3d588ba960" Oct 29 23:30:04.238362 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:04.238323 5057 generic.go:356] "Generic (PLEG): container finished" podID="f4d887af-8034-47ad-834c-c0097b7b48a6" containerID="d6f61d474b79c513feed9c5ce9d31ed506fe4548b3bd7c3481f92284d6a4bc61" exitCode=60 Oct 29 23:30:04.238470 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:04.238372 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-67363b-29888610-mfd4n" event={"ID":"f4d887af-8034-47ad-834c-c0097b7b48a6","Type":"ContainerDied","Data":"d6f61d474b79c513feed9c5ce9d31ed506fe4548b3bd7c3481f92284d6a4bc61"} Oct 29 23:30:04.238641 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:04.238625 5057 scope.go:122] "RemoveContainer" containerID="d6f61d474b79c513feed9c5ce9d31ed506fe4548b3bd7c3481f92284d6a4bc61" Oct 29 23:30:04.238901 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:30:04.238880 5057 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-67363b-29888610-mfd4n_odh-ai-gateway-infra(f4d887af-8034-47ad-834c-c0097b7b48a6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-67363b-29888610-mfd4n" podUID="f4d887af-8034-47ad-834c-c0097b7b48a6" Oct 29 23:30:04.247973 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:04.247956 5057 scope.go:122] "RemoveContainer" containerID="267187e5547d8c0fa2e9cf67a785751116cfc719dc0d51da420e349dfdcfa200" Oct 29 23:30:05.250004 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:05.249969 5057 scope.go:122] "RemoveContainer" containerID="5a231fa572e9853973ba688a094065ed409147dcc90dd9062ed8829ebe65a3a4" Oct 29 23:30:05.250531 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:30:05.250418 5057 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-e131af-29888610-mp6s5_odh-ai-gateway-infra(73ffc0bf-5804-444b-b081-9f68d4168890)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e131af-29888610-mp6s5" podUID="73ffc0bf-5804-444b-b081-9f68d4168890" Oct 29 23:30:05.252309 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:05.252284 5057 scope.go:122] "RemoveContainer" containerID="d6f61d474b79c513feed9c5ce9d31ed506fe4548b3bd7c3481f92284d6a4bc61" Oct 29 23:30:05.252611 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:30:05.252589 5057 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-67363b-29888610-mfd4n_odh-ai-gateway-infra(f4d887af-8034-47ad-834c-c0097b7b48a6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-67363b-29888610-mfd4n" podUID="f4d887af-8034-47ad-834c-c0097b7b48a6" Oct 29 23:30:05.461298 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:05.461266 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-qs6cg"] Oct 29 23:30:05.467537 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:05.467512 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-qs6cg" Oct 29 23:30:05.484134 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:05.484100 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w4-af237d-openshift-default-dockercfg-9g5k4\"" Oct 29 23:30:05.490057 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:05.490028 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-qs6cg"] Oct 29 23:30:05.593118 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:05.593031 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9d46ee5-61dd-4a34-93bc-bcf127bb983a-workload-socket\") pod \"e2e-shared-a-w4-af237d-openshift-default-64f945b797-qs6cg\" (UID: \"b9d46ee5-61dd-4a34-93bc-bcf127bb983a\") " pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-qs6cg" Oct 29 23:30:05.593118 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:05.593088 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9d46ee5-61dd-4a34-93bc-bcf127bb983a-credential-socket\") pod \"e2e-shared-a-w4-af237d-openshift-default-64f945b797-qs6cg\" (UID: \"b9d46ee5-61dd-4a34-93bc-bcf127bb983a\") " pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-qs6cg" Oct 29 23:30:05.593388 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:05.593184 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9d46ee5-61dd-4a34-93bc-bcf127bb983a-istio-envoy\") pod \"e2e-shared-a-w4-af237d-openshift-default-64f945b797-qs6cg\" (UID: \"b9d46ee5-61dd-4a34-93bc-bcf127bb983a\") " pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-qs6cg" Oct 29 23:30:05.593388 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:05.593256 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9d46ee5-61dd-4a34-93bc-bcf127bb983a-workload-certs\") pod \"e2e-shared-a-w4-af237d-openshift-default-64f945b797-qs6cg\" (UID: \"b9d46ee5-61dd-4a34-93bc-bcf127bb983a\") " pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-qs6cg" Oct 29 23:30:05.593388 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:05.593310 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9d46ee5-61dd-4a34-93bc-bcf127bb983a-istiod-ca-cert\") pod \"e2e-shared-a-w4-af237d-openshift-default-64f945b797-qs6cg\" (UID: \"b9d46ee5-61dd-4a34-93bc-bcf127bb983a\") " pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-qs6cg" Oct 29 23:30:05.593535 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:05.593403 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9d46ee5-61dd-4a34-93bc-bcf127bb983a-istio-podinfo\") pod \"e2e-shared-a-w4-af237d-openshift-default-64f945b797-qs6cg\" (UID: \"b9d46ee5-61dd-4a34-93bc-bcf127bb983a\") " pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-qs6cg" Oct 29 23:30:05.593597 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:05.593535 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gw879\" (UniqueName: \"kubernetes.io/projected/b9d46ee5-61dd-4a34-93bc-bcf127bb983a-kube-api-access-gw879\") pod \"e2e-shared-a-w4-af237d-openshift-default-64f945b797-qs6cg\" (UID: \"b9d46ee5-61dd-4a34-93bc-bcf127bb983a\") " pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-qs6cg" Oct 29 23:30:05.593672 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:05.593609 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9d46ee5-61dd-4a34-93bc-bcf127bb983a-istio-token\") pod \"e2e-shared-a-w4-af237d-openshift-default-64f945b797-qs6cg\" (UID: \"b9d46ee5-61dd-4a34-93bc-bcf127bb983a\") " pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-qs6cg" Oct 29 23:30:05.593726 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:05.593680 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9d46ee5-61dd-4a34-93bc-bcf127bb983a-istio-data\") pod \"e2e-shared-a-w4-af237d-openshift-default-64f945b797-qs6cg\" (UID: \"b9d46ee5-61dd-4a34-93bc-bcf127bb983a\") " pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-qs6cg" Oct 29 23:30:05.695015 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:05.694977 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9d46ee5-61dd-4a34-93bc-bcf127bb983a-workload-socket\") pod \"e2e-shared-a-w4-af237d-openshift-default-64f945b797-qs6cg\" (UID: \"b9d46ee5-61dd-4a34-93bc-bcf127bb983a\") " pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-qs6cg" Oct 29 23:30:05.695201 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:05.695041 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9d46ee5-61dd-4a34-93bc-bcf127bb983a-credential-socket\") pod \"e2e-shared-a-w4-af237d-openshift-default-64f945b797-qs6cg\" (UID: \"b9d46ee5-61dd-4a34-93bc-bcf127bb983a\") " pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-qs6cg" Oct 29 23:30:05.695201 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:05.695085 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9d46ee5-61dd-4a34-93bc-bcf127bb983a-istio-envoy\") pod \"e2e-shared-a-w4-af237d-openshift-default-64f945b797-qs6cg\" (UID: \"b9d46ee5-61dd-4a34-93bc-bcf127bb983a\") " pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-qs6cg" Oct 29 23:30:05.695201 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:05.695153 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9d46ee5-61dd-4a34-93bc-bcf127bb983a-workload-certs\") pod \"e2e-shared-a-w4-af237d-openshift-default-64f945b797-qs6cg\" (UID: \"b9d46ee5-61dd-4a34-93bc-bcf127bb983a\") " pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-qs6cg" Oct 29 23:30:05.695201 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:05.695181 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9d46ee5-61dd-4a34-93bc-bcf127bb983a-istiod-ca-cert\") pod \"e2e-shared-a-w4-af237d-openshift-default-64f945b797-qs6cg\" (UID: \"b9d46ee5-61dd-4a34-93bc-bcf127bb983a\") " pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-qs6cg" Oct 29 23:30:05.695487 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:05.695218 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9d46ee5-61dd-4a34-93bc-bcf127bb983a-istio-podinfo\") pod \"e2e-shared-a-w4-af237d-openshift-default-64f945b797-qs6cg\" (UID: \"b9d46ee5-61dd-4a34-93bc-bcf127bb983a\") " pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-qs6cg" Oct 29 23:30:05.695487 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:05.695308 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gw879\" (UniqueName: \"kubernetes.io/projected/b9d46ee5-61dd-4a34-93bc-bcf127bb983a-kube-api-access-gw879\") pod \"e2e-shared-a-w4-af237d-openshift-default-64f945b797-qs6cg\" (UID: \"b9d46ee5-61dd-4a34-93bc-bcf127bb983a\") " pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-qs6cg" Oct 29 23:30:05.695487 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:05.695382 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9d46ee5-61dd-4a34-93bc-bcf127bb983a-istio-token\") pod \"e2e-shared-a-w4-af237d-openshift-default-64f945b797-qs6cg\" (UID: \"b9d46ee5-61dd-4a34-93bc-bcf127bb983a\") " pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-qs6cg" Oct 29 23:30:05.695487 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:05.695416 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9d46ee5-61dd-4a34-93bc-bcf127bb983a-istio-data\") pod \"e2e-shared-a-w4-af237d-openshift-default-64f945b797-qs6cg\" (UID: \"b9d46ee5-61dd-4a34-93bc-bcf127bb983a\") " pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-qs6cg" Oct 29 23:30:05.695698 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:05.695486 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9d46ee5-61dd-4a34-93bc-bcf127bb983a-workload-socket\") pod \"e2e-shared-a-w4-af237d-openshift-default-64f945b797-qs6cg\" (UID: \"b9d46ee5-61dd-4a34-93bc-bcf127bb983a\") " pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-qs6cg" Oct 29 23:30:05.695698 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:05.695592 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9d46ee5-61dd-4a34-93bc-bcf127bb983a-credential-socket\") pod \"e2e-shared-a-w4-af237d-openshift-default-64f945b797-qs6cg\" (UID: \"b9d46ee5-61dd-4a34-93bc-bcf127bb983a\") " pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-qs6cg" Oct 29 23:30:05.695807 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:05.695714 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9d46ee5-61dd-4a34-93bc-bcf127bb983a-workload-certs\") pod \"e2e-shared-a-w4-af237d-openshift-default-64f945b797-qs6cg\" (UID: \"b9d46ee5-61dd-4a34-93bc-bcf127bb983a\") " pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-qs6cg" Oct 29 23:30:05.695807 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:05.695788 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9d46ee5-61dd-4a34-93bc-bcf127bb983a-istio-data\") pod \"e2e-shared-a-w4-af237d-openshift-default-64f945b797-qs6cg\" (UID: \"b9d46ee5-61dd-4a34-93bc-bcf127bb983a\") " pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-qs6cg" Oct 29 23:30:05.696215 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:05.696191 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9d46ee5-61dd-4a34-93bc-bcf127bb983a-istiod-ca-cert\") pod \"e2e-shared-a-w4-af237d-openshift-default-64f945b797-qs6cg\" (UID: \"b9d46ee5-61dd-4a34-93bc-bcf127bb983a\") " pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-qs6cg" Oct 29 23:30:05.697459 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:05.697435 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9d46ee5-61dd-4a34-93bc-bcf127bb983a-istio-envoy\") pod \"e2e-shared-a-w4-af237d-openshift-default-64f945b797-qs6cg\" (UID: \"b9d46ee5-61dd-4a34-93bc-bcf127bb983a\") " pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-qs6cg" Oct 29 23:30:05.697972 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:05.697954 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9d46ee5-61dd-4a34-93bc-bcf127bb983a-istio-podinfo\") pod \"e2e-shared-a-w4-af237d-openshift-default-64f945b797-qs6cg\" (UID: \"b9d46ee5-61dd-4a34-93bc-bcf127bb983a\") " pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-qs6cg" Oct 29 23:30:05.718311 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:05.718284 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9d46ee5-61dd-4a34-93bc-bcf127bb983a-istio-token\") pod \"e2e-shared-a-w4-af237d-openshift-default-64f945b797-qs6cg\" (UID: \"b9d46ee5-61dd-4a34-93bc-bcf127bb983a\") " pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-qs6cg" Oct 29 23:30:05.721441 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:05.721413 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gw879\" (UniqueName: \"kubernetes.io/projected/b9d46ee5-61dd-4a34-93bc-bcf127bb983a-kube-api-access-gw879\") pod \"e2e-shared-a-w4-af237d-openshift-default-64f945b797-qs6cg\" (UID: \"b9d46ee5-61dd-4a34-93bc-bcf127bb983a\") " pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-qs6cg" Oct 29 23:30:05.781861 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:05.781837 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-qs6cg" Oct 29 23:30:05.848107 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:05.848066 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:30:06.108197 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:06.108154 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-qs6cg"] Oct 29 23:30:06.109601 ip-10-0-140-74 kubenswrapper[5057]: W1029 23:30:06.109558 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9d46ee5_61dd_4a34_93bc_bcf127bb983a.slice/crio-7f2774f799d6cca60c8cbf02bf4f03fce9423905204d738a3788c876737671b0 WatchSource:0}: Error finding container 7f2774f799d6cca60c8cbf02bf4f03fce9423905204d738a3788c876737671b0: Status 404 returned error can't find the container with id 7f2774f799d6cca60c8cbf02bf4f03fce9423905204d738a3788c876737671b0 Oct 29 23:30:06.112765 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:06.112728 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 23:30:06.112867 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:06.112794 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 23:30:06.112867 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:06.112840 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 23:30:06.263718 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:06.263679 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-qs6cg" event={"ID":"b9d46ee5-61dd-4a34-93bc-bcf127bb983a","Type":"ContainerStarted","Data":"593966f5ad6e4a9a0b51acf162b3273a294c35c1175ba5f00d1e84541724fee7"} Oct 29 23:30:06.264220 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:06.263741 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-qs6cg" event={"ID":"b9d46ee5-61dd-4a34-93bc-bcf127bb983a","Type":"ContainerStarted","Data":"7f2774f799d6cca60c8cbf02bf4f03fce9423905204d738a3788c876737671b0"} Oct 29 23:30:06.310008 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:06.309957 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-qs6cg" podStartSLOduration=1.3099422729999999 podStartE2EDuration="1.309942273s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:30:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:30:06.30819459 +0000 UTC m=+3387.091408016" watchObservedRunningTime="2026-10-29 23:30:06.309942273 +0000 UTC m=+3387.093155700" Oct 29 23:30:06.782881 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:06.782848 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-qs6cg" Oct 29 23:30:07.600354 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:07.600019 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:30:07.628541 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:07.625448 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w0-4504f1/sub-test-model-4504f1-kserve-657f694cf7-lqr5k"] Oct 29 23:30:07.631608 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:07.631575 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-4504f1/sub-test-model-4504f1-kserve-657f694cf7-lqr5k"] Oct 29 23:30:07.631762 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:07.631704 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-4504f1/sub-test-model-4504f1-kserve-657f694cf7-lqr5k" Oct 29 23:30:07.658521 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:07.658496 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-4504f1\"/\"sub-test-model-4504f1-kserve-self-signed-certs\"" Oct 29 23:30:07.658634 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:07.658548 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-4504f1\"/\"default-dockercfg-gqfj8\"" Oct 29 23:30:07.658634 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:07.658589 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-4504f1\"/\"openshift-service-ca.crt\"" Oct 29 23:30:07.658762 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:07.658746 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-4504f1\"/\"kube-root-ca.crt\"" Oct 29 23:30:07.717692 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:07.717652 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:30:07.728782 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:07.728750 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/630519d9-72be-441c-b013-eff2d5f360cc-dshm\") pod \"sub-test-model-4504f1-kserve-657f694cf7-lqr5k\" (UID: \"630519d9-72be-441c-b013-eff2d5f360cc\") " pod="ai-tenant-e2e-shared-a-w0-4504f1/sub-test-model-4504f1-kserve-657f694cf7-lqr5k" Oct 29 23:30:07.728935 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:07.728800 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5d8q5\" (UniqueName: \"kubernetes.io/projected/630519d9-72be-441c-b013-eff2d5f360cc-kube-api-access-5d8q5\") pod \"sub-test-model-4504f1-kserve-657f694cf7-lqr5k\" (UID: \"630519d9-72be-441c-b013-eff2d5f360cc\") " pod="ai-tenant-e2e-shared-a-w0-4504f1/sub-test-model-4504f1-kserve-657f694cf7-lqr5k" Oct 29 23:30:07.728977 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:07.728938 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/630519d9-72be-441c-b013-eff2d5f360cc-home\") pod \"sub-test-model-4504f1-kserve-657f694cf7-lqr5k\" (UID: \"630519d9-72be-441c-b013-eff2d5f360cc\") " pod="ai-tenant-e2e-shared-a-w0-4504f1/sub-test-model-4504f1-kserve-657f694cf7-lqr5k" Oct 29 23:30:07.729062 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:07.729042 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/630519d9-72be-441c-b013-eff2d5f360cc-model-cache\") pod \"sub-test-model-4504f1-kserve-657f694cf7-lqr5k\" (UID: \"630519d9-72be-441c-b013-eff2d5f360cc\") " pod="ai-tenant-e2e-shared-a-w0-4504f1/sub-test-model-4504f1-kserve-657f694cf7-lqr5k" Oct 29 23:30:07.729113 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:07.729081 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/630519d9-72be-441c-b013-eff2d5f360cc-tls-certs\") pod \"sub-test-model-4504f1-kserve-657f694cf7-lqr5k\" (UID: \"630519d9-72be-441c-b013-eff2d5f360cc\") " pod="ai-tenant-e2e-shared-a-w0-4504f1/sub-test-model-4504f1-kserve-657f694cf7-lqr5k" Oct 29 23:30:07.729152 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:07.729137 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/630519d9-72be-441c-b013-eff2d5f360cc-tmp-dir\") pod \"sub-test-model-4504f1-kserve-657f694cf7-lqr5k\" (UID: \"630519d9-72be-441c-b013-eff2d5f360cc\") " pod="ai-tenant-e2e-shared-a-w0-4504f1/sub-test-model-4504f1-kserve-657f694cf7-lqr5k" Oct 29 23:30:07.803353 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:07.802442 5057 patch_prober.go:29] interesting pod/e2e-shared-a-w4-af237d-openshift-default-64f945b797-qs6cg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.17:15021/healthz/ready\": dial tcp 10.132.1.17:15021: connect: connection refused" start-of-body= Oct 29 23:30:07.803353 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:07.802507 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-qs6cg" podUID="b9d46ee5-61dd-4a34-93bc-bcf127bb983a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.17:15021/healthz/ready\": dial tcp 10.132.1.17:15021: connect: connection refused" Oct 29 23:30:07.834355 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:07.830356 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/630519d9-72be-441c-b013-eff2d5f360cc-dshm\") pod \"sub-test-model-4504f1-kserve-657f694cf7-lqr5k\" (UID: \"630519d9-72be-441c-b013-eff2d5f360cc\") " pod="ai-tenant-e2e-shared-a-w0-4504f1/sub-test-model-4504f1-kserve-657f694cf7-lqr5k" Oct 29 23:30:07.834355 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:07.830449 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5d8q5\" (UniqueName: \"kubernetes.io/projected/630519d9-72be-441c-b013-eff2d5f360cc-kube-api-access-5d8q5\") pod \"sub-test-model-4504f1-kserve-657f694cf7-lqr5k\" (UID: \"630519d9-72be-441c-b013-eff2d5f360cc\") " pod="ai-tenant-e2e-shared-a-w0-4504f1/sub-test-model-4504f1-kserve-657f694cf7-lqr5k" Oct 29 23:30:07.834355 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:07.830533 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/630519d9-72be-441c-b013-eff2d5f360cc-home\") pod \"sub-test-model-4504f1-kserve-657f694cf7-lqr5k\" (UID: \"630519d9-72be-441c-b013-eff2d5f360cc\") " pod="ai-tenant-e2e-shared-a-w0-4504f1/sub-test-model-4504f1-kserve-657f694cf7-lqr5k" Oct 29 23:30:07.834355 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:07.830608 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/630519d9-72be-441c-b013-eff2d5f360cc-model-cache\") pod \"sub-test-model-4504f1-kserve-657f694cf7-lqr5k\" (UID: \"630519d9-72be-441c-b013-eff2d5f360cc\") " pod="ai-tenant-e2e-shared-a-w0-4504f1/sub-test-model-4504f1-kserve-657f694cf7-lqr5k" Oct 29 23:30:07.834355 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:07.830644 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/630519d9-72be-441c-b013-eff2d5f360cc-tls-certs\") pod \"sub-test-model-4504f1-kserve-657f694cf7-lqr5k\" (UID: \"630519d9-72be-441c-b013-eff2d5f360cc\") " pod="ai-tenant-e2e-shared-a-w0-4504f1/sub-test-model-4504f1-kserve-657f694cf7-lqr5k" Oct 29 23:30:07.834355 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:07.830705 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/630519d9-72be-441c-b013-eff2d5f360cc-tmp-dir\") pod \"sub-test-model-4504f1-kserve-657f694cf7-lqr5k\" (UID: \"630519d9-72be-441c-b013-eff2d5f360cc\") " pod="ai-tenant-e2e-shared-a-w0-4504f1/sub-test-model-4504f1-kserve-657f694cf7-lqr5k" Oct 29 23:30:07.834355 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:07.831601 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/630519d9-72be-441c-b013-eff2d5f360cc-home\") pod \"sub-test-model-4504f1-kserve-657f694cf7-lqr5k\" (UID: \"630519d9-72be-441c-b013-eff2d5f360cc\") " pod="ai-tenant-e2e-shared-a-w0-4504f1/sub-test-model-4504f1-kserve-657f694cf7-lqr5k" Oct 29 23:30:07.834355 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:07.832620 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/630519d9-72be-441c-b013-eff2d5f360cc-tmp-dir\") pod \"sub-test-model-4504f1-kserve-657f694cf7-lqr5k\" (UID: \"630519d9-72be-441c-b013-eff2d5f360cc\") " pod="ai-tenant-e2e-shared-a-w0-4504f1/sub-test-model-4504f1-kserve-657f694cf7-lqr5k" Oct 29 23:30:07.834355 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:07.832797 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/630519d9-72be-441c-b013-eff2d5f360cc-model-cache\") pod \"sub-test-model-4504f1-kserve-657f694cf7-lqr5k\" (UID: \"630519d9-72be-441c-b013-eff2d5f360cc\") " pod="ai-tenant-e2e-shared-a-w0-4504f1/sub-test-model-4504f1-kserve-657f694cf7-lqr5k" Oct 29 23:30:07.844400 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:07.837449 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/630519d9-72be-441c-b013-eff2d5f360cc-dshm\") pod \"sub-test-model-4504f1-kserve-657f694cf7-lqr5k\" (UID: \"630519d9-72be-441c-b013-eff2d5f360cc\") " pod="ai-tenant-e2e-shared-a-w0-4504f1/sub-test-model-4504f1-kserve-657f694cf7-lqr5k" Oct 29 23:30:07.844400 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:07.838484 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/630519d9-72be-441c-b013-eff2d5f360cc-tls-certs\") pod \"sub-test-model-4504f1-kserve-657f694cf7-lqr5k\" (UID: \"630519d9-72be-441c-b013-eff2d5f360cc\") " pod="ai-tenant-e2e-shared-a-w0-4504f1/sub-test-model-4504f1-kserve-657f694cf7-lqr5k" Oct 29 23:30:07.860045 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:07.859965 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5d8q5\" (UniqueName: \"kubernetes.io/projected/630519d9-72be-441c-b013-eff2d5f360cc-kube-api-access-5d8q5\") pod \"sub-test-model-4504f1-kserve-657f694cf7-lqr5k\" (UID: \"630519d9-72be-441c-b013-eff2d5f360cc\") " pod="ai-tenant-e2e-shared-a-w0-4504f1/sub-test-model-4504f1-kserve-657f694cf7-lqr5k" Oct 29 23:30:07.949347 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:07.949306 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-4504f1/sub-test-model-4504f1-kserve-657f694cf7-lqr5k" Oct 29 23:30:08.314097 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:08.314065 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-4504f1/sub-test-model-4504f1-kserve-657f694cf7-lqr5k"] Oct 29 23:30:08.314214 ip-10-0-140-74 kubenswrapper[5057]: W1029 23:30:08.314073 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod630519d9_72be_441c_b013_eff2d5f360cc.slice/crio-567052a2e252eadb6d89cf6a21bf838e78f9b8dad58ebb7b364cd1e0e8f517a8 WatchSource:0}: Error finding container 567052a2e252eadb6d89cf6a21bf838e78f9b8dad58ebb7b364cd1e0e8f517a8: Status 404 returned error can't find the container with id 567052a2e252eadb6d89cf6a21bf838e78f9b8dad58ebb7b364cd1e0e8f517a8 Oct 29 23:30:08.788354 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:08.784675 5057 patch_prober.go:29] interesting pod/e2e-shared-a-w4-af237d-openshift-default-64f945b797-qs6cg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.17:15021/healthz/ready\": dial tcp 10.132.1.17:15021: connect: connection refused" start-of-body= Oct 29 23:30:08.788354 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:08.784736 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-qs6cg" podUID="b9d46ee5-61dd-4a34-93bc-bcf127bb983a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.17:15021/healthz/ready\": dial tcp 10.132.1.17:15021: connect: connection refused" Oct 29 23:30:09.118613 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:09.118509 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:30:09.318350 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:09.317596 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-4504f1/sub-test-model-4504f1-kserve-657f694cf7-lqr5k" event={"ID":"630519d9-72be-441c-b013-eff2d5f360cc","Type":"ContainerStarted","Data":"6adf4ca5f058014e45234e783c824b561ea789ec1c95e6b88930111388b332e6"} Oct 29 23:30:09.318350 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:09.317648 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-4504f1/sub-test-model-4504f1-kserve-657f694cf7-lqr5k" event={"ID":"630519d9-72be-441c-b013-eff2d5f360cc","Type":"ContainerStarted","Data":"567052a2e252eadb6d89cf6a21bf838e78f9b8dad58ebb7b364cd1e0e8f517a8"} Oct 29 23:30:09.362960 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:09.362909 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w0-4504f1/sub-test-model-4504f1-kserve-657f694cf7-lqr5k" podStartSLOduration=2.362895318 podStartE2EDuration="2.362895318s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:30:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:30:09.360457256 +0000 UTC m=+3390.143670682" watchObservedRunningTime="2026-10-29 23:30:09.362895318 +0000 UTC m=+3390.146108774" Oct 29 23:30:09.474179 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:09.474147 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-lhtzn"] Oct 29 23:30:09.523526 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:09.523495 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-lhtzn" Oct 29 23:30:09.539760 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:09.539711 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-lhtzn"] Oct 29 23:30:09.554053 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:09.554027 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-b0c3c3-openshift-default-dockercfg-nzrxs\"" Oct 29 23:30:09.658302 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:09.658258 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6b816a6-841c-405c-b1ae-d6f9d1a4982b-istio-token\") pod \"e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-lhtzn\" (UID: \"b6b816a6-841c-405c-b1ae-d6f9d1a4982b\") " pod="openshift-ingress/e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-lhtzn" Oct 29 23:30:09.658518 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:09.658314 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6b816a6-841c-405c-b1ae-d6f9d1a4982b-istiod-ca-cert\") pod \"e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-lhtzn\" (UID: \"b6b816a6-841c-405c-b1ae-d6f9d1a4982b\") " pod="openshift-ingress/e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-lhtzn" Oct 29 23:30:09.658518 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:09.658455 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6b816a6-841c-405c-b1ae-d6f9d1a4982b-istio-envoy\") pod \"e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-lhtzn\" (UID: \"b6b816a6-841c-405c-b1ae-d6f9d1a4982b\") " pod="openshift-ingress/e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-lhtzn" Oct 29 23:30:09.658518 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:09.658493 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6b816a6-841c-405c-b1ae-d6f9d1a4982b-credential-socket\") pod \"e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-lhtzn\" (UID: \"b6b816a6-841c-405c-b1ae-d6f9d1a4982b\") " pod="openshift-ingress/e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-lhtzn" Oct 29 23:30:09.658639 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:09.658585 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6b816a6-841c-405c-b1ae-d6f9d1a4982b-istio-podinfo\") pod \"e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-lhtzn\" (UID: \"b6b816a6-841c-405c-b1ae-d6f9d1a4982b\") " pod="openshift-ingress/e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-lhtzn" Oct 29 23:30:09.658705 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:09.658690 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6b816a6-841c-405c-b1ae-d6f9d1a4982b-istio-data\") pod \"e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-lhtzn\" (UID: \"b6b816a6-841c-405c-b1ae-d6f9d1a4982b\") " pod="openshift-ingress/e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-lhtzn" Oct 29 23:30:09.658776 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:09.658763 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6b816a6-841c-405c-b1ae-d6f9d1a4982b-workload-certs\") pod \"e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-lhtzn\" (UID: \"b6b816a6-841c-405c-b1ae-d6f9d1a4982b\") " pod="openshift-ingress/e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-lhtzn" Oct 29 23:30:09.658906 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:09.658895 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6b816a6-841c-405c-b1ae-d6f9d1a4982b-workload-socket\") pod \"e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-lhtzn\" (UID: \"b6b816a6-841c-405c-b1ae-d6f9d1a4982b\") " pod="openshift-ingress/e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-lhtzn" Oct 29 23:30:09.658942 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:09.658931 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6mzkm\" (UniqueName: \"kubernetes.io/projected/b6b816a6-841c-405c-b1ae-d6f9d1a4982b-kube-api-access-6mzkm\") pod \"e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-lhtzn\" (UID: \"b6b816a6-841c-405c-b1ae-d6f9d1a4982b\") " pod="openshift-ingress/e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-lhtzn" Oct 29 23:30:09.760441 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:09.760354 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6b816a6-841c-405c-b1ae-d6f9d1a4982b-istio-data\") pod \"e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-lhtzn\" (UID: \"b6b816a6-841c-405c-b1ae-d6f9d1a4982b\") " pod="openshift-ingress/e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-lhtzn" Oct 29 23:30:09.760594 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:09.760449 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6b816a6-841c-405c-b1ae-d6f9d1a4982b-workload-certs\") pod \"e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-lhtzn\" (UID: \"b6b816a6-841c-405c-b1ae-d6f9d1a4982b\") " pod="openshift-ingress/e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-lhtzn" Oct 29 23:30:09.760594 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:09.760504 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6b816a6-841c-405c-b1ae-d6f9d1a4982b-workload-socket\") pod \"e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-lhtzn\" (UID: \"b6b816a6-841c-405c-b1ae-d6f9d1a4982b\") " pod="openshift-ingress/e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-lhtzn" Oct 29 23:30:09.760594 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:09.760538 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mzkm\" (UniqueName: \"kubernetes.io/projected/b6b816a6-841c-405c-b1ae-d6f9d1a4982b-kube-api-access-6mzkm\") pod \"e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-lhtzn\" (UID: \"b6b816a6-841c-405c-b1ae-d6f9d1a4982b\") " pod="openshift-ingress/e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-lhtzn" Oct 29 23:30:09.760771 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:09.760597 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6b816a6-841c-405c-b1ae-d6f9d1a4982b-istio-token\") pod \"e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-lhtzn\" (UID: \"b6b816a6-841c-405c-b1ae-d6f9d1a4982b\") " pod="openshift-ingress/e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-lhtzn" Oct 29 23:30:09.760771 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:09.760629 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6b816a6-841c-405c-b1ae-d6f9d1a4982b-istiod-ca-cert\") pod \"e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-lhtzn\" (UID: \"b6b816a6-841c-405c-b1ae-d6f9d1a4982b\") " pod="openshift-ingress/e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-lhtzn" Oct 29 23:30:09.760771 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:09.760677 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6b816a6-841c-405c-b1ae-d6f9d1a4982b-istio-envoy\") pod \"e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-lhtzn\" (UID: \"b6b816a6-841c-405c-b1ae-d6f9d1a4982b\") " pod="openshift-ingress/e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-lhtzn" Oct 29 23:30:09.760771 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:09.760703 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6b816a6-841c-405c-b1ae-d6f9d1a4982b-credential-socket\") pod \"e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-lhtzn\" (UID: \"b6b816a6-841c-405c-b1ae-d6f9d1a4982b\") " pod="openshift-ingress/e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-lhtzn" Oct 29 23:30:09.760771 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:09.760739 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6b816a6-841c-405c-b1ae-d6f9d1a4982b-istio-data\") pod \"e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-lhtzn\" (UID: \"b6b816a6-841c-405c-b1ae-d6f9d1a4982b\") " pod="openshift-ingress/e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-lhtzn" Oct 29 23:30:09.761033 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:09.760771 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6b816a6-841c-405c-b1ae-d6f9d1a4982b-istio-podinfo\") pod \"e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-lhtzn\" (UID: \"b6b816a6-841c-405c-b1ae-d6f9d1a4982b\") " pod="openshift-ingress/e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-lhtzn" Oct 29 23:30:09.761033 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:09.760789 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6b816a6-841c-405c-b1ae-d6f9d1a4982b-workload-certs\") pod \"e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-lhtzn\" (UID: \"b6b816a6-841c-405c-b1ae-d6f9d1a4982b\") " pod="openshift-ingress/e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-lhtzn" Oct 29 23:30:09.761033 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:09.760890 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6b816a6-841c-405c-b1ae-d6f9d1a4982b-workload-socket\") pod \"e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-lhtzn\" (UID: \"b6b816a6-841c-405c-b1ae-d6f9d1a4982b\") " pod="openshift-ingress/e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-lhtzn" Oct 29 23:30:09.761245 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:09.761216 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6b816a6-841c-405c-b1ae-d6f9d1a4982b-credential-socket\") pod \"e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-lhtzn\" (UID: \"b6b816a6-841c-405c-b1ae-d6f9d1a4982b\") " pod="openshift-ingress/e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-lhtzn" Oct 29 23:30:09.761425 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:09.761406 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6b816a6-841c-405c-b1ae-d6f9d1a4982b-istiod-ca-cert\") pod \"e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-lhtzn\" (UID: \"b6b816a6-841c-405c-b1ae-d6f9d1a4982b\") " pod="openshift-ingress/e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-lhtzn" Oct 29 23:30:09.763080 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:09.763048 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6b816a6-841c-405c-b1ae-d6f9d1a4982b-istio-envoy\") pod \"e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-lhtzn\" (UID: \"b6b816a6-841c-405c-b1ae-d6f9d1a4982b\") " pod="openshift-ingress/e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-lhtzn" Oct 29 23:30:09.763236 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:09.763216 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6b816a6-841c-405c-b1ae-d6f9d1a4982b-istio-podinfo\") pod \"e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-lhtzn\" (UID: \"b6b816a6-841c-405c-b1ae-d6f9d1a4982b\") " pod="openshift-ingress/e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-lhtzn" Oct 29 23:30:09.794781 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:09.793691 5057 patch_prober.go:29] interesting pod/e2e-shared-a-w4-af237d-openshift-default-64f945b797-qs6cg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.17:15021/healthz/ready\": dial tcp 10.132.1.17:15021: connect: connection refused" start-of-body= Oct 29 23:30:09.794781 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:09.793746 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-qs6cg" podUID="b9d46ee5-61dd-4a34-93bc-bcf127bb983a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.17:15021/healthz/ready\": dial tcp 10.132.1.17:15021: connect: connection refused" Oct 29 23:30:09.806069 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:09.806042 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6mzkm\" (UniqueName: \"kubernetes.io/projected/b6b816a6-841c-405c-b1ae-d6f9d1a4982b-kube-api-access-6mzkm\") pod \"e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-lhtzn\" (UID: \"b6b816a6-841c-405c-b1ae-d6f9d1a4982b\") " pod="openshift-ingress/e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-lhtzn" Oct 29 23:30:09.806674 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:09.806653 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6b816a6-841c-405c-b1ae-d6f9d1a4982b-istio-token\") pod \"e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-lhtzn\" (UID: \"b6b816a6-841c-405c-b1ae-d6f9d1a4982b\") " pod="openshift-ingress/e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-lhtzn" Oct 29 23:30:09.866242 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:09.866215 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-lhtzn" Oct 29 23:30:09.950201 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:09.950170 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w0-4504f1/sub-test-model-4504f1-kserve-657f694cf7-lqr5k" Oct 29 23:30:10.230457 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:10.230424 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-lhtzn"] Oct 29 23:30:10.232666 ip-10-0-140-74 kubenswrapper[5057]: W1029 23:30:10.232635 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6b816a6_841c_405c_b1ae_d6f9d1a4982b.slice/crio-4145922ed14e1f83347a7e6563ceee735a559c680b838d90bb900260a83e5126 WatchSource:0}: Error finding container 4145922ed14e1f83347a7e6563ceee735a559c680b838d90bb900260a83e5126: Status 404 returned error can't find the container with id 4145922ed14e1f83347a7e6563ceee735a559c680b838d90bb900260a83e5126 Oct 29 23:30:10.236029 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:10.235992 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 23:30:10.236132 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:10.236083 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 23:30:10.236132 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:10.236115 5057 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 29 23:30:10.331260 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:10.331213 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-lhtzn" event={"ID":"b6b816a6-841c-405c-b1ae-d6f9d1a4982b","Type":"ContainerStarted","Data":"4145922ed14e1f83347a7e6563ceee735a559c680b838d90bb900260a83e5126"} Oct 29 23:30:10.691923 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:10.691787 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:30:10.791356 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:10.787754 5057 patch_prober.go:29] interesting pod/e2e-shared-a-w4-af237d-openshift-default-64f945b797-qs6cg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.17:15021/healthz/ready\": dial tcp 10.132.1.17:15021: connect: connection refused" start-of-body= Oct 29 23:30:10.791356 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:10.787819 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-qs6cg" podUID="b9d46ee5-61dd-4a34-93bc-bcf127bb983a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.17:15021/healthz/ready\": dial tcp 10.132.1.17:15021: connect: connection refused" Oct 29 23:30:11.358622 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:11.358579 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-lhtzn" event={"ID":"b6b816a6-841c-405c-b1ae-d6f9d1a4982b","Type":"ContainerStarted","Data":"3394233411ab32e0309e5f6408aa9ad64cde57d4c143fdc27c4a4518de954876"} Oct 29 23:30:11.435366 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:11.431501 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-lhtzn" podStartSLOduration=2.431478271 podStartE2EDuration="2.431478271s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:30:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:30:11.407634204 +0000 UTC m=+3392.190847630" watchObservedRunningTime="2026-10-29 23:30:11.431478271 +0000 UTC m=+3392.214691697" Oct 29 23:30:11.787355 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:11.784688 5057 patch_prober.go:29] interesting pod/e2e-shared-a-w4-af237d-openshift-default-64f945b797-qs6cg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.17:15021/healthz/ready\": dial tcp 10.132.1.17:15021: connect: connection refused" start-of-body= Oct 29 23:30:11.787355 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:11.784749 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-qs6cg" podUID="b9d46ee5-61dd-4a34-93bc-bcf127bb983a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.17:15021/healthz/ready\": dial tcp 10.132.1.17:15021: connect: connection refused" Oct 29 23:30:11.866957 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:11.866918 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-lhtzn" Oct 29 23:30:11.868894 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:11.868867 5057 patch_prober.go:29] interesting pod/e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-lhtzn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.19:15021/healthz/ready\": dial tcp 10.132.1.19:15021: connect: connection refused" start-of-body= Oct 29 23:30:11.869029 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:11.868912 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-lhtzn" podUID="b6b816a6-841c-405c-b1ae-d6f9d1a4982b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.19:15021/healthz/ready\": dial tcp 10.132.1.19:15021: connect: connection refused" Oct 29 23:30:12.198409 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:12.197271 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w4-65e2d3/sub-test-model-65e2d3-kserve-554d948579-9m65w" Oct 29 23:30:12.253813 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:12.253781 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w4-65e2d3/sub-test-model-65e2d3-kserve-554d948579-9m65w" Oct 29 23:30:12.298355 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:12.297483 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:30:12.452240 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:12.452147 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w4-65e2d3/sub-test-model-65e2d3-kserve-554d948579-9m65w" Oct 29 23:30:12.793410 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:12.791426 5057 patch_prober.go:29] interesting pod/e2e-shared-a-w4-af237d-openshift-default-64f945b797-qs6cg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.17:15021/healthz/ready\": dial tcp 10.132.1.17:15021: connect: connection refused" start-of-body= Oct 29 23:30:12.793410 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:12.791500 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-qs6cg" podUID="b9d46ee5-61dd-4a34-93bc-bcf127bb983a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.17:15021/healthz/ready\": dial tcp 10.132.1.17:15021: connect: connection refused" Oct 29 23:30:12.867211 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:12.867177 5057 patch_prober.go:29] interesting pod/e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-lhtzn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.19:15021/healthz/ready\": dial tcp 10.132.1.19:15021: connect: connection refused" start-of-body= Oct 29 23:30:12.867417 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:12.867228 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-lhtzn" podUID="b6b816a6-841c-405c-b1ae-d6f9d1a4982b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.19:15021/healthz/ready\": dial tcp 10.132.1.19:15021: connect: connection refused" Oct 29 23:30:13.805355 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:13.797710 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-qs6cg" Oct 29 23:30:13.805355 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:13.799107 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-qs6cg" Oct 29 23:30:13.805942 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:13.805520 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-qs6cg" Oct 29 23:30:13.866829 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:13.866796 5057 patch_prober.go:29] interesting pod/e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-lhtzn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.19:15021/healthz/ready\": dial tcp 10.132.1.19:15021: connect: connection refused" start-of-body= Oct 29 23:30:13.867029 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:13.866879 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-lhtzn" podUID="b6b816a6-841c-405c-b1ae-d6f9d1a4982b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.19:15021/healthz/ready\": dial tcp 10.132.1.19:15021: connect: connection refused" Oct 29 23:30:13.899354 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:13.894158 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:30:14.866728 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:14.866692 5057 patch_prober.go:29] interesting pod/e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-lhtzn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.19:15021/healthz/ready\": dial tcp 10.132.1.19:15021: connect: connection refused" start-of-body= Oct 29 23:30:14.867160 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:14.866754 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-lhtzn" podUID="b6b816a6-841c-405c-b1ae-d6f9d1a4982b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.19:15021/healthz/ready\": dial tcp 10.132.1.19:15021: connect: connection refused" Oct 29 23:30:15.320762 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:15.320735 5057 scope.go:122] "RemoveContainer" containerID="95f8da04f1bbf5848cf995e21621de78160f75c87cab126f85dcd962680dfc43" Oct 29 23:30:15.320965 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:15.320893 5057 scope.go:122] "RemoveContainer" containerID="d6f61d474b79c513feed9c5ce9d31ed506fe4548b3bd7c3481f92284d6a4bc61" Oct 29 23:30:15.368212 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:15.368186 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:30:15.866687 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:15.866654 5057 patch_prober.go:29] interesting pod/e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-lhtzn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.19:15021/healthz/ready\": dial tcp 10.132.1.19:15021: connect: connection refused" start-of-body= Oct 29 23:30:15.866836 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:15.866705 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-lhtzn" podUID="b6b816a6-841c-405c-b1ae-d6f9d1a4982b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.19:15021/healthz/ready\": dial tcp 10.132.1.19:15021: connect: connection refused" Oct 29 23:30:16.321013 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:16.320974 5057 scope.go:122] "RemoveContainer" containerID="56851d02b4512166d7a7d71e4e882aa10bea743ed967a34761ae49729f6ef78e" Oct 29 23:30:16.442910 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:16.442876 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-063b46-29888610-pbkwf" event={"ID":"1d9a1f4c-5134-4b26-8292-bf3d588ba960","Type":"ContainerStarted","Data":"03d1ab4f864f360874d6d817a279ed991ceb3163ebbb85860968f4c421cfb89f"} Oct 29 23:30:16.445425 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:16.445397 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-67363b-29888610-mfd4n" event={"ID":"f4d887af-8034-47ad-834c-c0097b7b48a6","Type":"ContainerStarted","Data":"db1bf4b89096aff86d229c5bddff5af3b6d9c24aead23efa1459fba2c80d5305"} Oct 29 23:30:16.517182 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:16.517123 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-063b46-29888610-pbkwf" podStartSLOduration=16.517105086 podStartE2EDuration="16.517105086s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:30:16.51463684 +0000 UTC m=+3397.297850267" watchObservedRunningTime="2026-10-29 23:30:16.517105086 +0000 UTC m=+3397.300318514" Oct 29 23:30:16.867315 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:16.867227 5057 patch_prober.go:29] interesting pod/e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-lhtzn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.19:15021/healthz/ready\": dial tcp 10.132.1.19:15021: connect: connection refused" start-of-body= Oct 29 23:30:16.867315 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:16.867283 5057 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-lhtzn" podUID="b6b816a6-841c-405c-b1ae-d6f9d1a4982b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.19:15021/healthz/ready\": dial tcp 10.132.1.19:15021: connect: connection refused" Oct 29 23:30:16.909354 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:16.909309 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:30:17.020141 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:17.020073 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v2tfm"] Oct 29 23:30:17.320607 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:17.320582 5057 scope.go:122] "RemoveContainer" containerID="5a231fa572e9853973ba688a094065ed409147dcc90dd9062ed8829ebe65a3a4" Oct 29 23:30:17.458407 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:17.458369 5057 generic.go:356] "Generic (PLEG): container finished" podID="4caf8e1b-81e4-4cd0-a870-fe55dccdbe6d" containerID="351be18bcc30788de478f458c3b739fb9aa9d6204d1395db0488161e4911ee92" exitCode=60 Oct 29 23:30:17.458407 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:17.458389 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-af237d-29888610-6hd66" event={"ID":"4caf8e1b-81e4-4cd0-a870-fe55dccdbe6d","Type":"ContainerDied","Data":"351be18bcc30788de478f458c3b739fb9aa9d6204d1395db0488161e4911ee92"} Oct 29 23:30:17.458666 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:17.458467 5057 scope.go:122] "RemoveContainer" containerID="56851d02b4512166d7a7d71e4e882aa10bea743ed967a34761ae49729f6ef78e" Oct 29 23:30:17.458898 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:17.458874 5057 scope.go:122] "RemoveContainer" containerID="351be18bcc30788de478f458c3b739fb9aa9d6204d1395db0488161e4911ee92" Oct 29 23:30:17.459294 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:30:17.459267 5057 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-af237d-29888610-6hd66_odh-ai-gateway-infra(4caf8e1b-81e4-4cd0-a870-fe55dccdbe6d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-af237d-29888610-6hd66" podUID="4caf8e1b-81e4-4cd0-a870-fe55dccdbe6d" Oct 29 23:30:17.460813 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:17.460789 5057 generic.go:356] "Generic (PLEG): container finished" podID="1d9a1f4c-5134-4b26-8292-bf3d588ba960" containerID="03d1ab4f864f360874d6d817a279ed991ceb3163ebbb85860968f4c421cfb89f" exitCode=60 Oct 29 23:30:17.460919 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:17.460865 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-063b46-29888610-pbkwf" event={"ID":"1d9a1f4c-5134-4b26-8292-bf3d588ba960","Type":"ContainerDied","Data":"03d1ab4f864f360874d6d817a279ed991ceb3163ebbb85860968f4c421cfb89f"} Oct 29 23:30:17.461287 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:17.461265 5057 scope.go:122] "RemoveContainer" containerID="03d1ab4f864f360874d6d817a279ed991ceb3163ebbb85860968f4c421cfb89f" Oct 29 23:30:17.461649 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:30:17.461626 5057 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-063b46-29888610-pbkwf_odh-ai-gateway-infra(1d9a1f4c-5134-4b26-8292-bf3d588ba960)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-063b46-29888610-pbkwf" podUID="1d9a1f4c-5134-4b26-8292-bf3d588ba960" Oct 29 23:30:17.463078 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:17.463055 5057 generic.go:356] "Generic (PLEG): container finished" podID="f4d887af-8034-47ad-834c-c0097b7b48a6" containerID="db1bf4b89096aff86d229c5bddff5af3b6d9c24aead23efa1459fba2c80d5305" exitCode=60 Oct 29 23:30:17.463193 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:17.463083 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-67363b-29888610-mfd4n" event={"ID":"f4d887af-8034-47ad-834c-c0097b7b48a6","Type":"ContainerDied","Data":"db1bf4b89096aff86d229c5bddff5af3b6d9c24aead23efa1459fba2c80d5305"} Oct 29 23:30:17.463516 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:17.463497 5057 scope.go:122] "RemoveContainer" containerID="db1bf4b89096aff86d229c5bddff5af3b6d9c24aead23efa1459fba2c80d5305" Oct 29 23:30:17.463850 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:30:17.463822 5057 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-67363b-29888610-mfd4n_odh-ai-gateway-infra(f4d887af-8034-47ad-834c-c0097b7b48a6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-67363b-29888610-mfd4n" podUID="f4d887af-8034-47ad-834c-c0097b7b48a6" Oct 29 23:30:17.482096 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:17.482077 5057 scope.go:122] "RemoveContainer" containerID="95f8da04f1bbf5848cf995e21621de78160f75c87cab126f85dcd962680dfc43" Oct 29 23:30:17.496501 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:17.496468 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-063b46-29888610-pbkwf"] Oct 29 23:30:17.509071 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:17.509048 5057 scope.go:122] "RemoveContainer" containerID="d6f61d474b79c513feed9c5ce9d31ed506fe4548b3bd7c3481f92284d6a4bc61" Oct 29 23:30:17.569766 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:17.569694 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-67363b-29888610-mfd4n"] Oct 29 23:30:17.871683 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:17.871601 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-lhtzn" Oct 29 23:30:17.872114 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:17.872099 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-lhtzn" Oct 29 23:30:17.872806 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:17.872786 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-lhtzn" Oct 29 23:30:17.950118 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:17.950080 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w0-4504f1/sub-test-model-4504f1-kserve-657f694cf7-lqr5k" Oct 29 23:30:17.969091 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:17.968626 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w0-4504f1/sub-test-model-4504f1-kserve-657f694cf7-lqr5k" Oct 29 23:30:18.483845 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:18.483805 5057 generic.go:356] "Generic (PLEG): container finished" podID="73ffc0bf-5804-444b-b081-9f68d4168890" containerID="e26d658cde5a5245883e08d66bd3856e621ce4a769df04c3be2cf319aeed094e" exitCode=60 Oct 29 23:30:18.483975 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:18.483866 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e131af-29888610-mp6s5" event={"ID":"73ffc0bf-5804-444b-b081-9f68d4168890","Type":"ContainerDied","Data":"e26d658cde5a5245883e08d66bd3856e621ce4a769df04c3be2cf319aeed094e"} Oct 29 23:30:18.483975 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:18.483950 5057 scope.go:122] "RemoveContainer" containerID="5a231fa572e9853973ba688a094065ed409147dcc90dd9062ed8829ebe65a3a4" Oct 29 23:30:18.484523 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:18.484498 5057 scope.go:122] "RemoveContainer" containerID="e26d658cde5a5245883e08d66bd3856e621ce4a769df04c3be2cf319aeed094e" Oct 29 23:30:18.484996 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:30:18.484962 5057 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-e131af-29888610-mp6s5_odh-ai-gateway-infra(73ffc0bf-5804-444b-b081-9f68d4168890)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e131af-29888610-mp6s5" podUID="73ffc0bf-5804-444b-b081-9f68d4168890" Oct 29 23:30:18.521406 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:18.519176 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w0-4504f1/sub-test-model-4504f1-kserve-657f694cf7-lqr5k" Oct 29 23:30:18.525354 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:18.524977 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-af237d-29888610-6hd66"] Oct 29 23:30:18.757494 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:18.757457 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:30:18.836056 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:18.836026 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-063b46-29888610-pbkwf" Oct 29 23:30:18.839874 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:18.839849 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-67363b-29888610-mfd4n" Oct 29 23:30:18.851276 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:18.851253 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-af237d-29888610-6hd66" Oct 29 23:30:18.922861 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:18.922758 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4d887af-8034-47ad-834c-c0097b7b48a6-kube-api-access-vsl9k\" (UniqueName: \"kubernetes.io/projected/f4d887af-8034-47ad-834c-c0097b7b48a6-kube-api-access-vsl9k\") pod \"f4d887af-8034-47ad-834c-c0097b7b48a6\" (UID: \"f4d887af-8034-47ad-834c-c0097b7b48a6\") " Oct 29 23:30:18.924063 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:18.922947 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d9a1f4c-5134-4b26-8292-bf3d588ba960-kube-api-access-dwmpr\" (UniqueName: \"kubernetes.io/projected/1d9a1f4c-5134-4b26-8292-bf3d588ba960-kube-api-access-dwmpr\") pod \"1d9a1f4c-5134-4b26-8292-bf3d588ba960\" (UID: \"1d9a1f4c-5134-4b26-8292-bf3d588ba960\") " Oct 29 23:30:18.924063 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:18.923079 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d9a1f4c-5134-4b26-8292-bf3d588ba960-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d9a1f4c-5134-4b26-8292-bf3d588ba960-maas-api-service-ca\") pod \"1d9a1f4c-5134-4b26-8292-bf3d588ba960\" (UID: \"1d9a1f4c-5134-4b26-8292-bf3d588ba960\") " Oct 29 23:30:18.924063 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:18.923127 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4d887af-8034-47ad-834c-c0097b7b48a6-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4d887af-8034-47ad-834c-c0097b7b48a6-maas-api-service-ca\") pod \"f4d887af-8034-47ad-834c-c0097b7b48a6\" (UID: \"f4d887af-8034-47ad-834c-c0097b7b48a6\") " Oct 29 23:30:18.924702 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:18.924656 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4d887af-8034-47ad-834c-c0097b7b48a6-maas-api-service-ca" pod "f4d887af-8034-47ad-834c-c0097b7b48a6" (UID: "f4d887af-8034-47ad-834c-c0097b7b48a6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:30:18.925615 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:18.925592 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d9a1f4c-5134-4b26-8292-bf3d588ba960-maas-api-service-ca" pod "1d9a1f4c-5134-4b26-8292-bf3d588ba960" (UID: "1d9a1f4c-5134-4b26-8292-bf3d588ba960"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:30:18.927597 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:18.927569 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4d887af-8034-47ad-834c-c0097b7b48a6-kube-api-access-vsl9k" pod "f4d887af-8034-47ad-834c-c0097b7b48a6" (UID: "f4d887af-8034-47ad-834c-c0097b7b48a6"). InnerVolumeSpecName "kube-api-access-vsl9k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:30:18.929101 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:18.929074 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d9a1f4c-5134-4b26-8292-bf3d588ba960-kube-api-access-dwmpr" pod "1d9a1f4c-5134-4b26-8292-bf3d588ba960" (UID: "1d9a1f4c-5134-4b26-8292-bf3d588ba960"). InnerVolumeSpecName "kube-api-access-dwmpr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:30:19.025007 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:19.024977 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4caf8e1b-81e4-4cd0-a870-fe55dccdbe6d-kube-api-access-xdnps\" (UniqueName: \"kubernetes.io/projected/4caf8e1b-81e4-4cd0-a870-fe55dccdbe6d-kube-api-access-xdnps\") pod \"4caf8e1b-81e4-4cd0-a870-fe55dccdbe6d\" (UID: \"4caf8e1b-81e4-4cd0-a870-fe55dccdbe6d\") " Oct 29 23:30:19.025189 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:19.025100 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4caf8e1b-81e4-4cd0-a870-fe55dccdbe6d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4caf8e1b-81e4-4cd0-a870-fe55dccdbe6d-maas-api-service-ca\") pod \"4caf8e1b-81e4-4cd0-a870-fe55dccdbe6d\" (UID: \"4caf8e1b-81e4-4cd0-a870-fe55dccdbe6d\") " Oct 29 23:30:19.025648 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:19.025618 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4caf8e1b-81e4-4cd0-a870-fe55dccdbe6d-maas-api-service-ca" pod "4caf8e1b-81e4-4cd0-a870-fe55dccdbe6d" (UID: "4caf8e1b-81e4-4cd0-a870-fe55dccdbe6d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:30:19.025751 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:19.025742 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vsl9k\" (UniqueName: \"kubernetes.io/projected/f4d887af-8034-47ad-834c-c0097b7b48a6-kube-api-access-vsl9k\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:30:19.025791 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:19.025754 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dwmpr\" (UniqueName: \"kubernetes.io/projected/1d9a1f4c-5134-4b26-8292-bf3d588ba960-kube-api-access-dwmpr\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:30:19.025791 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:19.025764 5057 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4caf8e1b-81e4-4cd0-a870-fe55dccdbe6d-maas-api-service-ca\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:30:19.025791 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:19.025773 5057 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d9a1f4c-5134-4b26-8292-bf3d588ba960-maas-api-service-ca\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:30:19.025791 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:19.025781 5057 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4d887af-8034-47ad-834c-c0097b7b48a6-maas-api-service-ca\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:30:19.027448 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:19.027419 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4caf8e1b-81e4-4cd0-a870-fe55dccdbe6d-kube-api-access-xdnps" pod "4caf8e1b-81e4-4cd0-a870-fe55dccdbe6d" (UID: "4caf8e1b-81e4-4cd0-a870-fe55dccdbe6d"). InnerVolumeSpecName "kube-api-access-xdnps". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:30:19.031473 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:30:19.031444 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d2dnn" containerName="extractor" Oct 29 23:30:19.127637 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:19.127606 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xdnps\" (UniqueName: \"kubernetes.io/projected/4caf8e1b-81e4-4cd0-a870-fe55dccdbe6d-kube-api-access-xdnps\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:30:19.248534 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:19.248498 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-b0c3c3/rate-test-model-b0c3c3-kserve-5b45fbb77-6jzx9"] Oct 29 23:30:19.284890 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:19.284860 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:30:19.498327 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:19.498296 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-67363b-29888610-mfd4n" Oct 29 23:30:19.498327 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:19.498344 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-67363b-29888610-mfd4n" event={"ID":"f4d887af-8034-47ad-834c-c0097b7b48a6","Type":"ContainerDied","Data":"3bf95f70309bfb3e6bc8e76ab110949a814eedb1ff920c24589a395fafe44c9e"} Oct 29 23:30:19.498619 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:19.498406 5057 scope.go:122] "RemoveContainer" containerID="db1bf4b89096aff86d229c5bddff5af3b6d9c24aead23efa1459fba2c80d5305" Oct 29 23:30:19.505251 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:19.504750 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-af237d-29888610-6hd66" event={"ID":"4caf8e1b-81e4-4cd0-a870-fe55dccdbe6d","Type":"ContainerDied","Data":"b427c1f5f08c7fbbe7295e75332f5384aa25bb375584e47e93feff75b7915b78"} Oct 29 23:30:19.506874 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:19.506423 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-063b46-29888610-pbkwf" event={"ID":"1d9a1f4c-5134-4b26-8292-bf3d588ba960","Type":"ContainerDied","Data":"65ee8ca1a47bd4150387f2efe5c66667a22b3b7f1a5d441c71939eb0259f694d"} Oct 29 23:30:19.506874 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:19.504779 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-af237d-29888610-6hd66" Oct 29 23:30:19.507650 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:19.507200 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-063b46-29888610-pbkwf" Oct 29 23:30:19.515482 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:19.515445 5057 scope.go:122] "RemoveContainer" containerID="351be18bcc30788de478f458c3b739fb9aa9d6204d1395db0488161e4911ee92" Oct 29 23:30:19.530427 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:19.530402 5057 scope.go:122] "RemoveContainer" containerID="03d1ab4f864f360874d6d817a279ed991ceb3163ebbb85860968f4c421cfb89f" Oct 29 23:30:19.569410 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:19.569380 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-af237d-29888610-6hd66"] Oct 29 23:30:19.574479 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:19.574453 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-af237d-29888610-6hd66"] Oct 29 23:30:19.610807 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:19.610774 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-67363b-29888610-mfd4n"] Oct 29 23:30:19.614960 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:19.614930 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-67363b-29888610-mfd4n"] Oct 29 23:30:19.635766 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:19.635739 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e131af-29888610-mp6s5"] Oct 29 23:30:19.649748 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:19.649254 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-063b46-29888610-pbkwf"] Oct 29 23:30:19.658462 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:19.658421 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-063b46-29888610-pbkwf"] Oct 29 23:30:19.858798 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:19.858770 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e131af-29888610-mp6s5" Oct 29 23:30:19.872090 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:19.872057 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73ffc0bf-5804-444b-b081-9f68d4168890-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/73ffc0bf-5804-444b-b081-9f68d4168890-maas-api-service-ca\") pod \"73ffc0bf-5804-444b-b081-9f68d4168890\" (UID: \"73ffc0bf-5804-444b-b081-9f68d4168890\") " Oct 29 23:30:19.872261 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:19.872214 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73ffc0bf-5804-444b-b081-9f68d4168890-kube-api-access-dkv99\" (UniqueName: \"kubernetes.io/projected/73ffc0bf-5804-444b-b081-9f68d4168890-kube-api-access-dkv99\") pod \"73ffc0bf-5804-444b-b081-9f68d4168890\" (UID: \"73ffc0bf-5804-444b-b081-9f68d4168890\") " Oct 29 23:30:19.872456 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:19.872434 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73ffc0bf-5804-444b-b081-9f68d4168890-maas-api-service-ca" pod "73ffc0bf-5804-444b-b081-9f68d4168890" (UID: "73ffc0bf-5804-444b-b081-9f68d4168890"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:30:19.872917 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:19.872892 5057 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/73ffc0bf-5804-444b-b081-9f68d4168890-maas-api-service-ca\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:30:19.874751 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:19.874727 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73ffc0bf-5804-444b-b081-9f68d4168890-kube-api-access-dkv99" pod "73ffc0bf-5804-444b-b081-9f68d4168890" (UID: "73ffc0bf-5804-444b-b081-9f68d4168890"). InnerVolumeSpecName "kube-api-access-dkv99". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:30:19.974058 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:19.974017 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dkv99\" (UniqueName: \"kubernetes.io/projected/73ffc0bf-5804-444b-b081-9f68d4168890-kube-api-access-dkv99\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:30:20.427153 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:20.427116 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:30:20.519884 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:20.519849 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e131af-29888610-mp6s5" Oct 29 23:30:20.520195 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:20.519866 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e131af-29888610-mp6s5" event={"ID":"73ffc0bf-5804-444b-b081-9f68d4168890","Type":"ContainerDied","Data":"313050aa892d698c139de3a625162a18ce6915713ae503bf487ff07591436c5d"} Oct 29 23:30:20.520300 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:20.520276 5057 scope.go:122] "RemoveContainer" containerID="e26d658cde5a5245883e08d66bd3856e621ce4a769df04c3be2cf319aeed094e" Oct 29 23:30:20.573656 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:20.573627 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e131af-29888610-mp6s5"] Oct 29 23:30:20.584889 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:20.584865 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e131af-29888610-mp6s5"] Oct 29 23:30:21.330761 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:21.330727 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d9a1f4c-5134-4b26-8292-bf3d588ba960" path="/var/lib/kubelet/pods/1d9a1f4c-5134-4b26-8292-bf3d588ba960/volumes" Oct 29 23:30:21.331400 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:21.331378 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4caf8e1b-81e4-4cd0-a870-fe55dccdbe6d" path="/var/lib/kubelet/pods/4caf8e1b-81e4-4cd0-a870-fe55dccdbe6d/volumes" Oct 29 23:30:21.331897 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:21.331877 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73ffc0bf-5804-444b-b081-9f68d4168890" path="/var/lib/kubelet/pods/73ffc0bf-5804-444b-b081-9f68d4168890/volumes" Oct 29 23:30:21.332640 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:21.332620 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4d887af-8034-47ad-834c-c0097b7b48a6" path="/var/lib/kubelet/pods/f4d887af-8034-47ad-834c-c0097b7b48a6/volumes" Oct 29 23:30:22.106550 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:22.106505 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:30:23.785153 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:23.785106 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:30:25.304435 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:25.304397 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:30:26.803585 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:26.803546 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:30:28.305988 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:28.305937 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:30:29.897216 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:29.897182 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:30:31.404106 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:31.404067 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:30:32.991628 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:32.991589 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:30:34.270385 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:34.270315 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w2-b0c3c3/rate-test-model-b0c3c3-kserve-5b45fbb77-6jzx9" podUID="84dd8e63-57ac-43db-9c6e-63725aa2e6a5" containerName="main" containerID="cri-o://b26ccc84f4f7356de821f8e20eb1b99344d31b4e48b3d350f083584ac3fa54fd" gracePeriod=45 Oct 29 23:30:34.499013 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:34.498973 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:30:34.571125 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:34.571090 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-b0c3c3/rate-test-model-b0c3c3-kserve-5b45fbb77-6jzx9" Oct 29 23:30:34.663656 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:34.663613 5057 generic.go:356] "Generic (PLEG): container finished" podID="84dd8e63-57ac-43db-9c6e-63725aa2e6a5" containerID="b26ccc84f4f7356de821f8e20eb1b99344d31b4e48b3d350f083584ac3fa54fd" exitCode=0 Oct 29 23:30:34.663823 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:34.663681 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-b0c3c3/rate-test-model-b0c3c3-kserve-5b45fbb77-6jzx9" Oct 29 23:30:34.663823 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:34.663707 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-b0c3c3/rate-test-model-b0c3c3-kserve-5b45fbb77-6jzx9" event={"ID":"84dd8e63-57ac-43db-9c6e-63725aa2e6a5","Type":"ContainerDied","Data":"b26ccc84f4f7356de821f8e20eb1b99344d31b4e48b3d350f083584ac3fa54fd"} Oct 29 23:30:34.663823 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:34.663759 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-b0c3c3/rate-test-model-b0c3c3-kserve-5b45fbb77-6jzx9" event={"ID":"84dd8e63-57ac-43db-9c6e-63725aa2e6a5","Type":"ContainerDied","Data":"ed042a51cd472ea613b240a70f1b32137b65950dbe694f3d8677881e6b0fbdd5"} Oct 29 23:30:34.663823 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:34.663784 5057 scope.go:122] "RemoveContainer" containerID="b26ccc84f4f7356de821f8e20eb1b99344d31b4e48b3d350f083584ac3fa54fd" Oct 29 23:30:34.673412 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:34.673389 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84dd8e63-57ac-43db-9c6e-63725aa2e6a5-dshm\" (UniqueName: \"kubernetes.io/empty-dir/84dd8e63-57ac-43db-9c6e-63725aa2e6a5-dshm\") pod \"84dd8e63-57ac-43db-9c6e-63725aa2e6a5\" (UID: \"84dd8e63-57ac-43db-9c6e-63725aa2e6a5\") " Oct 29 23:30:34.673548 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:34.673450 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84dd8e63-57ac-43db-9c6e-63725aa2e6a5-kube-api-access-gzw5m\" (UniqueName: \"kubernetes.io/projected/84dd8e63-57ac-43db-9c6e-63725aa2e6a5-kube-api-access-gzw5m\") pod \"84dd8e63-57ac-43db-9c6e-63725aa2e6a5\" (UID: \"84dd8e63-57ac-43db-9c6e-63725aa2e6a5\") " Oct 29 23:30:34.673548 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:34.673482 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84dd8e63-57ac-43db-9c6e-63725aa2e6a5-home\" (UniqueName: \"kubernetes.io/empty-dir/84dd8e63-57ac-43db-9c6e-63725aa2e6a5-home\") pod \"84dd8e63-57ac-43db-9c6e-63725aa2e6a5\" (UID: \"84dd8e63-57ac-43db-9c6e-63725aa2e6a5\") " Oct 29 23:30:34.673681 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:34.673551 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84dd8e63-57ac-43db-9c6e-63725aa2e6a5-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/84dd8e63-57ac-43db-9c6e-63725aa2e6a5-tmp-dir\") pod \"84dd8e63-57ac-43db-9c6e-63725aa2e6a5\" (UID: \"84dd8e63-57ac-43db-9c6e-63725aa2e6a5\") " Oct 29 23:30:34.673681 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:34.673589 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84dd8e63-57ac-43db-9c6e-63725aa2e6a5-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/84dd8e63-57ac-43db-9c6e-63725aa2e6a5-model-cache\") pod \"84dd8e63-57ac-43db-9c6e-63725aa2e6a5\" (UID: \"84dd8e63-57ac-43db-9c6e-63725aa2e6a5\") " Oct 29 23:30:34.673803 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:34.673743 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/84dd8e63-57ac-43db-9c6e-63725aa2e6a5-tls-certs\" (UniqueName: \"kubernetes.io/secret/84dd8e63-57ac-43db-9c6e-63725aa2e6a5-tls-certs\") pod \"84dd8e63-57ac-43db-9c6e-63725aa2e6a5\" (UID: \"84dd8e63-57ac-43db-9c6e-63725aa2e6a5\") " Oct 29 23:30:34.673877 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:34.673854 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84dd8e63-57ac-43db-9c6e-63725aa2e6a5-home" pod "84dd8e63-57ac-43db-9c6e-63725aa2e6a5" (UID: "84dd8e63-57ac-43db-9c6e-63725aa2e6a5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:30:34.673947 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:34.673906 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84dd8e63-57ac-43db-9c6e-63725aa2e6a5-model-cache" pod "84dd8e63-57ac-43db-9c6e-63725aa2e6a5" (UID: "84dd8e63-57ac-43db-9c6e-63725aa2e6a5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:30:34.674118 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:34.674097 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84dd8e63-57ac-43db-9c6e-63725aa2e6a5-tmp-dir" pod "84dd8e63-57ac-43db-9c6e-63725aa2e6a5" (UID: "84dd8e63-57ac-43db-9c6e-63725aa2e6a5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:30:34.674554 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:34.674525 5057 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/84dd8e63-57ac-43db-9c6e-63725aa2e6a5-tmp-dir\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:30:34.674554 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:34.674551 5057 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/84dd8e63-57ac-43db-9c6e-63725aa2e6a5-model-cache\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:30:34.674759 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:34.674565 5057 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/84dd8e63-57ac-43db-9c6e-63725aa2e6a5-home\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:30:34.675773 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:34.675753 5057 scope.go:122] "RemoveContainer" containerID="b26ccc84f4f7356de821f8e20eb1b99344d31b4e48b3d350f083584ac3fa54fd" Oct 29 23:30:34.675972 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:34.675907 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84dd8e63-57ac-43db-9c6e-63725aa2e6a5-kube-api-access-gzw5m" pod "84dd8e63-57ac-43db-9c6e-63725aa2e6a5" (UID: "84dd8e63-57ac-43db-9c6e-63725aa2e6a5"). InnerVolumeSpecName "kube-api-access-gzw5m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:30:34.675972 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:34.675942 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84dd8e63-57ac-43db-9c6e-63725aa2e6a5-tls-certs" pod "84dd8e63-57ac-43db-9c6e-63725aa2e6a5" (UID: "84dd8e63-57ac-43db-9c6e-63725aa2e6a5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:30:34.676154 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:34.676137 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84dd8e63-57ac-43db-9c6e-63725aa2e6a5-dshm" pod "84dd8e63-57ac-43db-9c6e-63725aa2e6a5" (UID: "84dd8e63-57ac-43db-9c6e-63725aa2e6a5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:30:34.676198 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:30:34.676153 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b26ccc84f4f7356de821f8e20eb1b99344d31b4e48b3d350f083584ac3fa54fd\": container with ID starting with b26ccc84f4f7356de821f8e20eb1b99344d31b4e48b3d350f083584ac3fa54fd not found: ID does not exist" containerID="b26ccc84f4f7356de821f8e20eb1b99344d31b4e48b3d350f083584ac3fa54fd" Oct 29 23:30:34.676235 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:34.676189 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b26ccc84f4f7356de821f8e20eb1b99344d31b4e48b3d350f083584ac3fa54fd"} err="failed to get container status \"b26ccc84f4f7356de821f8e20eb1b99344d31b4e48b3d350f083584ac3fa54fd\": rpc error: code = NotFound desc = could not find container \"b26ccc84f4f7356de821f8e20eb1b99344d31b4e48b3d350f083584ac3fa54fd\": container with ID starting with b26ccc84f4f7356de821f8e20eb1b99344d31b4e48b3d350f083584ac3fa54fd not found: ID does not exist" Oct 29 23:30:34.776312 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:34.776277 5057 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/84dd8e63-57ac-43db-9c6e-63725aa2e6a5-tls-certs\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:30:34.776312 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:34.776308 5057 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/84dd8e63-57ac-43db-9c6e-63725aa2e6a5-dshm\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:30:34.776312 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:34.776318 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gzw5m\" (UniqueName: \"kubernetes.io/projected/84dd8e63-57ac-43db-9c6e-63725aa2e6a5-kube-api-access-gzw5m\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:30:35.014070 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:35.014033 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-b0c3c3/rate-test-model-b0c3c3-kserve-5b45fbb77-6jzx9"] Oct 29 23:30:35.024861 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:35.024830 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w2-b0c3c3/rate-test-model-b0c3c3-kserve-5b45fbb77-6jzx9"] Oct 29 23:30:35.331031 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:35.330948 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84dd8e63-57ac-43db-9c6e-63725aa2e6a5" path="/var/lib/kubelet/pods/84dd8e63-57ac-43db-9c6e-63725aa2e6a5/volumes" Oct 29 23:30:36.096975 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:36.096936 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:30:36.642373 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:36.642313 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:30:37.588347 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:37.588282 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:30:39.089488 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:39.089447 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:30:40.242814 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:40.242777 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-ed3bf2/rate-test-model-ed3bf2-kserve-7bbcbb9dd9-xpmj5"] Oct 29 23:30:40.274141 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:40.274105 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:30:40.523203 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:40.522462 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b0c3c3-89fkw"] Oct 29 23:30:40.523667 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:40.523628 5057 cpu_manager.go:414] "containerMap: removing container" podUID="73ffc0bf-5804-444b-b081-9f68d4168890" containerName="cleanup" Oct 29 23:30:40.523667 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:40.523665 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73ffc0bf-5804-444b-b081-9f68d4168890" containerName="cleanup" Oct 29 23:30:40.523846 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:40.523684 5057 cpu_manager.go:414] "containerMap: removing container" podUID="84dd8e63-57ac-43db-9c6e-63725aa2e6a5" containerName="main" Oct 29 23:30:40.523846 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:40.523692 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84dd8e63-57ac-43db-9c6e-63725aa2e6a5" containerName="main" Oct 29 23:30:40.523846 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:40.523710 5057 cpu_manager.go:414] "containerMap: removing container" podUID="f4d887af-8034-47ad-834c-c0097b7b48a6" containerName="cleanup" Oct 29 23:30:40.523846 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:40.523718 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4d887af-8034-47ad-834c-c0097b7b48a6" containerName="cleanup" Oct 29 23:30:40.523846 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:40.523748 5057 cpu_manager.go:414] "containerMap: removing container" podUID="f4d887af-8034-47ad-834c-c0097b7b48a6" containerName="cleanup" Oct 29 23:30:40.523846 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:40.523756 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4d887af-8034-47ad-834c-c0097b7b48a6" containerName="cleanup" Oct 29 23:30:40.523846 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:40.523778 5057 cpu_manager.go:414] "containerMap: removing container" podUID="1d9a1f4c-5134-4b26-8292-bf3d588ba960" containerName="cleanup" Oct 29 23:30:40.523846 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:40.523786 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d9a1f4c-5134-4b26-8292-bf3d588ba960" containerName="cleanup" Oct 29 23:30:40.523846 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:40.523797 5057 cpu_manager.go:414] "containerMap: removing container" podUID="f4d887af-8034-47ad-834c-c0097b7b48a6" containerName="cleanup" Oct 29 23:30:40.523846 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:40.523804 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4d887af-8034-47ad-834c-c0097b7b48a6" containerName="cleanup" Oct 29 23:30:40.523846 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:40.523826 5057 cpu_manager.go:414] "containerMap: removing container" podUID="1d9a1f4c-5134-4b26-8292-bf3d588ba960" containerName="cleanup" Oct 29 23:30:40.523846 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:40.523834 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d9a1f4c-5134-4b26-8292-bf3d588ba960" containerName="cleanup" Oct 29 23:30:40.524282 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:40.523853 5057 cpu_manager.go:414] "containerMap: removing container" podUID="73ffc0bf-5804-444b-b081-9f68d4168890" containerName="cleanup" Oct 29 23:30:40.524282 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:40.523861 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73ffc0bf-5804-444b-b081-9f68d4168890" containerName="cleanup" Oct 29 23:30:40.524282 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:40.523879 5057 cpu_manager.go:414] "containerMap: removing container" podUID="1d9a1f4c-5134-4b26-8292-bf3d588ba960" containerName="cleanup" Oct 29 23:30:40.524282 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:40.523886 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d9a1f4c-5134-4b26-8292-bf3d588ba960" containerName="cleanup" Oct 29 23:30:40.524282 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:40.523898 5057 cpu_manager.go:414] "containerMap: removing container" podUID="4caf8e1b-81e4-4cd0-a870-fe55dccdbe6d" containerName="cleanup" Oct 29 23:30:40.524282 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:40.523905 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4caf8e1b-81e4-4cd0-a870-fe55dccdbe6d" containerName="cleanup" Oct 29 23:30:40.524282 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:40.523923 5057 cpu_manager.go:414] "containerMap: removing container" podUID="4caf8e1b-81e4-4cd0-a870-fe55dccdbe6d" containerName="cleanup" Oct 29 23:30:40.524282 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:40.523931 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4caf8e1b-81e4-4cd0-a870-fe55dccdbe6d" containerName="cleanup" Oct 29 23:30:40.524282 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:40.523944 5057 cpu_manager.go:414] "containerMap: removing container" podUID="4caf8e1b-81e4-4cd0-a870-fe55dccdbe6d" containerName="cleanup" Oct 29 23:30:40.524282 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:40.523951 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4caf8e1b-81e4-4cd0-a870-fe55dccdbe6d" containerName="cleanup" Oct 29 23:30:40.524282 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:40.523962 5057 cpu_manager.go:414] "containerMap: removing container" podUID="73ffc0bf-5804-444b-b081-9f68d4168890" containerName="cleanup" Oct 29 23:30:40.524282 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:40.523969 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73ffc0bf-5804-444b-b081-9f68d4168890" containerName="cleanup" Oct 29 23:30:40.524282 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:40.524148 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="4caf8e1b-81e4-4cd0-a870-fe55dccdbe6d" containerName="cleanup" Oct 29 23:30:40.524282 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:40.524161 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="73ffc0bf-5804-444b-b081-9f68d4168890" containerName="cleanup" Oct 29 23:30:40.524282 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:40.524178 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="84dd8e63-57ac-43db-9c6e-63725aa2e6a5" containerName="main" Oct 29 23:30:40.524282 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:40.524188 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4d887af-8034-47ad-834c-c0097b7b48a6" containerName="cleanup" Oct 29 23:30:40.524282 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:40.524201 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4d887af-8034-47ad-834c-c0097b7b48a6" containerName="cleanup" Oct 29 23:30:40.524282 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:40.524212 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d9a1f4c-5134-4b26-8292-bf3d588ba960" containerName="cleanup" Oct 29 23:30:40.524282 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:40.524225 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="73ffc0bf-5804-444b-b081-9f68d4168890" containerName="cleanup" Oct 29 23:30:40.524282 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:40.524236 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d9a1f4c-5134-4b26-8292-bf3d588ba960" containerName="cleanup" Oct 29 23:30:40.524282 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:40.524246 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="4caf8e1b-81e4-4cd0-a870-fe55dccdbe6d" containerName="cleanup" Oct 29 23:30:40.529725 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:40.529703 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b0c3c3-89fkw" Oct 29 23:30:40.545996 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:40.545868 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dmx57\"" Oct 29 23:30:40.545996 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:40.545874 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:30:40.547012 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:40.546990 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b0c3c3-89fkw"] Oct 29 23:30:40.668735 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:40.668690 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6880653f-fa32-4858-a99b-69ee0170dceb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-b0c3c3-89fkw\" (UID: \"6880653f-fa32-4858-a99b-69ee0170dceb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b0c3c3-89fkw" Oct 29 23:30:40.668954 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:40.668803 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fn7kf\" (UniqueName: \"kubernetes.io/projected/6880653f-fa32-4858-a99b-69ee0170dceb-kube-api-access-fn7kf\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-b0c3c3-89fkw\" (UID: \"6880653f-fa32-4858-a99b-69ee0170dceb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b0c3c3-89fkw" Oct 29 23:30:40.668954 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:40.668933 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6880653f-fa32-4858-a99b-69ee0170dceb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-b0c3c3-89fkw\" (UID: \"6880653f-fa32-4858-a99b-69ee0170dceb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b0c3c3-89fkw" Oct 29 23:30:40.770071 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:40.770030 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fn7kf\" (UniqueName: \"kubernetes.io/projected/6880653f-fa32-4858-a99b-69ee0170dceb-kube-api-access-fn7kf\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-b0c3c3-89fkw\" (UID: \"6880653f-fa32-4858-a99b-69ee0170dceb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b0c3c3-89fkw" Oct 29 23:30:40.770951 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:40.770913 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6880653f-fa32-4858-a99b-69ee0170dceb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-b0c3c3-89fkw\" (UID: \"6880653f-fa32-4858-a99b-69ee0170dceb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b0c3c3-89fkw" Oct 29 23:30:40.771296 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:40.771278 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6880653f-fa32-4858-a99b-69ee0170dceb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-b0c3c3-89fkw\" (UID: \"6880653f-fa32-4858-a99b-69ee0170dceb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b0c3c3-89fkw" Oct 29 23:30:40.772533 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:40.772511 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6880653f-fa32-4858-a99b-69ee0170dceb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-b0c3c3-89fkw\" (UID: \"6880653f-fa32-4858-a99b-69ee0170dceb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b0c3c3-89fkw" Oct 29 23:30:40.773985 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:40.773937 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6880653f-fa32-4858-a99b-69ee0170dceb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-b0c3c3-89fkw\" (UID: \"6880653f-fa32-4858-a99b-69ee0170dceb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b0c3c3-89fkw" Oct 29 23:30:40.797381 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:40.794977 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fn7kf\" (UniqueName: \"kubernetes.io/projected/6880653f-fa32-4858-a99b-69ee0170dceb-kube-api-access-fn7kf\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-b0c3c3-89fkw\" (UID: \"6880653f-fa32-4858-a99b-69ee0170dceb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b0c3c3-89fkw" Oct 29 23:30:40.845828 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:40.845791 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b0c3c3-89fkw" Oct 29 23:30:40.928696 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:40.928660 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:30:41.180227 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:41.180196 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b0c3c3-89fkw"] Oct 29 23:30:41.183008 ip-10-0-140-74 kubenswrapper[5057]: W1029 23:30:41.182983 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6880653f_fa32_4858_a99b_69ee0170dceb.slice/crio-303a395be470c7097ab8c568b31e125af98ef951629451f766e620250ee4343d WatchSource:0}: Error finding container 303a395be470c7097ab8c568b31e125af98ef951629451f766e620250ee4343d: Status 404 returned error can't find the container with id 303a395be470c7097ab8c568b31e125af98ef951629451f766e620250ee4343d Oct 29 23:30:41.739081 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:41.738986 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b0c3c3-89fkw" event={"ID":"6880653f-fa32-4858-a99b-69ee0170dceb","Type":"ContainerStarted","Data":"ac146beed95924162eec1f0550f85e24f383b09cda833091eb02247192fbf341"} Oct 29 23:30:41.739081 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:41.739041 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b0c3c3-89fkw" event={"ID":"6880653f-fa32-4858-a99b-69ee0170dceb","Type":"ContainerStarted","Data":"303a395be470c7097ab8c568b31e125af98ef951629451f766e620250ee4343d"} Oct 29 23:30:41.781529 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:41.781476 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b0c3c3-89fkw" podStartSLOduration=1.781456436 podStartE2EDuration="1.781456436s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:30:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:30:41.779342611 +0000 UTC m=+3422.562556057" watchObservedRunningTime="2026-10-29 23:30:41.781456436 +0000 UTC m=+3422.564669864" Oct 29 23:30:41.871521 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:41.871489 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w0-67363b/sub-test-model-67363b-kserve-54bb84cd5f-dpl7n"] Oct 29 23:30:41.872423 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:41.872405 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d9a1f4c-5134-4b26-8292-bf3d588ba960" containerName="cleanup" Oct 29 23:30:41.872503 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:41.872430 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="4caf8e1b-81e4-4cd0-a870-fe55dccdbe6d" containerName="cleanup" Oct 29 23:30:41.872503 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:41.872438 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="73ffc0bf-5804-444b-b081-9f68d4168890" containerName="cleanup" Oct 29 23:30:41.872503 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:41.872452 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4d887af-8034-47ad-834c-c0097b7b48a6" containerName="cleanup" Oct 29 23:30:41.876487 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:41.876427 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-67363b/sub-test-model-67363b-kserve-54bb84cd5f-dpl7n" Oct 29 23:30:41.902581 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:41.898439 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-67363b/sub-test-model-67363b-kserve-54bb84cd5f-dpl7n"] Oct 29 23:30:41.902581 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:41.898744 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-67363b\"/\"kube-root-ca.crt\"" Oct 29 23:30:41.902581 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:41.898936 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-67363b\"/\"sub-test-model-67363b-kserve-self-signed-certs\"" Oct 29 23:30:41.902581 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:41.899597 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-67363b\"/\"default-dockercfg-2wwt5\"" Oct 29 23:30:41.902581 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:41.900097 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-67363b\"/\"openshift-service-ca.crt\"" Oct 29 23:30:41.902581 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:41.901025 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-glrcr\" (UniqueName: \"kubernetes.io/projected/63e3a68d-90a1-4e09-8967-a1fb5d99b87a-kube-api-access-glrcr\") pod \"sub-test-model-67363b-kserve-54bb84cd5f-dpl7n\" (UID: \"63e3a68d-90a1-4e09-8967-a1fb5d99b87a\") " pod="ai-tenant-e2e-shared-b-w0-67363b/sub-test-model-67363b-kserve-54bb84cd5f-dpl7n" Oct 29 23:30:41.902581 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:41.901220 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/63e3a68d-90a1-4e09-8967-a1fb5d99b87a-dshm\") pod \"sub-test-model-67363b-kserve-54bb84cd5f-dpl7n\" (UID: \"63e3a68d-90a1-4e09-8967-a1fb5d99b87a\") " pod="ai-tenant-e2e-shared-b-w0-67363b/sub-test-model-67363b-kserve-54bb84cd5f-dpl7n" Oct 29 23:30:41.902581 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:41.901386 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/63e3a68d-90a1-4e09-8967-a1fb5d99b87a-home\") pod \"sub-test-model-67363b-kserve-54bb84cd5f-dpl7n\" (UID: \"63e3a68d-90a1-4e09-8967-a1fb5d99b87a\") " pod="ai-tenant-e2e-shared-b-w0-67363b/sub-test-model-67363b-kserve-54bb84cd5f-dpl7n" Oct 29 23:30:41.902581 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:41.901417 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/63e3a68d-90a1-4e09-8967-a1fb5d99b87a-tls-certs\") pod \"sub-test-model-67363b-kserve-54bb84cd5f-dpl7n\" (UID: \"63e3a68d-90a1-4e09-8967-a1fb5d99b87a\") " pod="ai-tenant-e2e-shared-b-w0-67363b/sub-test-model-67363b-kserve-54bb84cd5f-dpl7n" Oct 29 23:30:41.902581 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:41.901449 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/63e3a68d-90a1-4e09-8967-a1fb5d99b87a-tmp-dir\") pod \"sub-test-model-67363b-kserve-54bb84cd5f-dpl7n\" (UID: \"63e3a68d-90a1-4e09-8967-a1fb5d99b87a\") " pod="ai-tenant-e2e-shared-b-w0-67363b/sub-test-model-67363b-kserve-54bb84cd5f-dpl7n" Oct 29 23:30:41.902581 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:41.901522 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/63e3a68d-90a1-4e09-8967-a1fb5d99b87a-model-cache\") pod \"sub-test-model-67363b-kserve-54bb84cd5f-dpl7n\" (UID: \"63e3a68d-90a1-4e09-8967-a1fb5d99b87a\") " pod="ai-tenant-e2e-shared-b-w0-67363b/sub-test-model-67363b-kserve-54bb84cd5f-dpl7n" Oct 29 23:30:41.930417 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:41.930368 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:30:42.003157 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:42.003068 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-glrcr\" (UniqueName: \"kubernetes.io/projected/63e3a68d-90a1-4e09-8967-a1fb5d99b87a-kube-api-access-glrcr\") pod \"sub-test-model-67363b-kserve-54bb84cd5f-dpl7n\" (UID: \"63e3a68d-90a1-4e09-8967-a1fb5d99b87a\") " pod="ai-tenant-e2e-shared-b-w0-67363b/sub-test-model-67363b-kserve-54bb84cd5f-dpl7n" Oct 29 23:30:42.003350 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:42.003168 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/63e3a68d-90a1-4e09-8967-a1fb5d99b87a-dshm\") pod \"sub-test-model-67363b-kserve-54bb84cd5f-dpl7n\" (UID: \"63e3a68d-90a1-4e09-8967-a1fb5d99b87a\") " pod="ai-tenant-e2e-shared-b-w0-67363b/sub-test-model-67363b-kserve-54bb84cd5f-dpl7n" Oct 29 23:30:42.003350 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:42.003294 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/63e3a68d-90a1-4e09-8967-a1fb5d99b87a-home\") pod \"sub-test-model-67363b-kserve-54bb84cd5f-dpl7n\" (UID: \"63e3a68d-90a1-4e09-8967-a1fb5d99b87a\") " pod="ai-tenant-e2e-shared-b-w0-67363b/sub-test-model-67363b-kserve-54bb84cd5f-dpl7n" Oct 29 23:30:42.003350 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:42.003326 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/63e3a68d-90a1-4e09-8967-a1fb5d99b87a-tls-certs\") pod \"sub-test-model-67363b-kserve-54bb84cd5f-dpl7n\" (UID: \"63e3a68d-90a1-4e09-8967-a1fb5d99b87a\") " pod="ai-tenant-e2e-shared-b-w0-67363b/sub-test-model-67363b-kserve-54bb84cd5f-dpl7n" Oct 29 23:30:42.003530 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:42.003369 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/63e3a68d-90a1-4e09-8967-a1fb5d99b87a-tmp-dir\") pod \"sub-test-model-67363b-kserve-54bb84cd5f-dpl7n\" (UID: \"63e3a68d-90a1-4e09-8967-a1fb5d99b87a\") " pod="ai-tenant-e2e-shared-b-w0-67363b/sub-test-model-67363b-kserve-54bb84cd5f-dpl7n" Oct 29 23:30:42.003530 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:42.003441 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/63e3a68d-90a1-4e09-8967-a1fb5d99b87a-model-cache\") pod \"sub-test-model-67363b-kserve-54bb84cd5f-dpl7n\" (UID: \"63e3a68d-90a1-4e09-8967-a1fb5d99b87a\") " pod="ai-tenant-e2e-shared-b-w0-67363b/sub-test-model-67363b-kserve-54bb84cd5f-dpl7n" Oct 29 23:30:42.003744 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:42.003721 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/63e3a68d-90a1-4e09-8967-a1fb5d99b87a-home\") pod \"sub-test-model-67363b-kserve-54bb84cd5f-dpl7n\" (UID: \"63e3a68d-90a1-4e09-8967-a1fb5d99b87a\") " pod="ai-tenant-e2e-shared-b-w0-67363b/sub-test-model-67363b-kserve-54bb84cd5f-dpl7n" Oct 29 23:30:42.003820 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:42.003784 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/63e3a68d-90a1-4e09-8967-a1fb5d99b87a-model-cache\") pod \"sub-test-model-67363b-kserve-54bb84cd5f-dpl7n\" (UID: \"63e3a68d-90a1-4e09-8967-a1fb5d99b87a\") " pod="ai-tenant-e2e-shared-b-w0-67363b/sub-test-model-67363b-kserve-54bb84cd5f-dpl7n" Oct 29 23:30:42.003966 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:42.003948 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/63e3a68d-90a1-4e09-8967-a1fb5d99b87a-tmp-dir\") pod \"sub-test-model-67363b-kserve-54bb84cd5f-dpl7n\" (UID: \"63e3a68d-90a1-4e09-8967-a1fb5d99b87a\") " pod="ai-tenant-e2e-shared-b-w0-67363b/sub-test-model-67363b-kserve-54bb84cd5f-dpl7n" Oct 29 23:30:42.005898 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:42.005865 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/63e3a68d-90a1-4e09-8967-a1fb5d99b87a-dshm\") pod \"sub-test-model-67363b-kserve-54bb84cd5f-dpl7n\" (UID: \"63e3a68d-90a1-4e09-8967-a1fb5d99b87a\") " pod="ai-tenant-e2e-shared-b-w0-67363b/sub-test-model-67363b-kserve-54bb84cd5f-dpl7n" Oct 29 23:30:42.006044 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:42.005908 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/63e3a68d-90a1-4e09-8967-a1fb5d99b87a-tls-certs\") pod \"sub-test-model-67363b-kserve-54bb84cd5f-dpl7n\" (UID: \"63e3a68d-90a1-4e09-8967-a1fb5d99b87a\") " pod="ai-tenant-e2e-shared-b-w0-67363b/sub-test-model-67363b-kserve-54bb84cd5f-dpl7n" Oct 29 23:30:42.027091 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:42.027058 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-glrcr\" (UniqueName: \"kubernetes.io/projected/63e3a68d-90a1-4e09-8967-a1fb5d99b87a-kube-api-access-glrcr\") pod \"sub-test-model-67363b-kserve-54bb84cd5f-dpl7n\" (UID: \"63e3a68d-90a1-4e09-8967-a1fb5d99b87a\") " pod="ai-tenant-e2e-shared-b-w0-67363b/sub-test-model-67363b-kserve-54bb84cd5f-dpl7n" Oct 29 23:30:42.204202 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:42.204162 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-67363b/sub-test-model-67363b-kserve-54bb84cd5f-dpl7n" Oct 29 23:30:42.510602 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:42.510570 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-67363b/sub-test-model-67363b-kserve-54bb84cd5f-dpl7n"] Oct 29 23:30:42.510979 ip-10-0-140-74 kubenswrapper[5057]: W1029 23:30:42.510951 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod63e3a68d_90a1_4e09_8967_a1fb5d99b87a.slice/crio-28c10fb6a882fd395ce6c43e14fdbb405e93ca0f288257b1702e1e296ce552b3 WatchSource:0}: Error finding container 28c10fb6a882fd395ce6c43e14fdbb405e93ca0f288257b1702e1e296ce552b3: Status 404 returned error can't find the container with id 28c10fb6a882fd395ce6c43e14fdbb405e93ca0f288257b1702e1e296ce552b3 Oct 29 23:30:42.607857 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:42.607815 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:30:42.749729 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:42.749689 5057 generic.go:356] "Generic (PLEG): container finished" podID="6880653f-fa32-4858-a99b-69ee0170dceb" containerID="ac146beed95924162eec1f0550f85e24f383b09cda833091eb02247192fbf341" exitCode=0 Oct 29 23:30:42.750195 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:42.749781 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b0c3c3-89fkw" event={"ID":"6880653f-fa32-4858-a99b-69ee0170dceb","Type":"ContainerDied","Data":"ac146beed95924162eec1f0550f85e24f383b09cda833091eb02247192fbf341"} Oct 29 23:30:42.751972 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:42.751945 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-67363b/sub-test-model-67363b-kserve-54bb84cd5f-dpl7n" event={"ID":"63e3a68d-90a1-4e09-8967-a1fb5d99b87a","Type":"ContainerStarted","Data":"7a393b1e5357e8959f70b848e1b094f27bc7302c338e45443b05e5b5ad409157"} Oct 29 23:30:42.752113 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:42.751977 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-67363b/sub-test-model-67363b-kserve-54bb84cd5f-dpl7n" event={"ID":"63e3a68d-90a1-4e09-8967-a1fb5d99b87a","Type":"ContainerStarted","Data":"28c10fb6a882fd395ce6c43e14fdbb405e93ca0f288257b1702e1e296ce552b3"} Oct 29 23:30:42.835489 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:42.835385 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w0-67363b/sub-test-model-67363b-kserve-54bb84cd5f-dpl7n" podStartSLOduration=1.835370645 podStartE2EDuration="1.835370645s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:30:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:30:42.833231692 +0000 UTC m=+3423.616445118" watchObservedRunningTime="2026-10-29 23:30:42.835370645 +0000 UTC m=+3423.618584065" Oct 29 23:30:43.204794 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:43.204759 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w0-67363b/sub-test-model-67363b-kserve-54bb84cd5f-dpl7n" Oct 29 23:30:43.943534 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:43.943503 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b0c3c3-89fkw" Oct 29 23:30:44.039294 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:44.039188 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6880653f-fa32-4858-a99b-69ee0170dceb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6880653f-fa32-4858-a99b-69ee0170dceb-maas-api-openshift-ca\") pod \"6880653f-fa32-4858-a99b-69ee0170dceb\" (UID: \"6880653f-fa32-4858-a99b-69ee0170dceb\") " Oct 29 23:30:44.039574 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:44.039378 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6880653f-fa32-4858-a99b-69ee0170dceb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6880653f-fa32-4858-a99b-69ee0170dceb-maas-api-xks-ca\") pod \"6880653f-fa32-4858-a99b-69ee0170dceb\" (UID: \"6880653f-fa32-4858-a99b-69ee0170dceb\") " Oct 29 23:30:44.039574 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:44.039516 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6880653f-fa32-4858-a99b-69ee0170dceb-kube-api-access-fn7kf\" (UniqueName: \"kubernetes.io/projected/6880653f-fa32-4858-a99b-69ee0170dceb-kube-api-access-fn7kf\") pod \"6880653f-fa32-4858-a99b-69ee0170dceb\" (UID: \"6880653f-fa32-4858-a99b-69ee0170dceb\") " Oct 29 23:30:44.040459 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:44.040408 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6880653f-fa32-4858-a99b-69ee0170dceb-maas-api-openshift-ca" pod "6880653f-fa32-4858-a99b-69ee0170dceb" (UID: "6880653f-fa32-4858-a99b-69ee0170dceb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:30:44.042907 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:44.042881 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6880653f-fa32-4858-a99b-69ee0170dceb-maas-api-xks-ca" pod "6880653f-fa32-4858-a99b-69ee0170dceb" (UID: "6880653f-fa32-4858-a99b-69ee0170dceb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:30:44.049833 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:44.049803 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6880653f-fa32-4858-a99b-69ee0170dceb-kube-api-access-fn7kf" pod "6880653f-fa32-4858-a99b-69ee0170dceb" (UID: "6880653f-fa32-4858-a99b-69ee0170dceb"). InnerVolumeSpecName "kube-api-access-fn7kf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:30:44.141291 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:44.141251 5057 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6880653f-fa32-4858-a99b-69ee0170dceb-maas-api-openshift-ca\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:30:44.141291 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:44.141285 5057 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6880653f-fa32-4858-a99b-69ee0170dceb-maas-api-xks-ca\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:30:44.141520 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:44.141299 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fn7kf\" (UniqueName: \"kubernetes.io/projected/6880653f-fa32-4858-a99b-69ee0170dceb-kube-api-access-fn7kf\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:30:44.196049 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:44.196014 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:30:44.400705 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:44.400613 5057 scope.go:122] "RemoveContainer" containerID="6aab7b4b33e9fe9a90e8953ddc3741dee968cfb8d23b118ddc04fdac74827168" Oct 29 23:30:44.412740 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:44.412716 5057 scope.go:122] "RemoveContainer" containerID="c78d29d3d8e8b0093bbb73cd61b996e54910b45024427dc93a98bbc0f44a71a8" Oct 29 23:30:44.426209 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:44.426183 5057 scope.go:122] "RemoveContainer" containerID="b0f1a55b3c652898148768d4a20760b3fadcc936a6919318572cc2d1af4536ac" Oct 29 23:30:44.436832 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:44.436806 5057 scope.go:122] "RemoveContainer" containerID="66bb09abfff0bfd4ff98b4b202dd35234eb6d192448c9b84d205e5bee42ba498" Oct 29 23:30:44.445971 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:44.445953 5057 scope.go:122] "RemoveContainer" containerID="5f642f6da1321340d8836d25584c21bac0d601e4535e3cfe4c49802b7593c203" Oct 29 23:30:44.771460 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:44.771422 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b0c3c3-89fkw" Oct 29 23:30:44.771640 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:44.771465 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b0c3c3-89fkw" event={"ID":"6880653f-fa32-4858-a99b-69ee0170dceb","Type":"ContainerDied","Data":"303a395be470c7097ab8c568b31e125af98ef951629451f766e620250ee4343d"} Oct 29 23:30:44.771640 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:44.771512 5057 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="303a395be470c7097ab8c568b31e125af98ef951629451f766e620250ee4343d" Oct 29 23:30:45.031416 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:45.031285 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3b55d0dc-sbbh9"] Oct 29 23:30:45.035122 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:45.035096 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3b55d0dc-sbbh9"] Oct 29 23:30:45.330971 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:45.330891 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="288b2965-dc7b-4096-81dd-5369e2f9f3ce" path="/var/lib/kubelet/pods/288b2965-dc7b-4096-81dd-5369e2f9f3ce/volumes" Oct 29 23:30:45.537794 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:45.537759 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:30:45.999752 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:45.999710 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:30:47.518908 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:47.518872 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:30:48.033796 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:48.033761 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd70c84b-f9pp7"] Oct 29 23:30:48.044450 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:48.044408 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd70c84b-f9pp7"] Oct 29 23:30:49.028458 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:30:49.028417 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d2dnn" containerName="extractor" Oct 29 23:30:49.029514 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:49.029485 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:30:49.339657 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:49.339572 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dbd19eb3-9ab3-473c-9a3c-1f63abca271c" path="/var/lib/kubelet/pods/dbd19eb3-9ab3-473c-9a3c-1f63abca271c/volumes" Oct 29 23:30:50.533954 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:50.533916 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:30:50.995899 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:50.995864 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-b0c3c3-5867dcd5b4-xs5ll"] Oct 29 23:30:51.033441 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:51.033406 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b0c3c3-75d548567-9267h"] Oct 29 23:30:51.143222 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:51.143141 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:30:51.254073 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:51.253958 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:30:52.157025 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:52.156992 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:30:52.204570 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:52.204515 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w0-67363b/sub-test-model-67363b-kserve-54bb84cd5f-dpl7n" Oct 29 23:30:52.231033 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:52.231003 5057 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w0-67363b/sub-test-model-67363b-kserve-54bb84cd5f-dpl7n" Oct 29 23:30:52.850079 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:52.850050 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w0-67363b/sub-test-model-67363b-kserve-54bb84cd5f-dpl7n" Oct 29 23:30:53.814087 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:53.814042 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:30:54.030373 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:54.030344 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6210df6f-lcb4g"] Oct 29 23:30:54.041322 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:54.041294 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6210df6f-lcb4g"] Oct 29 23:30:55.077132 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:55.077085 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:30:55.298608 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:55.296381 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w2-ed3bf2/rate-test-model-ed3bf2-kserve-7bbcbb9dd9-xpmj5" podUID="6084f4e5-c0ca-4f04-9aec-b1236e3d66db" containerName="main" containerID="cri-o://c6747472cb2c4c62b7ad8448c9ee1151edf2210f7c43034b63f2f28f1620a975" gracePeriod=45 Oct 29 23:30:55.331651 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:55.331581 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40d8fc20-6f9a-42af-b255-ce95dd53761b" path="/var/lib/kubelet/pods/40d8fc20-6f9a-42af-b255-ce95dd53761b/volumes" Oct 29 23:30:55.631212 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:55.631183 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-ed3bf2/rate-test-model-ed3bf2-kserve-7bbcbb9dd9-xpmj5" Oct 29 23:30:55.660400 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:55.660365 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:30:55.706624 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:55.706584 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6084f4e5-c0ca-4f04-9aec-b1236e3d66db-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6084f4e5-c0ca-4f04-9aec-b1236e3d66db-dshm\") pod \"6084f4e5-c0ca-4f04-9aec-b1236e3d66db\" (UID: \"6084f4e5-c0ca-4f04-9aec-b1236e3d66db\") " Oct 29 23:30:55.706808 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:55.706707 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6084f4e5-c0ca-4f04-9aec-b1236e3d66db-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6084f4e5-c0ca-4f04-9aec-b1236e3d66db-model-cache\") pod \"6084f4e5-c0ca-4f04-9aec-b1236e3d66db\" (UID: \"6084f4e5-c0ca-4f04-9aec-b1236e3d66db\") " Oct 29 23:30:55.706808 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:55.706791 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6084f4e5-c0ca-4f04-9aec-b1236e3d66db-tls-certs\" (UniqueName: \"kubernetes.io/secret/6084f4e5-c0ca-4f04-9aec-b1236e3d66db-tls-certs\") pod \"6084f4e5-c0ca-4f04-9aec-b1236e3d66db\" (UID: \"6084f4e5-c0ca-4f04-9aec-b1236e3d66db\") " Oct 29 23:30:55.706950 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:55.706881 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6084f4e5-c0ca-4f04-9aec-b1236e3d66db-home\" (UniqueName: \"kubernetes.io/empty-dir/6084f4e5-c0ca-4f04-9aec-b1236e3d66db-home\") pod \"6084f4e5-c0ca-4f04-9aec-b1236e3d66db\" (UID: \"6084f4e5-c0ca-4f04-9aec-b1236e3d66db\") " Oct 29 23:30:55.707012 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:55.706965 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6084f4e5-c0ca-4f04-9aec-b1236e3d66db-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6084f4e5-c0ca-4f04-9aec-b1236e3d66db-tmp-dir\") pod \"6084f4e5-c0ca-4f04-9aec-b1236e3d66db\" (UID: \"6084f4e5-c0ca-4f04-9aec-b1236e3d66db\") " Oct 29 23:30:55.707070 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:55.707010 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6084f4e5-c0ca-4f04-9aec-b1236e3d66db-kube-api-access-pp5bq\" (UniqueName: \"kubernetes.io/projected/6084f4e5-c0ca-4f04-9aec-b1236e3d66db-kube-api-access-pp5bq\") pod \"6084f4e5-c0ca-4f04-9aec-b1236e3d66db\" (UID: \"6084f4e5-c0ca-4f04-9aec-b1236e3d66db\") " Oct 29 23:30:55.708220 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:55.708190 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6084f4e5-c0ca-4f04-9aec-b1236e3d66db-home" pod "6084f4e5-c0ca-4f04-9aec-b1236e3d66db" (UID: "6084f4e5-c0ca-4f04-9aec-b1236e3d66db"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:30:55.708460 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:55.708433 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6084f4e5-c0ca-4f04-9aec-b1236e3d66db-tmp-dir" pod "6084f4e5-c0ca-4f04-9aec-b1236e3d66db" (UID: "6084f4e5-c0ca-4f04-9aec-b1236e3d66db"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:30:55.708651 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:55.708630 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6084f4e5-c0ca-4f04-9aec-b1236e3d66db-model-cache" pod "6084f4e5-c0ca-4f04-9aec-b1236e3d66db" (UID: "6084f4e5-c0ca-4f04-9aec-b1236e3d66db"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:30:55.709884 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:55.709846 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6084f4e5-c0ca-4f04-9aec-b1236e3d66db-tls-certs" pod "6084f4e5-c0ca-4f04-9aec-b1236e3d66db" (UID: "6084f4e5-c0ca-4f04-9aec-b1236e3d66db"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:30:55.725806 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:55.725778 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6084f4e5-c0ca-4f04-9aec-b1236e3d66db-dshm" pod "6084f4e5-c0ca-4f04-9aec-b1236e3d66db" (UID: "6084f4e5-c0ca-4f04-9aec-b1236e3d66db"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:30:55.726021 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:55.726004 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6084f4e5-c0ca-4f04-9aec-b1236e3d66db-kube-api-access-pp5bq" pod "6084f4e5-c0ca-4f04-9aec-b1236e3d66db" (UID: "6084f4e5-c0ca-4f04-9aec-b1236e3d66db"). InnerVolumeSpecName "kube-api-access-pp5bq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:30:55.768422 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:55.768395 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:30:55.774887 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:55.774862 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b0c3c3-89fkw"] Oct 29 23:30:55.778625 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:55.778595 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b0c3c3-89fkw"] Oct 29 23:30:55.809024 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:55.808996 5057 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6084f4e5-c0ca-4f04-9aec-b1236e3d66db-dshm\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:30:55.809024 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:55.809020 5057 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6084f4e5-c0ca-4f04-9aec-b1236e3d66db-model-cache\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:30:55.809024 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:55.809030 5057 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6084f4e5-c0ca-4f04-9aec-b1236e3d66db-tls-certs\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:30:55.809201 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:55.809038 5057 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6084f4e5-c0ca-4f04-9aec-b1236e3d66db-home\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:30:55.809201 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:55.809045 5057 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6084f4e5-c0ca-4f04-9aec-b1236e3d66db-tmp-dir\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:30:55.809201 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:55.809055 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pp5bq\" (UniqueName: \"kubernetes.io/projected/6084f4e5-c0ca-4f04-9aec-b1236e3d66db-kube-api-access-pp5bq\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:30:55.868094 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:55.868007 5057 generic.go:356] "Generic (PLEG): container finished" podID="6084f4e5-c0ca-4f04-9aec-b1236e3d66db" containerID="c6747472cb2c4c62b7ad8448c9ee1151edf2210f7c43034b63f2f28f1620a975" exitCode=0 Oct 29 23:30:55.868094 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:55.868055 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-ed3bf2/rate-test-model-ed3bf2-kserve-7bbcbb9dd9-xpmj5" Oct 29 23:30:55.868309 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:55.868091 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-ed3bf2/rate-test-model-ed3bf2-kserve-7bbcbb9dd9-xpmj5" event={"ID":"6084f4e5-c0ca-4f04-9aec-b1236e3d66db","Type":"ContainerDied","Data":"c6747472cb2c4c62b7ad8448c9ee1151edf2210f7c43034b63f2f28f1620a975"} Oct 29 23:30:55.868309 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:55.868140 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-ed3bf2/rate-test-model-ed3bf2-kserve-7bbcbb9dd9-xpmj5" event={"ID":"6084f4e5-c0ca-4f04-9aec-b1236e3d66db","Type":"ContainerDied","Data":"319b9103138b6f2241c89f9cab27ff4049f5a6ec9123fc3235cf6cd57e13ac95"} Oct 29 23:30:55.868309 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:55.868163 5057 scope.go:122] "RemoveContainer" containerID="c6747472cb2c4c62b7ad8448c9ee1151edf2210f7c43034b63f2f28f1620a975" Oct 29 23:30:55.875463 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:55.875430 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:30:55.884682 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:55.884662 5057 scope.go:122] "RemoveContainer" containerID="c6747472cb2c4c62b7ad8448c9ee1151edf2210f7c43034b63f2f28f1620a975" Oct 29 23:30:55.884981 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:30:55.884953 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6747472cb2c4c62b7ad8448c9ee1151edf2210f7c43034b63f2f28f1620a975\": container with ID starting with c6747472cb2c4c62b7ad8448c9ee1151edf2210f7c43034b63f2f28f1620a975 not found: ID does not exist" containerID="c6747472cb2c4c62b7ad8448c9ee1151edf2210f7c43034b63f2f28f1620a975" Oct 29 23:30:55.885049 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:55.884994 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6747472cb2c4c62b7ad8448c9ee1151edf2210f7c43034b63f2f28f1620a975"} err="failed to get container status \"c6747472cb2c4c62b7ad8448c9ee1151edf2210f7c43034b63f2f28f1620a975\": rpc error: code = NotFound desc = could not find container \"c6747472cb2c4c62b7ad8448c9ee1151edf2210f7c43034b63f2f28f1620a975\": container with ID starting with c6747472cb2c4c62b7ad8448c9ee1151edf2210f7c43034b63f2f28f1620a975 not found: ID does not exist" Oct 29 23:30:55.922775 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:55.922751 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-ed3bf2/rate-test-model-ed3bf2-kserve-7bbcbb9dd9-xpmj5"] Oct 29 23:30:55.927717 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:55.927686 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w2-ed3bf2/rate-test-model-ed3bf2-kserve-7bbcbb9dd9-xpmj5"] Oct 29 23:30:55.996296 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:55.996257 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b0c3c3-5867dcd5b4-xs5ll" podUID="93b0246c-b176-4daa-8090-6bb557b4fa84" containerName="payload-processing" containerID="cri-o://c4d8ab74870b6d6166812a9f91d9a3a42f00578efa4d9806c00f59c3a09fcbc5" gracePeriod=25 Oct 29 23:30:56.033809 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:56.033777 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b0c3c3-75d548567-9267h" podUID="34be4000-c954-4c58-a304-1944cace0093" containerName="payload-pre-processing" containerID="cri-o://6794564be13c7916c1a2d4e55916bade68a5ef4d697eed35e8e76c4545b2b8cc" gracePeriod=25 Oct 29 23:30:56.292453 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:56.292412 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:30:57.029779 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:57.029742 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96f97c8a-5hxsl"] Oct 29 23:30:57.040557 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:57.040529 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96f97c8a-5hxsl"] Oct 29 23:30:57.331289 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:57.331204 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6084f4e5-c0ca-4f04-9aec-b1236e3d66db" path="/var/lib/kubelet/pods/6084f4e5-c0ca-4f04-9aec-b1236e3d66db/volumes" Oct 29 23:30:57.331860 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:57.331839 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6880653f-fa32-4858-a99b-69ee0170dceb" path="/var/lib/kubelet/pods/6880653f-fa32-4858-a99b-69ee0170dceb/volumes" Oct 29 23:30:57.332429 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:57.332406 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="baaefff9-3dbc-4f65-80bb-f02ac48ae739" path="/var/lib/kubelet/pods/baaefff9-3dbc-4f65-80bb-f02ac48ae739/volumes" Oct 29 23:30:57.468312 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:57.468276 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:30:57.887504 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:57.887460 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v2tfm"] Oct 29 23:30:59.136235 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:30:59.136200 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:31:00.735528 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:00.735491 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:31:01.141769 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:01.141676 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b0c3c3-75d548567-9267h" podUID="34be4000-c954-4c58-a304-1944cace0093" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:31:02.149432 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:02.149397 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b0c3c3-5867dcd5b4-xs5ll" podUID="93b0246c-b176-4daa-8090-6bb557b4fa84" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:31:02.158170 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:02.158140 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e50149e-slmz8"] Oct 29 23:31:02.158990 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:02.158965 5057 cpu_manager.go:414] "containerMap: removing container" podUID="6880653f-fa32-4858-a99b-69ee0170dceb" containerName="revoke-keys" Oct 29 23:31:02.159116 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:02.158990 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6880653f-fa32-4858-a99b-69ee0170dceb" containerName="revoke-keys" Oct 29 23:31:02.159116 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:02.159019 5057 cpu_manager.go:414] "containerMap: removing container" podUID="6084f4e5-c0ca-4f04-9aec-b1236e3d66db" containerName="main" Oct 29 23:31:02.159116 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:02.159027 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6084f4e5-c0ca-4f04-9aec-b1236e3d66db" containerName="main" Oct 29 23:31:02.159274 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:02.159255 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="6084f4e5-c0ca-4f04-9aec-b1236e3d66db" containerName="main" Oct 29 23:31:02.159372 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:02.159287 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="6880653f-fa32-4858-a99b-69ee0170dceb" containerName="revoke-keys" Oct 29 23:31:02.164579 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:02.164559 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e50149e-slmz8" Oct 29 23:31:02.183828 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:02.183138 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e50149e-slmz8"] Oct 29 23:31:02.189780 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:02.189759 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:31:02.189935 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:02.189917 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dmx57\"" Oct 29 23:31:02.210715 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:02.210687 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1851a9f-eb0f-493f-87f3-6246aaa79dde-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9e50149e-slmz8\" (UID: \"a1851a9f-eb0f-493f-87f3-6246aaa79dde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e50149e-slmz8" Oct 29 23:31:02.210863 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:02.210823 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1851a9f-eb0f-493f-87f3-6246aaa79dde-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9e50149e-slmz8\" (UID: \"a1851a9f-eb0f-493f-87f3-6246aaa79dde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e50149e-slmz8" Oct 29 23:31:02.211056 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:02.211037 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sh5q2\" (UniqueName: \"kubernetes.io/projected/a1851a9f-eb0f-493f-87f3-6246aaa79dde-kube-api-access-sh5q2\") pod \"maas-api-revoke-sub-9e50149e-slmz8\" (UID: \"a1851a9f-eb0f-493f-87f3-6246aaa79dde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e50149e-slmz8" Oct 29 23:31:02.250569 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:02.250532 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:31:02.312430 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:02.312394 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sh5q2\" (UniqueName: \"kubernetes.io/projected/a1851a9f-eb0f-493f-87f3-6246aaa79dde-kube-api-access-sh5q2\") pod \"maas-api-revoke-sub-9e50149e-slmz8\" (UID: \"a1851a9f-eb0f-493f-87f3-6246aaa79dde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e50149e-slmz8" Oct 29 23:31:02.312651 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:02.312573 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1851a9f-eb0f-493f-87f3-6246aaa79dde-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9e50149e-slmz8\" (UID: \"a1851a9f-eb0f-493f-87f3-6246aaa79dde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e50149e-slmz8" Oct 29 23:31:02.312732 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:02.312657 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1851a9f-eb0f-493f-87f3-6246aaa79dde-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9e50149e-slmz8\" (UID: \"a1851a9f-eb0f-493f-87f3-6246aaa79dde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e50149e-slmz8" Oct 29 23:31:02.313436 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:02.313396 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1851a9f-eb0f-493f-87f3-6246aaa79dde-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9e50149e-slmz8\" (UID: \"a1851a9f-eb0f-493f-87f3-6246aaa79dde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e50149e-slmz8" Oct 29 23:31:02.315514 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:02.315487 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1851a9f-eb0f-493f-87f3-6246aaa79dde-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9e50149e-slmz8\" (UID: \"a1851a9f-eb0f-493f-87f3-6246aaa79dde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e50149e-slmz8" Oct 29 23:31:02.335267 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:02.335245 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sh5q2\" (UniqueName: \"kubernetes.io/projected/a1851a9f-eb0f-493f-87f3-6246aaa79dde-kube-api-access-sh5q2\") pod \"maas-api-revoke-sub-9e50149e-slmz8\" (UID: \"a1851a9f-eb0f-493f-87f3-6246aaa79dde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e50149e-slmz8" Oct 29 23:31:02.478948 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:02.478925 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e50149e-slmz8" Oct 29 23:31:02.784581 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:02.784547 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e50149e-slmz8"] Oct 29 23:31:02.786302 ip-10-0-140-74 kubenswrapper[5057]: W1029 23:31:02.786271 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1851a9f_eb0f_493f_87f3_6246aaa79dde.slice/crio-4917ae62f64bc23de58a13d341d07d44d32ac3bea511dbb92c363070436956e7 WatchSource:0}: Error finding container 4917ae62f64bc23de58a13d341d07d44d32ac3bea511dbb92c363070436956e7: Status 404 returned error can't find the container with id 4917ae62f64bc23de58a13d341d07d44d32ac3bea511dbb92c363070436956e7 Oct 29 23:31:02.930316 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:02.930283 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e50149e-slmz8" event={"ID":"a1851a9f-eb0f-493f-87f3-6246aaa79dde","Type":"ContainerStarted","Data":"4917ae62f64bc23de58a13d341d07d44d32ac3bea511dbb92c363070436956e7"} Oct 29 23:31:03.334907 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:03.334866 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:31:03.444551 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:03.444458 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:31:03.663232 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:03.663197 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-lhtzn"] Oct 29 23:31:03.663518 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:03.663490 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-lhtzn" podUID="b6b816a6-841c-405c-b1ae-d6f9d1a4982b" containerName="istio-proxy" containerID="cri-o://3394233411ab32e0309e5f6408aa9ad64cde57d4c143fdc27c4a4518de954876" gracePeriod=30 Oct 29 23:31:03.699813 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:03.699784 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:31:03.919021 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:03.918982 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:31:03.940862 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:03.940829 5057 generic.go:356] "Generic (PLEG): container finished" podID="a1851a9f-eb0f-493f-87f3-6246aaa79dde" containerID="c493c3cc132ff36827afa8ab7c0ca35a7a18fb0f44136b1c6db4be1ef70686fc" exitCode=0 Oct 29 23:31:03.941035 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:03.940909 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e50149e-slmz8" event={"ID":"a1851a9f-eb0f-493f-87f3-6246aaa79dde","Type":"ContainerDied","Data":"c493c3cc132ff36827afa8ab7c0ca35a7a18fb0f44136b1c6db4be1ef70686fc"} Oct 29 23:31:05.126569 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:05.126539 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e50149e-slmz8" Oct 29 23:31:05.154188 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:05.154161 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1851a9f-eb0f-493f-87f3-6246aaa79dde-kube-api-access-sh5q2\" (UniqueName: \"kubernetes.io/projected/a1851a9f-eb0f-493f-87f3-6246aaa79dde-kube-api-access-sh5q2\") pod \"a1851a9f-eb0f-493f-87f3-6246aaa79dde\" (UID: \"a1851a9f-eb0f-493f-87f3-6246aaa79dde\") " Oct 29 23:31:05.154321 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:05.154282 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a1851a9f-eb0f-493f-87f3-6246aaa79dde-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1851a9f-eb0f-493f-87f3-6246aaa79dde-maas-api-xks-ca\") pod \"a1851a9f-eb0f-493f-87f3-6246aaa79dde\" (UID: \"a1851a9f-eb0f-493f-87f3-6246aaa79dde\") " Oct 29 23:31:05.154493 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:05.154477 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1851a9f-eb0f-493f-87f3-6246aaa79dde-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1851a9f-eb0f-493f-87f3-6246aaa79dde-maas-api-openshift-ca\") pod \"a1851a9f-eb0f-493f-87f3-6246aaa79dde\" (UID: \"a1851a9f-eb0f-493f-87f3-6246aaa79dde\") " Oct 29 23:31:05.154805 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:05.154781 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1851a9f-eb0f-493f-87f3-6246aaa79dde-maas-api-openshift-ca" pod "a1851a9f-eb0f-493f-87f3-6246aaa79dde" (UID: "a1851a9f-eb0f-493f-87f3-6246aaa79dde"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:31:05.155025 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:05.155007 5057 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1851a9f-eb0f-493f-87f3-6246aaa79dde-maas-api-openshift-ca\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:31:05.156307 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:05.156281 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1851a9f-eb0f-493f-87f3-6246aaa79dde-kube-api-access-sh5q2" pod "a1851a9f-eb0f-493f-87f3-6246aaa79dde" (UID: "a1851a9f-eb0f-493f-87f3-6246aaa79dde"). InnerVolumeSpecName "kube-api-access-sh5q2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:31:05.156469 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:05.156347 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1851a9f-eb0f-493f-87f3-6246aaa79dde-maas-api-xks-ca" pod "a1851a9f-eb0f-493f-87f3-6246aaa79dde" (UID: "a1851a9f-eb0f-493f-87f3-6246aaa79dde"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:31:05.256695 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:05.256615 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sh5q2\" (UniqueName: \"kubernetes.io/projected/a1851a9f-eb0f-493f-87f3-6246aaa79dde-kube-api-access-sh5q2\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:31:05.256695 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:05.256647 5057 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1851a9f-eb0f-493f-87f3-6246aaa79dde-maas-api-xks-ca\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:31:05.620875 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:05.620789 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:31:05.962636 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:05.962605 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e50149e-slmz8" Oct 29 23:31:05.962850 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:05.962639 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e50149e-slmz8" event={"ID":"a1851a9f-eb0f-493f-87f3-6246aaa79dde","Type":"ContainerDied","Data":"4917ae62f64bc23de58a13d341d07d44d32ac3bea511dbb92c363070436956e7"} Oct 29 23:31:05.962850 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:05.962695 5057 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4917ae62f64bc23de58a13d341d07d44d32ac3bea511dbb92c363070436956e7" Oct 29 23:31:07.307869 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:07.307778 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:31:08.894638 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:08.894580 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:31:08.977419 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:08.977391 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-lhtzn" Oct 29 23:31:08.990774 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:08.990741 5057 generic.go:356] "Generic (PLEG): container finished" podID="b6b816a6-841c-405c-b1ae-d6f9d1a4982b" containerID="3394233411ab32e0309e5f6408aa9ad64cde57d4c143fdc27c4a4518de954876" exitCode=0 Oct 29 23:31:08.990954 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:08.990797 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-lhtzn" Oct 29 23:31:08.990954 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:08.990835 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-lhtzn" event={"ID":"b6b816a6-841c-405c-b1ae-d6f9d1a4982b","Type":"ContainerDied","Data":"3394233411ab32e0309e5f6408aa9ad64cde57d4c143fdc27c4a4518de954876"} Oct 29 23:31:08.990954 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:08.990885 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-lhtzn" event={"ID":"b6b816a6-841c-405c-b1ae-d6f9d1a4982b","Type":"ContainerDied","Data":"4145922ed14e1f83347a7e6563ceee735a559c680b838d90bb900260a83e5126"} Oct 29 23:31:08.990954 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:08.990913 5057 scope.go:122] "RemoveContainer" containerID="3394233411ab32e0309e5f6408aa9ad64cde57d4c143fdc27c4a4518de954876" Oct 29 23:31:09.002347 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:09.002318 5057 scope.go:122] "RemoveContainer" containerID="3394233411ab32e0309e5f6408aa9ad64cde57d4c143fdc27c4a4518de954876" Oct 29 23:31:09.002651 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:31:09.002625 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3394233411ab32e0309e5f6408aa9ad64cde57d4c143fdc27c4a4518de954876\": container with ID starting with 3394233411ab32e0309e5f6408aa9ad64cde57d4c143fdc27c4a4518de954876 not found: ID does not exist" containerID="3394233411ab32e0309e5f6408aa9ad64cde57d4c143fdc27c4a4518de954876" Oct 29 23:31:09.002729 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:09.002657 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3394233411ab32e0309e5f6408aa9ad64cde57d4c143fdc27c4a4518de954876"} err="failed to get container status \"3394233411ab32e0309e5f6408aa9ad64cde57d4c143fdc27c4a4518de954876\": rpc error: code = NotFound desc = could not find container \"3394233411ab32e0309e5f6408aa9ad64cde57d4c143fdc27c4a4518de954876\": container with ID starting with 3394233411ab32e0309e5f6408aa9ad64cde57d4c143fdc27c4a4518de954876 not found: ID does not exist" Oct 29 23:31:09.107591 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:09.107553 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6b816a6-841c-405c-b1ae-d6f9d1a4982b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6b816a6-841c-405c-b1ae-d6f9d1a4982b-istio-data\") pod \"b6b816a6-841c-405c-b1ae-d6f9d1a4982b\" (UID: \"b6b816a6-841c-405c-b1ae-d6f9d1a4982b\") " Oct 29 23:31:09.107750 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:09.107616 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b6b816a6-841c-405c-b1ae-d6f9d1a4982b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6b816a6-841c-405c-b1ae-d6f9d1a4982b-istio-podinfo\") pod \"b6b816a6-841c-405c-b1ae-d6f9d1a4982b\" (UID: \"b6b816a6-841c-405c-b1ae-d6f9d1a4982b\") " Oct 29 23:31:09.107797 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:09.107773 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6b816a6-841c-405c-b1ae-d6f9d1a4982b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6b816a6-841c-405c-b1ae-d6f9d1a4982b-workload-socket\") pod \"b6b816a6-841c-405c-b1ae-d6f9d1a4982b\" (UID: \"b6b816a6-841c-405c-b1ae-d6f9d1a4982b\") " Oct 29 23:31:09.107857 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:09.107832 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6b816a6-841c-405c-b1ae-d6f9d1a4982b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6b816a6-841c-405c-b1ae-d6f9d1a4982b-istio-envoy\") pod \"b6b816a6-841c-405c-b1ae-d6f9d1a4982b\" (UID: \"b6b816a6-841c-405c-b1ae-d6f9d1a4982b\") " Oct 29 23:31:09.107929 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:09.107894 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6b816a6-841c-405c-b1ae-d6f9d1a4982b-istio-data" pod "b6b816a6-841c-405c-b1ae-d6f9d1a4982b" (UID: "b6b816a6-841c-405c-b1ae-d6f9d1a4982b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:31:09.108043 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:09.107911 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b6b816a6-841c-405c-b1ae-d6f9d1a4982b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6b816a6-841c-405c-b1ae-d6f9d1a4982b-istiod-ca-cert\") pod \"b6b816a6-841c-405c-b1ae-d6f9d1a4982b\" (UID: \"b6b816a6-841c-405c-b1ae-d6f9d1a4982b\") " Oct 29 23:31:09.108108 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:09.108035 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6b816a6-841c-405c-b1ae-d6f9d1a4982b-workload-socket" pod "b6b816a6-841c-405c-b1ae-d6f9d1a4982b" (UID: "b6b816a6-841c-405c-b1ae-d6f9d1a4982b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:31:09.108108 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:09.108049 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6b816a6-841c-405c-b1ae-d6f9d1a4982b-istio-token\" (UniqueName: \"kubernetes.io/projected/b6b816a6-841c-405c-b1ae-d6f9d1a4982b-istio-token\") pod \"b6b816a6-841c-405c-b1ae-d6f9d1a4982b\" (UID: \"b6b816a6-841c-405c-b1ae-d6f9d1a4982b\") " Oct 29 23:31:09.108436 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:09.108136 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6b816a6-841c-405c-b1ae-d6f9d1a4982b-kube-api-access-6mzkm\" (UniqueName: \"kubernetes.io/projected/b6b816a6-841c-405c-b1ae-d6f9d1a4982b-kube-api-access-6mzkm\") pod \"b6b816a6-841c-405c-b1ae-d6f9d1a4982b\" (UID: \"b6b816a6-841c-405c-b1ae-d6f9d1a4982b\") " Oct 29 23:31:09.108436 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:09.108180 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6b816a6-841c-405c-b1ae-d6f9d1a4982b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6b816a6-841c-405c-b1ae-d6f9d1a4982b-credential-socket\") pod \"b6b816a6-841c-405c-b1ae-d6f9d1a4982b\" (UID: \"b6b816a6-841c-405c-b1ae-d6f9d1a4982b\") " Oct 29 23:31:09.108436 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:09.108246 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6b816a6-841c-405c-b1ae-d6f9d1a4982b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6b816a6-841c-405c-b1ae-d6f9d1a4982b-workload-certs\") pod \"b6b816a6-841c-405c-b1ae-d6f9d1a4982b\" (UID: \"b6b816a6-841c-405c-b1ae-d6f9d1a4982b\") " Oct 29 23:31:09.108436 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:09.108225 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6b816a6-841c-405c-b1ae-d6f9d1a4982b-istiod-ca-cert" pod "b6b816a6-841c-405c-b1ae-d6f9d1a4982b" (UID: "b6b816a6-841c-405c-b1ae-d6f9d1a4982b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:31:09.109018 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:09.108892 5057 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6b816a6-841c-405c-b1ae-d6f9d1a4982b-istiod-ca-cert\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:31:09.109018 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:09.108916 5057 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6b816a6-841c-405c-b1ae-d6f9d1a4982b-istio-data\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:31:09.109018 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:09.108929 5057 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6b816a6-841c-405c-b1ae-d6f9d1a4982b-workload-socket\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:31:09.109018 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:09.108965 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6b816a6-841c-405c-b1ae-d6f9d1a4982b-credential-socket" pod "b6b816a6-841c-405c-b1ae-d6f9d1a4982b" (UID: "b6b816a6-841c-405c-b1ae-d6f9d1a4982b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:31:09.109374 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:09.109182 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6b816a6-841c-405c-b1ae-d6f9d1a4982b-workload-certs" pod "b6b816a6-841c-405c-b1ae-d6f9d1a4982b" (UID: "b6b816a6-841c-405c-b1ae-d6f9d1a4982b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:31:09.110348 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:09.110303 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b6b816a6-841c-405c-b1ae-d6f9d1a4982b-istio-podinfo" pod "b6b816a6-841c-405c-b1ae-d6f9d1a4982b" (UID: "b6b816a6-841c-405c-b1ae-d6f9d1a4982b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:31:09.110733 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:09.110687 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6b816a6-841c-405c-b1ae-d6f9d1a4982b-istio-token" pod "b6b816a6-841c-405c-b1ae-d6f9d1a4982b" (UID: "b6b816a6-841c-405c-b1ae-d6f9d1a4982b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:31:09.111238 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:09.111214 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6b816a6-841c-405c-b1ae-d6f9d1a4982b-kube-api-access-6mzkm" pod "b6b816a6-841c-405c-b1ae-d6f9d1a4982b" (UID: "b6b816a6-841c-405c-b1ae-d6f9d1a4982b"). InnerVolumeSpecName "kube-api-access-6mzkm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:31:09.111369 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:09.111310 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6b816a6-841c-405c-b1ae-d6f9d1a4982b-istio-envoy" pod "b6b816a6-841c-405c-b1ae-d6f9d1a4982b" (UID: "b6b816a6-841c-405c-b1ae-d6f9d1a4982b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:31:09.210627 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:09.210598 5057 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6b816a6-841c-405c-b1ae-d6f9d1a4982b-istio-podinfo\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:31:09.210755 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:09.210631 5057 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6b816a6-841c-405c-b1ae-d6f9d1a4982b-istio-envoy\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:31:09.210755 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:09.210644 5057 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6b816a6-841c-405c-b1ae-d6f9d1a4982b-istio-token\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:31:09.210755 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:09.210657 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6mzkm\" (UniqueName: \"kubernetes.io/projected/b6b816a6-841c-405c-b1ae-d6f9d1a4982b-kube-api-access-6mzkm\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:31:09.210755 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:09.210673 5057 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6b816a6-841c-405c-b1ae-d6f9d1a4982b-credential-socket\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:31:09.210755 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:09.210687 5057 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6b816a6-841c-405c-b1ae-d6f9d1a4982b-workload-certs\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:31:09.339552 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:09.339525 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-lhtzn"] Oct 29 23:31:09.350296 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:09.350271 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-lhtzn"] Oct 29 23:31:10.442405 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:10.442366 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:31:11.141989 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:11.141944 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b0c3c3-75d548567-9267h" podUID="34be4000-c954-4c58-a304-1944cace0093" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:31:11.331556 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:11.331523 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b6b816a6-841c-405c-b1ae-d6f9d1a4982b" path="/var/lib/kubelet/pods/b6b816a6-841c-405c-b1ae-d6f9d1a4982b/volumes" Oct 29 23:31:11.987570 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:11.987530 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:31:12.149184 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:12.149142 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b0c3c3-5867dcd5b4-xs5ll" podUID="93b0246c-b176-4daa-8090-6bb557b4fa84" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:31:12.529860 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:12.529825 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e26a6b27-6wjrq"] Oct 29 23:31:12.530582 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:12.530565 5057 cpu_manager.go:414] "containerMap: removing container" podUID="a1851a9f-eb0f-493f-87f3-6246aaa79dde" containerName="revoke-keys" Oct 29 23:31:12.530582 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:12.530582 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1851a9f-eb0f-493f-87f3-6246aaa79dde" containerName="revoke-keys" Oct 29 23:31:12.530692 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:12.530614 5057 cpu_manager.go:414] "containerMap: removing container" podUID="b6b816a6-841c-405c-b1ae-d6f9d1a4982b" containerName="istio-proxy" Oct 29 23:31:12.530692 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:12.530619 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6b816a6-841c-405c-b1ae-d6f9d1a4982b" containerName="istio-proxy" Oct 29 23:31:12.530766 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:12.530745 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6b816a6-841c-405c-b1ae-d6f9d1a4982b" containerName="istio-proxy" Oct 29 23:31:12.530766 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:12.530763 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1851a9f-eb0f-493f-87f3-6246aaa79dde" containerName="revoke-keys" Oct 29 23:31:12.536046 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:12.536026 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e26a6b27-6wjrq" Oct 29 23:31:12.550771 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:12.550748 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:31:12.553682 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:12.553657 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e26a6b27-6wjrq"] Oct 29 23:31:12.567351 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:12.560836 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dmx57\"" Oct 29 23:31:12.662950 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:12.662916 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81e3720d-b019-448f-9a08-49d5ccad4cb1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e26a6b27-6wjrq\" (UID: \"81e3720d-b019-448f-9a08-49d5ccad4cb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e26a6b27-6wjrq" Oct 29 23:31:12.663107 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:12.662964 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81e3720d-b019-448f-9a08-49d5ccad4cb1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e26a6b27-6wjrq\" (UID: \"81e3720d-b019-448f-9a08-49d5ccad4cb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e26a6b27-6wjrq" Oct 29 23:31:12.663189 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:12.663140 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6rlv\" (UniqueName: \"kubernetes.io/projected/81e3720d-b019-448f-9a08-49d5ccad4cb1-kube-api-access-s6rlv\") pod \"maas-api-revoke-sub-e26a6b27-6wjrq\" (UID: \"81e3720d-b019-448f-9a08-49d5ccad4cb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e26a6b27-6wjrq" Oct 29 23:31:12.764100 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:12.764066 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6rlv\" (UniqueName: \"kubernetes.io/projected/81e3720d-b019-448f-9a08-49d5ccad4cb1-kube-api-access-s6rlv\") pod \"maas-api-revoke-sub-e26a6b27-6wjrq\" (UID: \"81e3720d-b019-448f-9a08-49d5ccad4cb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e26a6b27-6wjrq" Oct 29 23:31:12.764258 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:12.764212 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81e3720d-b019-448f-9a08-49d5ccad4cb1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e26a6b27-6wjrq\" (UID: \"81e3720d-b019-448f-9a08-49d5ccad4cb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e26a6b27-6wjrq" Oct 29 23:31:12.764258 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:12.764243 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81e3720d-b019-448f-9a08-49d5ccad4cb1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e26a6b27-6wjrq\" (UID: \"81e3720d-b019-448f-9a08-49d5ccad4cb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e26a6b27-6wjrq" Oct 29 23:31:12.764905 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:12.764880 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81e3720d-b019-448f-9a08-49d5ccad4cb1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e26a6b27-6wjrq\" (UID: \"81e3720d-b019-448f-9a08-49d5ccad4cb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e26a6b27-6wjrq" Oct 29 23:31:12.766703 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:12.766683 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81e3720d-b019-448f-9a08-49d5ccad4cb1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e26a6b27-6wjrq\" (UID: \"81e3720d-b019-448f-9a08-49d5ccad4cb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e26a6b27-6wjrq" Oct 29 23:31:12.786968 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:12.786908 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6rlv\" (UniqueName: \"kubernetes.io/projected/81e3720d-b019-448f-9a08-49d5ccad4cb1-kube-api-access-s6rlv\") pod \"maas-api-revoke-sub-e26a6b27-6wjrq\" (UID: \"81e3720d-b019-448f-9a08-49d5ccad4cb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e26a6b27-6wjrq" Oct 29 23:31:12.850176 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:12.850144 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e26a6b27-6wjrq" Oct 29 23:31:13.158248 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:13.158215 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e26a6b27-6wjrq"] Oct 29 23:31:13.159496 ip-10-0-140-74 kubenswrapper[5057]: W1029 23:31:13.159461 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81e3720d_b019_448f_9a08_49d5ccad4cb1.slice/crio-21edbe68b0343ad532e19ece14f7dfe8e6a0ce1206e2eb206135373873656dc5 WatchSource:0}: Error finding container 21edbe68b0343ad532e19ece14f7dfe8e6a0ce1206e2eb206135373873656dc5: Status 404 returned error can't find the container with id 21edbe68b0343ad532e19ece14f7dfe8e6a0ce1206e2eb206135373873656dc5 Oct 29 23:31:13.571153 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:13.571116 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:31:14.053255 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:14.053215 5057 generic.go:356] "Generic (PLEG): container finished" podID="81e3720d-b019-448f-9a08-49d5ccad4cb1" containerID="4a9159435aa7cd2dc72acfb2d59e79f25af9224034498c9a0c2a322e24a3718d" exitCode=0 Oct 29 23:31:14.053444 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:14.053360 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e26a6b27-6wjrq" event={"ID":"81e3720d-b019-448f-9a08-49d5ccad4cb1","Type":"ContainerDied","Data":"4a9159435aa7cd2dc72acfb2d59e79f25af9224034498c9a0c2a322e24a3718d"} Oct 29 23:31:14.053444 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:14.053421 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e26a6b27-6wjrq" event={"ID":"81e3720d-b019-448f-9a08-49d5ccad4cb1","Type":"ContainerStarted","Data":"21edbe68b0343ad532e19ece14f7dfe8e6a0ce1206e2eb206135373873656dc5"} Oct 29 23:31:14.688359 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:14.688295 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-ed3bf2-7cbcbd6758-2gkwj"] Oct 29 23:31:14.725426 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:14.725383 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ed3bf2-dd855d495-fj85v"] Oct 29 23:31:14.851715 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:14.851659 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:31:14.961870 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:14.961837 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:31:15.098136 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:15.098089 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:31:15.240934 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:15.240904 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e26a6b27-6wjrq" Oct 29 23:31:15.401721 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:15.401675 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/81e3720d-b019-448f-9a08-49d5ccad4cb1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81e3720d-b019-448f-9a08-49d5ccad4cb1-maas-api-xks-ca\") pod \"81e3720d-b019-448f-9a08-49d5ccad4cb1\" (UID: \"81e3720d-b019-448f-9a08-49d5ccad4cb1\") " Oct 29 23:31:15.401928 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:15.401905 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/81e3720d-b019-448f-9a08-49d5ccad4cb1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81e3720d-b019-448f-9a08-49d5ccad4cb1-maas-api-openshift-ca\") pod \"81e3720d-b019-448f-9a08-49d5ccad4cb1\" (UID: \"81e3720d-b019-448f-9a08-49d5ccad4cb1\") " Oct 29 23:31:15.402001 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:15.401968 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81e3720d-b019-448f-9a08-49d5ccad4cb1-kube-api-access-s6rlv\" (UniqueName: \"kubernetes.io/projected/81e3720d-b019-448f-9a08-49d5ccad4cb1-kube-api-access-s6rlv\") pod \"81e3720d-b019-448f-9a08-49d5ccad4cb1\" (UID: \"81e3720d-b019-448f-9a08-49d5ccad4cb1\") " Oct 29 23:31:15.403837 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:15.403804 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81e3720d-b019-448f-9a08-49d5ccad4cb1-maas-api-openshift-ca" pod "81e3720d-b019-448f-9a08-49d5ccad4cb1" (UID: "81e3720d-b019-448f-9a08-49d5ccad4cb1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:31:15.405199 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:15.405169 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81e3720d-b019-448f-9a08-49d5ccad4cb1-kube-api-access-s6rlv" pod "81e3720d-b019-448f-9a08-49d5ccad4cb1" (UID: "81e3720d-b019-448f-9a08-49d5ccad4cb1"). InnerVolumeSpecName "kube-api-access-s6rlv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:31:15.405433 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:15.405412 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/81e3720d-b019-448f-9a08-49d5ccad4cb1-maas-api-xks-ca" pod "81e3720d-b019-448f-9a08-49d5ccad4cb1" (UID: "81e3720d-b019-448f-9a08-49d5ccad4cb1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:31:15.503877 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:15.503800 5057 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81e3720d-b019-448f-9a08-49d5ccad4cb1-maas-api-openshift-ca\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:31:15.503877 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:15.503833 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s6rlv\" (UniqueName: \"kubernetes.io/projected/81e3720d-b019-448f-9a08-49d5ccad4cb1-kube-api-access-s6rlv\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:31:15.503877 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:15.503844 5057 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81e3720d-b019-448f-9a08-49d5ccad4cb1-maas-api-xks-ca\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:31:16.075964 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:16.075829 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e26a6b27-6wjrq" Oct 29 23:31:16.075964 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:16.075853 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e26a6b27-6wjrq" event={"ID":"81e3720d-b019-448f-9a08-49d5ccad4cb1","Type":"ContainerDied","Data":"21edbe68b0343ad532e19ece14f7dfe8e6a0ce1206e2eb206135373873656dc5"} Oct 29 23:31:16.075964 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:16.075945 5057 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="21edbe68b0343ad532e19ece14f7dfe8e6a0ce1206e2eb206135373873656dc5" Oct 29 23:31:16.404784 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:16.404758 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b0c3c3-75d548567-9267h" Oct 29 23:31:16.408296 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:16.408271 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b0c3c3-5867dcd5b4-xs5ll" Oct 29 23:31:16.542122 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:16.542033 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93b0246c-b176-4daa-8090-6bb557b4fa84-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93b0246c-b176-4daa-8090-6bb557b4fa84-plugins-config-volume\") pod \"93b0246c-b176-4daa-8090-6bb557b4fa84\" (UID: \"93b0246c-b176-4daa-8090-6bb557b4fa84\") " Oct 29 23:31:16.542283 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:16.542155 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34be4000-c954-4c58-a304-1944cace0093-kube-api-access-l6fls\" (UniqueName: \"kubernetes.io/projected/34be4000-c954-4c58-a304-1944cace0093-kube-api-access-l6fls\") pod \"34be4000-c954-4c58-a304-1944cace0093\" (UID: \"34be4000-c954-4c58-a304-1944cace0093\") " Oct 29 23:31:16.542283 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:16.542224 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34be4000-c954-4c58-a304-1944cace0093-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34be4000-c954-4c58-a304-1944cace0093-plugins-config-volume\") pod \"34be4000-c954-4c58-a304-1944cace0093\" (UID: \"34be4000-c954-4c58-a304-1944cace0093\") " Oct 29 23:31:16.542428 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:16.542379 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93b0246c-b176-4daa-8090-6bb557b4fa84-kube-api-access-8pcch\" (UniqueName: \"kubernetes.io/projected/93b0246c-b176-4daa-8090-6bb557b4fa84-kube-api-access-8pcch\") pod \"93b0246c-b176-4daa-8090-6bb557b4fa84\" (UID: \"93b0246c-b176-4daa-8090-6bb557b4fa84\") " Oct 29 23:31:16.542482 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:16.542423 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93b0246c-b176-4daa-8090-6bb557b4fa84-plugins-config-volume" pod "93b0246c-b176-4daa-8090-6bb557b4fa84" (UID: "93b0246c-b176-4daa-8090-6bb557b4fa84"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:31:16.542697 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:16.542672 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34be4000-c954-4c58-a304-1944cace0093-plugins-config-volume" pod "34be4000-c954-4c58-a304-1944cace0093" (UID: "34be4000-c954-4c58-a304-1944cace0093"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:31:16.542958 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:16.542921 5057 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34be4000-c954-4c58-a304-1944cace0093-plugins-config-volume\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:31:16.542958 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:16.542946 5057 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93b0246c-b176-4daa-8090-6bb557b4fa84-plugins-config-volume\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:31:16.544496 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:16.544470 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93b0246c-b176-4daa-8090-6bb557b4fa84-kube-api-access-8pcch" pod "93b0246c-b176-4daa-8090-6bb557b4fa84" (UID: "93b0246c-b176-4daa-8090-6bb557b4fa84"). InnerVolumeSpecName "kube-api-access-8pcch". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:31:16.544611 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:16.544520 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34be4000-c954-4c58-a304-1944cace0093-kube-api-access-l6fls" pod "34be4000-c954-4c58-a304-1944cace0093" (UID: "34be4000-c954-4c58-a304-1944cace0093"). InnerVolumeSpecName "kube-api-access-l6fls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:31:16.643822 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:16.643779 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l6fls\" (UniqueName: \"kubernetes.io/projected/34be4000-c954-4c58-a304-1944cace0093-kube-api-access-l6fls\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:31:16.643822 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:16.643814 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8pcch\" (UniqueName: \"kubernetes.io/projected/93b0246c-b176-4daa-8090-6bb557b4fa84-kube-api-access-8pcch\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:31:16.780544 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:16.780501 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:31:17.087603 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:17.087556 5057 generic.go:356] "Generic (PLEG): container finished" podID="34be4000-c954-4c58-a304-1944cace0093" containerID="6794564be13c7916c1a2d4e55916bade68a5ef4d697eed35e8e76c4545b2b8cc" exitCode=0 Oct 29 23:31:17.088164 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:17.087614 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b0c3c3-75d548567-9267h" event={"ID":"34be4000-c954-4c58-a304-1944cace0093","Type":"ContainerDied","Data":"6794564be13c7916c1a2d4e55916bade68a5ef4d697eed35e8e76c4545b2b8cc"} Oct 29 23:31:17.088164 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:17.087658 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b0c3c3-75d548567-9267h" Oct 29 23:31:17.088164 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:17.087734 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b0c3c3-75d548567-9267h" event={"ID":"34be4000-c954-4c58-a304-1944cace0093","Type":"ContainerDied","Data":"193fafe5a3daf3825a3bbe1670f6c73600763fa990f199fe7c9b36fcfe1918bb"} Oct 29 23:31:17.088164 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:17.087768 5057 scope.go:122] "RemoveContainer" containerID="6794564be13c7916c1a2d4e55916bade68a5ef4d697eed35e8e76c4545b2b8cc" Oct 29 23:31:17.089561 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:17.089539 5057 generic.go:356] "Generic (PLEG): container finished" podID="93b0246c-b176-4daa-8090-6bb557b4fa84" containerID="c4d8ab74870b6d6166812a9f91d9a3a42f00578efa4d9806c00f59c3a09fcbc5" exitCode=0 Oct 29 23:31:17.089717 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:17.089699 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b0c3c3-5867dcd5b4-xs5ll" event={"ID":"93b0246c-b176-4daa-8090-6bb557b4fa84","Type":"ContainerDied","Data":"c4d8ab74870b6d6166812a9f91d9a3a42f00578efa4d9806c00f59c3a09fcbc5"} Oct 29 23:31:17.089794 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:17.089744 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b0c3c3-5867dcd5b4-xs5ll" event={"ID":"93b0246c-b176-4daa-8090-6bb557b4fa84","Type":"ContainerDied","Data":"695f4fa4c9287bcb836c7f4d1ad500efb1806738ed52eecfdd492b7f2b4f1416"} Oct 29 23:31:17.089907 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:17.089890 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b0c3c3-5867dcd5b4-xs5ll" Oct 29 23:31:17.103028 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:17.103005 5057 scope.go:122] "RemoveContainer" containerID="6794564be13c7916c1a2d4e55916bade68a5ef4d697eed35e8e76c4545b2b8cc" Oct 29 23:31:17.103381 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:31:17.103345 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6794564be13c7916c1a2d4e55916bade68a5ef4d697eed35e8e76c4545b2b8cc\": container with ID starting with 6794564be13c7916c1a2d4e55916bade68a5ef4d697eed35e8e76c4545b2b8cc not found: ID does not exist" containerID="6794564be13c7916c1a2d4e55916bade68a5ef4d697eed35e8e76c4545b2b8cc" Oct 29 23:31:17.103482 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:17.103388 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6794564be13c7916c1a2d4e55916bade68a5ef4d697eed35e8e76c4545b2b8cc"} err="failed to get container status \"6794564be13c7916c1a2d4e55916bade68a5ef4d697eed35e8e76c4545b2b8cc\": rpc error: code = NotFound desc = could not find container \"6794564be13c7916c1a2d4e55916bade68a5ef4d697eed35e8e76c4545b2b8cc\": container with ID starting with 6794564be13c7916c1a2d4e55916bade68a5ef4d697eed35e8e76c4545b2b8cc not found: ID does not exist" Oct 29 23:31:17.103482 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:17.103411 5057 scope.go:122] "RemoveContainer" containerID="c4d8ab74870b6d6166812a9f91d9a3a42f00578efa4d9806c00f59c3a09fcbc5" Oct 29 23:31:17.114471 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:17.114447 5057 scope.go:122] "RemoveContainer" containerID="c4d8ab74870b6d6166812a9f91d9a3a42f00578efa4d9806c00f59c3a09fcbc5" Oct 29 23:31:17.114803 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:31:17.114774 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4d8ab74870b6d6166812a9f91d9a3a42f00578efa4d9806c00f59c3a09fcbc5\": container with ID starting with c4d8ab74870b6d6166812a9f91d9a3a42f00578efa4d9806c00f59c3a09fcbc5 not found: ID does not exist" containerID="c4d8ab74870b6d6166812a9f91d9a3a42f00578efa4d9806c00f59c3a09fcbc5" Oct 29 23:31:17.114896 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:17.114811 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4d8ab74870b6d6166812a9f91d9a3a42f00578efa4d9806c00f59c3a09fcbc5"} err="failed to get container status \"c4d8ab74870b6d6166812a9f91d9a3a42f00578efa4d9806c00f59c3a09fcbc5\": rpc error: code = NotFound desc = could not find container \"c4d8ab74870b6d6166812a9f91d9a3a42f00578efa4d9806c00f59c3a09fcbc5\": container with ID starting with c4d8ab74870b6d6166812a9f91d9a3a42f00578efa4d9806c00f59c3a09fcbc5 not found: ID does not exist" Oct 29 23:31:17.156423 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:17.156391 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-b0c3c3-5867dcd5b4-xs5ll"] Oct 29 23:31:17.168438 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:17.168403 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-b0c3c3-5867dcd5b4-xs5ll"] Oct 29 23:31:17.202439 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:17.202407 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b0c3c3-75d548567-9267h"] Oct 29 23:31:17.203854 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:17.203828 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b0c3c3-75d548567-9267h"] Oct 29 23:31:17.331981 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:17.331944 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34be4000-c954-4c58-a304-1944cace0093" path="/var/lib/kubelet/pods/34be4000-c954-4c58-a304-1944cace0093/volumes" Oct 29 23:31:17.332572 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:17.332554 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93b0246c-b176-4daa-8090-6bb557b4fa84" path="/var/lib/kubelet/pods/93b0246c-b176-4daa-8090-6bb557b4fa84/volumes" Oct 29 23:31:18.348161 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:18.348126 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:31:18.803639 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:18.803601 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b1f808d-jv69s"] Oct 29 23:31:18.804740 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:18.804717 5057 cpu_manager.go:414] "containerMap: removing container" podUID="81e3720d-b019-448f-9a08-49d5ccad4cb1" containerName="revoke-keys" Oct 29 23:31:18.804812 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:18.804744 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81e3720d-b019-448f-9a08-49d5ccad4cb1" containerName="revoke-keys" Oct 29 23:31:18.804812 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:18.804783 5057 cpu_manager.go:414] "containerMap: removing container" podUID="34be4000-c954-4c58-a304-1944cace0093" containerName="payload-pre-processing" Oct 29 23:31:18.804812 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:18.804791 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34be4000-c954-4c58-a304-1944cace0093" containerName="payload-pre-processing" Oct 29 23:31:18.804916 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:18.804825 5057 cpu_manager.go:414] "containerMap: removing container" podUID="93b0246c-b176-4daa-8090-6bb557b4fa84" containerName="payload-processing" Oct 29 23:31:18.804916 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:18.804833 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93b0246c-b176-4daa-8090-6bb557b4fa84" containerName="payload-processing" Oct 29 23:31:18.805045 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:18.805033 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="81e3720d-b019-448f-9a08-49d5ccad4cb1" containerName="revoke-keys" Oct 29 23:31:18.805084 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:18.805054 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="34be4000-c954-4c58-a304-1944cace0093" containerName="payload-pre-processing" Oct 29 23:31:18.805084 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:18.805067 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="93b0246c-b176-4daa-8090-6bb557b4fa84" containerName="payload-processing" Oct 29 23:31:18.810744 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:18.810722 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b1f808d-jv69s" Oct 29 23:31:18.827045 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:18.827021 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b1f808d-jv69s"] Oct 29 23:31:18.833140 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:18.833031 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:31:18.833140 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:18.833118 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dmx57\"" Oct 29 23:31:18.876729 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:18.876695 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d60d011e-d215-4b9d-ae66-62cdf573a9ed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7b1f808d-jv69s\" (UID: \"d60d011e-d215-4b9d-ae66-62cdf573a9ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b1f808d-jv69s" Oct 29 23:31:18.877020 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:18.876994 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d60d011e-d215-4b9d-ae66-62cdf573a9ed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7b1f808d-jv69s\" (UID: \"d60d011e-d215-4b9d-ae66-62cdf573a9ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b1f808d-jv69s" Oct 29 23:31:18.877148 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:18.877127 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxt4g\" (UniqueName: \"kubernetes.io/projected/d60d011e-d215-4b9d-ae66-62cdf573a9ed-kube-api-access-bxt4g\") pod \"maas-api-revoke-sub-7b1f808d-jv69s\" (UID: \"d60d011e-d215-4b9d-ae66-62cdf573a9ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b1f808d-jv69s" Oct 29 23:31:18.979022 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:18.978988 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d60d011e-d215-4b9d-ae66-62cdf573a9ed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7b1f808d-jv69s\" (UID: \"d60d011e-d215-4b9d-ae66-62cdf573a9ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b1f808d-jv69s" Oct 29 23:31:18.979271 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:18.979252 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d60d011e-d215-4b9d-ae66-62cdf573a9ed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7b1f808d-jv69s\" (UID: \"d60d011e-d215-4b9d-ae66-62cdf573a9ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b1f808d-jv69s" Oct 29 23:31:18.979389 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:18.979373 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxt4g\" (UniqueName: \"kubernetes.io/projected/d60d011e-d215-4b9d-ae66-62cdf573a9ed-kube-api-access-bxt4g\") pod \"maas-api-revoke-sub-7b1f808d-jv69s\" (UID: \"d60d011e-d215-4b9d-ae66-62cdf573a9ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b1f808d-jv69s" Oct 29 23:31:18.980085 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:18.980055 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d60d011e-d215-4b9d-ae66-62cdf573a9ed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7b1f808d-jv69s\" (UID: \"d60d011e-d215-4b9d-ae66-62cdf573a9ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b1f808d-jv69s" Oct 29 23:31:18.981400 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:18.981379 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d60d011e-d215-4b9d-ae66-62cdf573a9ed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7b1f808d-jv69s\" (UID: \"d60d011e-d215-4b9d-ae66-62cdf573a9ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b1f808d-jv69s" Oct 29 23:31:19.002784 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:19.002758 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxt4g\" (UniqueName: \"kubernetes.io/projected/d60d011e-d215-4b9d-ae66-62cdf573a9ed-kube-api-access-bxt4g\") pod \"maas-api-revoke-sub-7b1f808d-jv69s\" (UID: \"d60d011e-d215-4b9d-ae66-62cdf573a9ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b1f808d-jv69s" Oct 29 23:31:19.029441 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:31:19.029410 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d2dnn" containerName="extractor" Oct 29 23:31:19.129258 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:19.129187 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b1f808d-jv69s" Oct 29 23:31:19.435613 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:19.435580 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b1f808d-jv69s"] Oct 29 23:31:19.438528 ip-10-0-140-74 kubenswrapper[5057]: W1029 23:31:19.438501 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd60d011e_d215_4b9d_ae66_62cdf573a9ed.slice/crio-c1d3a640062cb04a665b5ac13f569d139553a199983e465f28d0c7bffac9df8a WatchSource:0}: Error finding container c1d3a640062cb04a665b5ac13f569d139553a199983e465f28d0c7bffac9df8a: Status 404 returned error can't find the container with id c1d3a640062cb04a665b5ac13f569d139553a199983e465f28d0c7bffac9df8a Oct 29 23:31:19.689421 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:19.689316 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ed3bf2-7cbcbd6758-2gkwj" podUID="56962a7e-7f67-4151-b04a-c4161d2de549" containerName="payload-processing" containerID="cri-o://c1474dda4e4d0b5a231c642a7efa78b6c0bc880aec7432a2396b95379b37ce36" gracePeriod=25 Oct 29 23:31:19.701462 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:19.701423 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:31:19.726227 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:19.726191 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ed3bf2-dd855d495-fj85v" podUID="695c8042-ace4-420f-bdaf-11ca7e09ef5f" containerName="payload-pre-processing" containerID="cri-o://d21b8bfa1035c412881c363719348641f520a2be385a9318dfad63b541eb798d" gracePeriod=25 Oct 29 23:31:20.122569 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:20.122533 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b1f808d-jv69s" event={"ID":"d60d011e-d215-4b9d-ae66-62cdf573a9ed","Type":"ContainerStarted","Data":"e6af21d38eb11d038b0eb2949c389294cb798b91c167db1c914e643ddb5eb28a"} Oct 29 23:31:20.122783 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:20.122592 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b1f808d-jv69s" event={"ID":"d60d011e-d215-4b9d-ae66-62cdf573a9ed","Type":"ContainerStarted","Data":"c1d3a640062cb04a665b5ac13f569d139553a199983e465f28d0c7bffac9df8a"} Oct 29 23:31:20.164985 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:20.164933 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b1f808d-jv69s" podStartSLOduration=2.164919488 podStartE2EDuration="2.164919488s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:31:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:31:20.162212344 +0000 UTC m=+3460.945425771" watchObservedRunningTime="2026-10-29 23:31:20.164919488 +0000 UTC m=+3460.948132911" Oct 29 23:31:20.251522 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:20.251465 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:31:20.361736 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:20.361693 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:31:20.472280 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:20.472231 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:31:20.866434 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:20.866289 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:31:21.132460 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:21.132377 5057 generic.go:356] "Generic (PLEG): container finished" podID="d60d011e-d215-4b9d-ae66-62cdf573a9ed" containerID="e6af21d38eb11d038b0eb2949c389294cb798b91c167db1c914e643ddb5eb28a" exitCode=0 Oct 29 23:31:21.132610 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:21.132459 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b1f808d-jv69s" event={"ID":"d60d011e-d215-4b9d-ae66-62cdf573a9ed","Type":"ContainerDied","Data":"e6af21d38eb11d038b0eb2949c389294cb798b91c167db1c914e643ddb5eb28a"} Oct 29 23:31:22.094670 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:22.094627 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:31:22.349259 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:22.349203 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b1f808d-jv69s" Oct 29 23:31:22.440383 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:22.440346 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d60d011e-d215-4b9d-ae66-62cdf573a9ed-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d60d011e-d215-4b9d-ae66-62cdf573a9ed-maas-api-openshift-ca\") pod \"d60d011e-d215-4b9d-ae66-62cdf573a9ed\" (UID: \"d60d011e-d215-4b9d-ae66-62cdf573a9ed\") " Oct 29 23:31:22.440558 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:22.440531 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d60d011e-d215-4b9d-ae66-62cdf573a9ed-kube-api-access-bxt4g\" (UniqueName: \"kubernetes.io/projected/d60d011e-d215-4b9d-ae66-62cdf573a9ed-kube-api-access-bxt4g\") pod \"d60d011e-d215-4b9d-ae66-62cdf573a9ed\" (UID: \"d60d011e-d215-4b9d-ae66-62cdf573a9ed\") " Oct 29 23:31:22.440632 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:22.440567 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d60d011e-d215-4b9d-ae66-62cdf573a9ed-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d60d011e-d215-4b9d-ae66-62cdf573a9ed-maas-api-xks-ca\") pod \"d60d011e-d215-4b9d-ae66-62cdf573a9ed\" (UID: \"d60d011e-d215-4b9d-ae66-62cdf573a9ed\") " Oct 29 23:31:22.440730 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:22.440699 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d60d011e-d215-4b9d-ae66-62cdf573a9ed-maas-api-openshift-ca" pod "d60d011e-d215-4b9d-ae66-62cdf573a9ed" (UID: "d60d011e-d215-4b9d-ae66-62cdf573a9ed"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:31:22.441454 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:22.441342 5057 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d60d011e-d215-4b9d-ae66-62cdf573a9ed-maas-api-openshift-ca\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:31:22.442905 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:22.442876 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d60d011e-d215-4b9d-ae66-62cdf573a9ed-kube-api-access-bxt4g" pod "d60d011e-d215-4b9d-ae66-62cdf573a9ed" (UID: "d60d011e-d215-4b9d-ae66-62cdf573a9ed"). InnerVolumeSpecName "kube-api-access-bxt4g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:31:22.444185 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:22.444165 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d60d011e-d215-4b9d-ae66-62cdf573a9ed-maas-api-xks-ca" pod "d60d011e-d215-4b9d-ae66-62cdf573a9ed" (UID: "d60d011e-d215-4b9d-ae66-62cdf573a9ed"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:31:22.542817 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:22.542785 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bxt4g\" (UniqueName: \"kubernetes.io/projected/d60d011e-d215-4b9d-ae66-62cdf573a9ed-kube-api-access-bxt4g\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:31:22.542817 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:22.542814 5057 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d60d011e-d215-4b9d-ae66-62cdf573a9ed-maas-api-xks-ca\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:31:22.565418 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:22.565388 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ed3bf2-7cbcbd6758-2gkwj" podUID="56962a7e-7f67-4151-b04a-c4161d2de549" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:31:22.565704 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:22.565682 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ed3bf2-dd855d495-fj85v" podUID="695c8042-ace4-420f-bdaf-11ca7e09ef5f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:31:23.039525 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:23.039482 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d7861bca-cz6gc"] Oct 29 23:31:23.043020 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:23.042994 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d7861bca-cz6gc"] Oct 29 23:31:23.154885 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:23.154856 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b1f808d-jv69s" Oct 29 23:31:23.154885 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:23.154874 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b1f808d-jv69s" event={"ID":"d60d011e-d215-4b9d-ae66-62cdf573a9ed","Type":"ContainerDied","Data":"c1d3a640062cb04a665b5ac13f569d139553a199983e465f28d0c7bffac9df8a"} Oct 29 23:31:23.155312 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:23.154923 5057 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c1d3a640062cb04a665b5ac13f569d139553a199983e465f28d0c7bffac9df8a" Oct 29 23:31:23.332168 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:23.332093 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d884577-666d-418f-91be-0d0744c3bf09" path="/var/lib/kubelet/pods/9d884577-666d-418f-91be-0d0744c3bf09/volumes" Oct 29 23:31:23.412017 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:23.411976 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-59eefa/rate-test-model-59eefa-kserve-5c6dcf6849-6wq8t"] Oct 29 23:31:23.451397 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:23.451364 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:31:23.724910 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:23.724873 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:31:24.652425 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:24.652381 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:31:24.758837 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:24.758799 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:31:25.264300 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:25.264260 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:31:26.806462 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:26.806420 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:31:27.615419 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:27.615387 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:31:27.725351 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:27.724736 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:31:27.976139 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:27.976101 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:31:28.413386 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:28.413267 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:31:28.523521 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:28.523484 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:31:29.233729 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:29.233695 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-94e8042f-pxbc5"] Oct 29 23:31:29.234534 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:29.234515 5057 cpu_manager.go:414] "containerMap: removing container" podUID="d60d011e-d215-4b9d-ae66-62cdf573a9ed" containerName="revoke-keys" Oct 29 23:31:29.234534 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:29.234533 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d60d011e-d215-4b9d-ae66-62cdf573a9ed" containerName="revoke-keys" Oct 29 23:31:29.234683 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:29.234672 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="d60d011e-d215-4b9d-ae66-62cdf573a9ed" containerName="revoke-keys" Oct 29 23:31:29.239662 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:29.239634 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94e8042f-pxbc5" Oct 29 23:31:29.262629 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:29.262605 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-94e8042f-pxbc5"] Oct 29 23:31:29.270160 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:29.270137 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:31:29.270309 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:29.270236 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dmx57\"" Oct 29 23:31:29.339691 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:29.339662 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-px9zc\" (UniqueName: \"kubernetes.io/projected/189db1b7-a3e4-4dd9-90aa-811ec21845e3-kube-api-access-px9zc\") pod \"maas-api-revoke-sub-94e8042f-pxbc5\" (UID: \"189db1b7-a3e4-4dd9-90aa-811ec21845e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94e8042f-pxbc5" Oct 29 23:31:29.339879 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:29.339861 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/189db1b7-a3e4-4dd9-90aa-811ec21845e3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-94e8042f-pxbc5\" (UID: \"189db1b7-a3e4-4dd9-90aa-811ec21845e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94e8042f-pxbc5" Oct 29 23:31:29.339940 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:29.339901 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/189db1b7-a3e4-4dd9-90aa-811ec21845e3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-94e8042f-pxbc5\" (UID: \"189db1b7-a3e4-4dd9-90aa-811ec21845e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94e8042f-pxbc5" Oct 29 23:31:29.440767 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:29.440732 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/189db1b7-a3e4-4dd9-90aa-811ec21845e3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-94e8042f-pxbc5\" (UID: \"189db1b7-a3e4-4dd9-90aa-811ec21845e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94e8042f-pxbc5" Oct 29 23:31:29.440942 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:29.440774 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/189db1b7-a3e4-4dd9-90aa-811ec21845e3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-94e8042f-pxbc5\" (UID: \"189db1b7-a3e4-4dd9-90aa-811ec21845e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94e8042f-pxbc5" Oct 29 23:31:29.440942 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:29.440817 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-px9zc\" (UniqueName: \"kubernetes.io/projected/189db1b7-a3e4-4dd9-90aa-811ec21845e3-kube-api-access-px9zc\") pod \"maas-api-revoke-sub-94e8042f-pxbc5\" (UID: \"189db1b7-a3e4-4dd9-90aa-811ec21845e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94e8042f-pxbc5" Oct 29 23:31:29.441517 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:29.441490 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/189db1b7-a3e4-4dd9-90aa-811ec21845e3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-94e8042f-pxbc5\" (UID: \"189db1b7-a3e4-4dd9-90aa-811ec21845e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94e8042f-pxbc5" Oct 29 23:31:29.443212 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:29.443184 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/189db1b7-a3e4-4dd9-90aa-811ec21845e3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-94e8042f-pxbc5\" (UID: \"189db1b7-a3e4-4dd9-90aa-811ec21845e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94e8042f-pxbc5" Oct 29 23:31:29.463757 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:29.463730 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-px9zc\" (UniqueName: \"kubernetes.io/projected/189db1b7-a3e4-4dd9-90aa-811ec21845e3-kube-api-access-px9zc\") pod \"maas-api-revoke-sub-94e8042f-pxbc5\" (UID: \"189db1b7-a3e4-4dd9-90aa-811ec21845e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94e8042f-pxbc5" Oct 29 23:31:29.562550 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:29.562475 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94e8042f-pxbc5" Oct 29 23:31:29.885215 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:29.885187 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-94e8042f-pxbc5"] Oct 29 23:31:29.888640 ip-10-0-140-74 kubenswrapper[5057]: W1029 23:31:29.888604 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod189db1b7_a3e4_4dd9_90aa_811ec21845e3.slice/crio-77741f7fd53b09a50910b5e321ad6590a9b0bee021807e51a4478bcd07e20c98 WatchSource:0}: Error finding container 77741f7fd53b09a50910b5e321ad6590a9b0bee021807e51a4478bcd07e20c98: Status 404 returned error can't find the container with id 77741f7fd53b09a50910b5e321ad6590a9b0bee021807e51a4478bcd07e20c98 Oct 29 23:31:30.212088 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:30.212054 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94e8042f-pxbc5" event={"ID":"189db1b7-a3e4-4dd9-90aa-811ec21845e3","Type":"ContainerStarted","Data":"77741f7fd53b09a50910b5e321ad6590a9b0bee021807e51a4478bcd07e20c98"} Oct 29 23:31:30.214943 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:30.214907 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:31:31.222161 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:31.222120 5057 generic.go:356] "Generic (PLEG): container finished" podID="189db1b7-a3e4-4dd9-90aa-811ec21845e3" containerID="2f98e7da52759dd31a1c8e66affd12bf5706abf096e658c54caacfb19a3a82f3" exitCode=0 Oct 29 23:31:31.222602 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:31.222203 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94e8042f-pxbc5" event={"ID":"189db1b7-a3e4-4dd9-90aa-811ec21845e3","Type":"ContainerDied","Data":"2f98e7da52759dd31a1c8e66affd12bf5706abf096e658c54caacfb19a3a82f3"} Oct 29 23:31:31.922820 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:31.922779 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:31:32.088462 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:32.088414 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-e170ed/rate-test-model-e170ed-kserve-544667c877-hc2sg"] Oct 29 23:31:32.135786 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:32.135750 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:31:32.435433 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:32.435399 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94e8042f-pxbc5" Oct 29 23:31:32.565560 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:32.565392 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ed3bf2-dd855d495-fj85v" podUID="695c8042-ace4-420f-bdaf-11ca7e09ef5f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:31:32.566147 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:32.565782 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ed3bf2-7cbcbd6758-2gkwj" podUID="56962a7e-7f67-4151-b04a-c4161d2de549" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:31:32.602180 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:32.602144 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/189db1b7-a3e4-4dd9-90aa-811ec21845e3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/189db1b7-a3e4-4dd9-90aa-811ec21845e3-maas-api-xks-ca\") pod \"189db1b7-a3e4-4dd9-90aa-811ec21845e3\" (UID: \"189db1b7-a3e4-4dd9-90aa-811ec21845e3\") " Oct 29 23:31:32.602375 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:32.602342 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/189db1b7-a3e4-4dd9-90aa-811ec21845e3-kube-api-access-px9zc\" (UniqueName: \"kubernetes.io/projected/189db1b7-a3e4-4dd9-90aa-811ec21845e3-kube-api-access-px9zc\") pod \"189db1b7-a3e4-4dd9-90aa-811ec21845e3\" (UID: \"189db1b7-a3e4-4dd9-90aa-811ec21845e3\") " Oct 29 23:31:32.602425 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:32.602401 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/189db1b7-a3e4-4dd9-90aa-811ec21845e3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/189db1b7-a3e4-4dd9-90aa-811ec21845e3-maas-api-openshift-ca\") pod \"189db1b7-a3e4-4dd9-90aa-811ec21845e3\" (UID: \"189db1b7-a3e4-4dd9-90aa-811ec21845e3\") " Oct 29 23:31:32.602857 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:32.602822 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/189db1b7-a3e4-4dd9-90aa-811ec21845e3-maas-api-openshift-ca" pod "189db1b7-a3e4-4dd9-90aa-811ec21845e3" (UID: "189db1b7-a3e4-4dd9-90aa-811ec21845e3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:31:32.603079 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:32.603044 5057 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/189db1b7-a3e4-4dd9-90aa-811ec21845e3-maas-api-openshift-ca\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:31:32.604545 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:32.604518 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/189db1b7-a3e4-4dd9-90aa-811ec21845e3-maas-api-xks-ca" pod "189db1b7-a3e4-4dd9-90aa-811ec21845e3" (UID: "189db1b7-a3e4-4dd9-90aa-811ec21845e3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:31:32.604638 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:32.604595 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/189db1b7-a3e4-4dd9-90aa-811ec21845e3-kube-api-access-px9zc" pod "189db1b7-a3e4-4dd9-90aa-811ec21845e3" (UID: "189db1b7-a3e4-4dd9-90aa-811ec21845e3"). InnerVolumeSpecName "kube-api-access-px9zc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:31:32.704927 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:32.704888 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-px9zc\" (UniqueName: \"kubernetes.io/projected/189db1b7-a3e4-4dd9-90aa-811ec21845e3-kube-api-access-px9zc\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:31:32.705171 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:32.705154 5057 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/189db1b7-a3e4-4dd9-90aa-811ec21845e3-maas-api-xks-ca\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:31:33.251458 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:33.251422 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94e8042f-pxbc5" event={"ID":"189db1b7-a3e4-4dd9-90aa-811ec21845e3","Type":"ContainerDied","Data":"77741f7fd53b09a50910b5e321ad6590a9b0bee021807e51a4478bcd07e20c98"} Oct 29 23:31:33.251674 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:33.251482 5057 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="77741f7fd53b09a50910b5e321ad6590a9b0bee021807e51a4478bcd07e20c98" Oct 29 23:31:33.251674 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:33.251549 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94e8042f-pxbc5" Oct 29 23:31:33.374918 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:33.374882 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-575d59/distinct-2-w2-575d59-kserve-d4565b76c-hdgbf"] Oct 29 23:31:33.491208 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:33.491170 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:31:33.695228 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:33.695139 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:31:35.406854 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:35.406819 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:31:36.764553 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:36.764516 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:31:36.906435 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:36.906393 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:31:38.450355 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:38.450304 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w1-59eefa/rate-test-model-59eefa-kserve-5c6dcf6849-6wq8t" podUID="4ae48191-676a-43fa-9ffd-a872f474e461" containerName="main" containerID="cri-o://59f1fedccc4f40fb83438854918fcfcb89a24a8243973e1038659a602782a9e1" gracePeriod=45 Oct 29 23:31:38.492095 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:38.492047 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:31:38.749524 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:38.749499 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-59eefa/rate-test-model-59eefa-kserve-5c6dcf6849-6wq8t" Oct 29 23:31:38.786436 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:38.786407 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ae48191-676a-43fa-9ffd-a872f474e461-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ae48191-676a-43fa-9ffd-a872f474e461-model-cache\") pod \"4ae48191-676a-43fa-9ffd-a872f474e461\" (UID: \"4ae48191-676a-43fa-9ffd-a872f474e461\") " Oct 29 23:31:38.786591 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:38.786497 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4ae48191-676a-43fa-9ffd-a872f474e461-tls-certs\" (UniqueName: \"kubernetes.io/secret/4ae48191-676a-43fa-9ffd-a872f474e461-tls-certs\") pod \"4ae48191-676a-43fa-9ffd-a872f474e461\" (UID: \"4ae48191-676a-43fa-9ffd-a872f474e461\") " Oct 29 23:31:38.786591 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:38.786524 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ae48191-676a-43fa-9ffd-a872f474e461-home\" (UniqueName: \"kubernetes.io/empty-dir/4ae48191-676a-43fa-9ffd-a872f474e461-home\") pod \"4ae48191-676a-43fa-9ffd-a872f474e461\" (UID: \"4ae48191-676a-43fa-9ffd-a872f474e461\") " Oct 29 23:31:38.786591 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:38.786546 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ae48191-676a-43fa-9ffd-a872f474e461-kube-api-access-jd6qt\" (UniqueName: \"kubernetes.io/projected/4ae48191-676a-43fa-9ffd-a872f474e461-kube-api-access-jd6qt\") pod \"4ae48191-676a-43fa-9ffd-a872f474e461\" (UID: \"4ae48191-676a-43fa-9ffd-a872f474e461\") " Oct 29 23:31:38.786824 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:38.786603 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ae48191-676a-43fa-9ffd-a872f474e461-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ae48191-676a-43fa-9ffd-a872f474e461-dshm\") pod \"4ae48191-676a-43fa-9ffd-a872f474e461\" (UID: \"4ae48191-676a-43fa-9ffd-a872f474e461\") " Oct 29 23:31:38.786824 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:38.786711 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ae48191-676a-43fa-9ffd-a872f474e461-model-cache" pod "4ae48191-676a-43fa-9ffd-a872f474e461" (UID: "4ae48191-676a-43fa-9ffd-a872f474e461"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:31:38.786824 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:38.786797 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ae48191-676a-43fa-9ffd-a872f474e461-home" pod "4ae48191-676a-43fa-9ffd-a872f474e461" (UID: "4ae48191-676a-43fa-9ffd-a872f474e461"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:31:38.786824 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:38.786809 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ae48191-676a-43fa-9ffd-a872f474e461-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ae48191-676a-43fa-9ffd-a872f474e461-tmp-dir\") pod \"4ae48191-676a-43fa-9ffd-a872f474e461\" (UID: \"4ae48191-676a-43fa-9ffd-a872f474e461\") " Oct 29 23:31:38.787105 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:38.787070 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ae48191-676a-43fa-9ffd-a872f474e461-tmp-dir" pod "4ae48191-676a-43fa-9ffd-a872f474e461" (UID: "4ae48191-676a-43fa-9ffd-a872f474e461"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:31:38.787612 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:38.787590 5057 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ae48191-676a-43fa-9ffd-a872f474e461-model-cache\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:31:38.787729 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:38.787618 5057 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ae48191-676a-43fa-9ffd-a872f474e461-home\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:31:38.787729 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:38.787632 5057 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ae48191-676a-43fa-9ffd-a872f474e461-tmp-dir\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:31:38.788954 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:38.788932 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ae48191-676a-43fa-9ffd-a872f474e461-tls-certs" pod "4ae48191-676a-43fa-9ffd-a872f474e461" (UID: "4ae48191-676a-43fa-9ffd-a872f474e461"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:31:38.789061 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:38.789024 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ae48191-676a-43fa-9ffd-a872f474e461-kube-api-access-jd6qt" pod "4ae48191-676a-43fa-9ffd-a872f474e461" (UID: "4ae48191-676a-43fa-9ffd-a872f474e461"). InnerVolumeSpecName "kube-api-access-jd6qt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:31:38.789129 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:38.789064 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ae48191-676a-43fa-9ffd-a872f474e461-dshm" pod "4ae48191-676a-43fa-9ffd-a872f474e461" (UID: "4ae48191-676a-43fa-9ffd-a872f474e461"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:31:38.889030 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:38.888990 5057 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ae48191-676a-43fa-9ffd-a872f474e461-tls-certs\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:31:38.889030 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:38.889019 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jd6qt\" (UniqueName: \"kubernetes.io/projected/4ae48191-676a-43fa-9ffd-a872f474e461-kube-api-access-jd6qt\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:31:38.889030 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:38.889029 5057 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ae48191-676a-43fa-9ffd-a872f474e461-dshm\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:31:39.088263 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:39.088177 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-575d59-65c58457dd-c4d2z"] Oct 29 23:31:39.305671 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:39.305622 5057 generic.go:356] "Generic (PLEG): container finished" podID="4ae48191-676a-43fa-9ffd-a872f474e461" containerID="59f1fedccc4f40fb83438854918fcfcb89a24a8243973e1038659a602782a9e1" exitCode=0 Oct 29 23:31:39.305865 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:39.305704 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-59eefa/rate-test-model-59eefa-kserve-5c6dcf6849-6wq8t" event={"ID":"4ae48191-676a-43fa-9ffd-a872f474e461","Type":"ContainerDied","Data":"59f1fedccc4f40fb83438854918fcfcb89a24a8243973e1038659a602782a9e1"} Oct 29 23:31:39.305865 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:39.305739 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-59eefa/rate-test-model-59eefa-kserve-5c6dcf6849-6wq8t" event={"ID":"4ae48191-676a-43fa-9ffd-a872f474e461","Type":"ContainerDied","Data":"c8387bd6be7f0c8c65dd8efacb609fa6350c51d2f77ce239436fd4c46b3cfbf5"} Oct 29 23:31:39.305865 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:39.305761 5057 scope.go:122] "RemoveContainer" containerID="59f1fedccc4f40fb83438854918fcfcb89a24a8243973e1038659a602782a9e1" Oct 29 23:31:39.306042 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:39.305918 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-59eefa/rate-test-model-59eefa-kserve-5c6dcf6849-6wq8t" Oct 29 23:31:39.312464 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:39.312431 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:31:39.344797 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:39.332090 5057 scope.go:122] "RemoveContainer" containerID="59f1fedccc4f40fb83438854918fcfcb89a24a8243973e1038659a602782a9e1" Oct 29 23:31:39.344797 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:31:39.333630 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59f1fedccc4f40fb83438854918fcfcb89a24a8243973e1038659a602782a9e1\": container with ID starting with 59f1fedccc4f40fb83438854918fcfcb89a24a8243973e1038659a602782a9e1 not found: ID does not exist" containerID="59f1fedccc4f40fb83438854918fcfcb89a24a8243973e1038659a602782a9e1" Oct 29 23:31:39.344797 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:39.333696 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59f1fedccc4f40fb83438854918fcfcb89a24a8243973e1038659a602782a9e1"} err="failed to get container status \"59f1fedccc4f40fb83438854918fcfcb89a24a8243973e1038659a602782a9e1\": rpc error: code = NotFound desc = could not find container \"59f1fedccc4f40fb83438854918fcfcb89a24a8243973e1038659a602782a9e1\": container with ID starting with 59f1fedccc4f40fb83438854918fcfcb89a24a8243973e1038659a602782a9e1 not found: ID does not exist" Oct 29 23:31:39.369549 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:39.369524 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-59eefa/rate-test-model-59eefa-kserve-5c6dcf6849-6wq8t"] Oct 29 23:31:39.373824 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:39.373800 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w1-59eefa/rate-test-model-59eefa-kserve-5c6dcf6849-6wq8t"] Oct 29 23:31:39.428352 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:39.427995 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:31:39.787136 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:39.787098 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v2tfm"] Oct 29 23:31:39.869735 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:39.869699 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5babc19a-cjh8n"] Oct 29 23:31:39.870840 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:39.870816 5057 cpu_manager.go:414] "containerMap: removing container" podUID="4ae48191-676a-43fa-9ffd-a872f474e461" containerName="main" Oct 29 23:31:39.870840 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:39.870840 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ae48191-676a-43fa-9ffd-a872f474e461" containerName="main" Oct 29 23:31:39.871022 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:39.870937 5057 cpu_manager.go:414] "containerMap: removing container" podUID="189db1b7-a3e4-4dd9-90aa-811ec21845e3" containerName="revoke-keys" Oct 29 23:31:39.871022 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:39.870950 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="189db1b7-a3e4-4dd9-90aa-811ec21845e3" containerName="revoke-keys" Oct 29 23:31:39.871265 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:39.871218 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="189db1b7-a3e4-4dd9-90aa-811ec21845e3" containerName="revoke-keys" Oct 29 23:31:39.871265 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:39.871248 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ae48191-676a-43fa-9ffd-a872f474e461" containerName="main" Oct 29 23:31:39.875382 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:39.875361 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5babc19a-cjh8n" Oct 29 23:31:39.879807 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:39.879762 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5babc19a-cjh8n"] Oct 29 23:31:39.892550 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:39.890689 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dmx57\"" Oct 29 23:31:39.892550 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:39.891306 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:31:40.032221 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:40.032171 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2cb5a4ef-c702-49a4-83e0-0d1f0ff1a549-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5babc19a-cjh8n\" (UID: \"2cb5a4ef-c702-49a4-83e0-0d1f0ff1a549\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5babc19a-cjh8n" Oct 29 23:31:40.032528 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:40.032501 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2cb5a4ef-c702-49a4-83e0-0d1f0ff1a549-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5babc19a-cjh8n\" (UID: \"2cb5a4ef-c702-49a4-83e0-0d1f0ff1a549\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5babc19a-cjh8n" Oct 29 23:31:40.032638 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:40.032555 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ckhrn\" (UniqueName: \"kubernetes.io/projected/2cb5a4ef-c702-49a4-83e0-0d1f0ff1a549-kube-api-access-ckhrn\") pod \"maas-api-revoke-sub-5babc19a-cjh8n\" (UID: \"2cb5a4ef-c702-49a4-83e0-0d1f0ff1a549\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5babc19a-cjh8n" Oct 29 23:31:40.118323 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:40.118292 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ed3bf2-dd855d495-fj85v" Oct 29 23:31:40.120628 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:40.120609 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ed3bf2-7cbcbd6758-2gkwj" Oct 29 23:31:40.133925 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:40.133897 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2cb5a4ef-c702-49a4-83e0-0d1f0ff1a549-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5babc19a-cjh8n\" (UID: \"2cb5a4ef-c702-49a4-83e0-0d1f0ff1a549\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5babc19a-cjh8n" Oct 29 23:31:40.134054 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:40.134005 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2cb5a4ef-c702-49a4-83e0-0d1f0ff1a549-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5babc19a-cjh8n\" (UID: \"2cb5a4ef-c702-49a4-83e0-0d1f0ff1a549\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5babc19a-cjh8n" Oct 29 23:31:40.134054 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:40.134034 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ckhrn\" (UniqueName: \"kubernetes.io/projected/2cb5a4ef-c702-49a4-83e0-0d1f0ff1a549-kube-api-access-ckhrn\") pod \"maas-api-revoke-sub-5babc19a-cjh8n\" (UID: \"2cb5a4ef-c702-49a4-83e0-0d1f0ff1a549\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5babc19a-cjh8n" Oct 29 23:31:40.134643 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:40.134623 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2cb5a4ef-c702-49a4-83e0-0d1f0ff1a549-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5babc19a-cjh8n\" (UID: \"2cb5a4ef-c702-49a4-83e0-0d1f0ff1a549\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5babc19a-cjh8n" Oct 29 23:31:40.136920 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:40.136897 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2cb5a4ef-c702-49a4-83e0-0d1f0ff1a549-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5babc19a-cjh8n\" (UID: \"2cb5a4ef-c702-49a4-83e0-0d1f0ff1a549\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5babc19a-cjh8n" Oct 29 23:31:40.156835 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:40.156806 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ckhrn\" (UniqueName: \"kubernetes.io/projected/2cb5a4ef-c702-49a4-83e0-0d1f0ff1a549-kube-api-access-ckhrn\") pod \"maas-api-revoke-sub-5babc19a-cjh8n\" (UID: \"2cb5a4ef-c702-49a4-83e0-0d1f0ff1a549\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5babc19a-cjh8n" Oct 29 23:31:40.236833 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:40.236726 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5babc19a-cjh8n" Oct 29 23:31:40.237012 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:40.236907 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56962a7e-7f67-4151-b04a-c4161d2de549-kube-api-access-qnlqc\" (UniqueName: \"kubernetes.io/projected/56962a7e-7f67-4151-b04a-c4161d2de549-kube-api-access-qnlqc\") pod \"56962a7e-7f67-4151-b04a-c4161d2de549\" (UID: \"56962a7e-7f67-4151-b04a-c4161d2de549\") " Oct 29 23:31:40.237012 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:40.236965 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/695c8042-ace4-420f-bdaf-11ca7e09ef5f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/695c8042-ace4-420f-bdaf-11ca7e09ef5f-plugins-config-volume\") pod \"695c8042-ace4-420f-bdaf-11ca7e09ef5f\" (UID: \"695c8042-ace4-420f-bdaf-11ca7e09ef5f\") " Oct 29 23:31:40.237136 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:40.237089 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/695c8042-ace4-420f-bdaf-11ca7e09ef5f-kube-api-access-888cr\" (UniqueName: \"kubernetes.io/projected/695c8042-ace4-420f-bdaf-11ca7e09ef5f-kube-api-access-888cr\") pod \"695c8042-ace4-420f-bdaf-11ca7e09ef5f\" (UID: \"695c8042-ace4-420f-bdaf-11ca7e09ef5f\") " Oct 29 23:31:40.237239 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:40.237223 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56962a7e-7f67-4151-b04a-c4161d2de549-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56962a7e-7f67-4151-b04a-c4161d2de549-plugins-config-volume\") pod \"56962a7e-7f67-4151-b04a-c4161d2de549\" (UID: \"56962a7e-7f67-4151-b04a-c4161d2de549\") " Oct 29 23:31:40.238623 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:40.238595 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/695c8042-ace4-420f-bdaf-11ca7e09ef5f-plugins-config-volume" pod "695c8042-ace4-420f-bdaf-11ca7e09ef5f" (UID: "695c8042-ace4-420f-bdaf-11ca7e09ef5f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:31:40.238799 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:40.238761 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56962a7e-7f67-4151-b04a-c4161d2de549-plugins-config-volume" pod "56962a7e-7f67-4151-b04a-c4161d2de549" (UID: "56962a7e-7f67-4151-b04a-c4161d2de549"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:31:40.242379 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:40.241598 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56962a7e-7f67-4151-b04a-c4161d2de549-kube-api-access-qnlqc" pod "56962a7e-7f67-4151-b04a-c4161d2de549" (UID: "56962a7e-7f67-4151-b04a-c4161d2de549"). InnerVolumeSpecName "kube-api-access-qnlqc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:31:40.244515 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:40.244472 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/695c8042-ace4-420f-bdaf-11ca7e09ef5f-kube-api-access-888cr" pod "695c8042-ace4-420f-bdaf-11ca7e09ef5f" (UID: "695c8042-ace4-420f-bdaf-11ca7e09ef5f"). InnerVolumeSpecName "kube-api-access-888cr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:31:40.314014 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:40.313981 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:31:40.330306 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:40.330258 5057 generic.go:356] "Generic (PLEG): container finished" podID="56962a7e-7f67-4151-b04a-c4161d2de549" containerID="c1474dda4e4d0b5a231c642a7efa78b6c0bc880aec7432a2396b95379b37ce36" exitCode=0 Oct 29 23:31:40.330477 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:40.330462 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ed3bf2-7cbcbd6758-2gkwj" Oct 29 23:31:40.331096 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:40.331073 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ed3bf2-7cbcbd6758-2gkwj" event={"ID":"56962a7e-7f67-4151-b04a-c4161d2de549","Type":"ContainerDied","Data":"c1474dda4e4d0b5a231c642a7efa78b6c0bc880aec7432a2396b95379b37ce36"} Oct 29 23:31:40.331146 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:40.331130 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ed3bf2-7cbcbd6758-2gkwj" event={"ID":"56962a7e-7f67-4151-b04a-c4161d2de549","Type":"ContainerDied","Data":"3a780ecb4825e316aac3531e77733f553506580f7bb00d95cc849ade2c567f85"} Oct 29 23:31:40.331196 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:40.331159 5057 scope.go:122] "RemoveContainer" containerID="c1474dda4e4d0b5a231c642a7efa78b6c0bc880aec7432a2396b95379b37ce36" Oct 29 23:31:40.342728 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:40.342700 5057 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56962a7e-7f67-4151-b04a-c4161d2de549-plugins-config-volume\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:31:40.342862 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:40.342733 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qnlqc\" (UniqueName: \"kubernetes.io/projected/56962a7e-7f67-4151-b04a-c4161d2de549-kube-api-access-qnlqc\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:31:40.342862 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:40.342749 5057 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/695c8042-ace4-420f-bdaf-11ca7e09ef5f-plugins-config-volume\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:31:40.342862 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:40.342764 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-888cr\" (UniqueName: \"kubernetes.io/projected/695c8042-ace4-420f-bdaf-11ca7e09ef5f-kube-api-access-888cr\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:31:40.346244 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:40.346199 5057 generic.go:356] "Generic (PLEG): container finished" podID="695c8042-ace4-420f-bdaf-11ca7e09ef5f" containerID="d21b8bfa1035c412881c363719348641f520a2be385a9318dfad63b541eb798d" exitCode=0 Oct 29 23:31:40.349268 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:40.346535 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ed3bf2-dd855d495-fj85v" event={"ID":"695c8042-ace4-420f-bdaf-11ca7e09ef5f","Type":"ContainerDied","Data":"d21b8bfa1035c412881c363719348641f520a2be385a9318dfad63b541eb798d"} Oct 29 23:31:40.349268 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:40.346582 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ed3bf2-dd855d495-fj85v" event={"ID":"695c8042-ace4-420f-bdaf-11ca7e09ef5f","Type":"ContainerDied","Data":"aac3c58bc99e59162f229bc530500eebade7d9b7de4ef21c7dcd4fda5c6b9c04"} Oct 29 23:31:40.349268 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:40.346645 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ed3bf2-dd855d495-fj85v" Oct 29 23:31:40.360486 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:40.360414 5057 scope.go:122] "RemoveContainer" containerID="c1474dda4e4d0b5a231c642a7efa78b6c0bc880aec7432a2396b95379b37ce36" Oct 29 23:31:40.360700 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:31:40.360683 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1474dda4e4d0b5a231c642a7efa78b6c0bc880aec7432a2396b95379b37ce36\": container with ID starting with c1474dda4e4d0b5a231c642a7efa78b6c0bc880aec7432a2396b95379b37ce36 not found: ID does not exist" containerID="c1474dda4e4d0b5a231c642a7efa78b6c0bc880aec7432a2396b95379b37ce36" Oct 29 23:31:40.360785 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:40.360769 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1474dda4e4d0b5a231c642a7efa78b6c0bc880aec7432a2396b95379b37ce36"} err="failed to get container status \"c1474dda4e4d0b5a231c642a7efa78b6c0bc880aec7432a2396b95379b37ce36\": rpc error: code = NotFound desc = could not find container \"c1474dda4e4d0b5a231c642a7efa78b6c0bc880aec7432a2396b95379b37ce36\": container with ID starting with c1474dda4e4d0b5a231c642a7efa78b6c0bc880aec7432a2396b95379b37ce36 not found: ID does not exist" Oct 29 23:31:40.360854 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:40.360847 5057 scope.go:122] "RemoveContainer" containerID="d21b8bfa1035c412881c363719348641f520a2be385a9318dfad63b541eb798d" Oct 29 23:31:40.388373 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:40.382558 5057 scope.go:122] "RemoveContainer" containerID="d21b8bfa1035c412881c363719348641f520a2be385a9318dfad63b541eb798d" Oct 29 23:31:40.442480 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:40.442416 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-ed3bf2-7cbcbd6758-2gkwj"] Oct 29 23:31:40.443068 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:31:40.443003 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d21b8bfa1035c412881c363719348641f520a2be385a9318dfad63b541eb798d\": container with ID starting with d21b8bfa1035c412881c363719348641f520a2be385a9318dfad63b541eb798d not found: ID does not exist" containerID="d21b8bfa1035c412881c363719348641f520a2be385a9318dfad63b541eb798d" Oct 29 23:31:40.443068 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:40.443042 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d21b8bfa1035c412881c363719348641f520a2be385a9318dfad63b541eb798d"} err="failed to get container status \"d21b8bfa1035c412881c363719348641f520a2be385a9318dfad63b541eb798d\": rpc error: code = NotFound desc = could not find container \"d21b8bfa1035c412881c363719348641f520a2be385a9318dfad63b541eb798d\": container with ID starting with d21b8bfa1035c412881c363719348641f520a2be385a9318dfad63b541eb798d not found: ID does not exist" Oct 29 23:31:40.443292 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:40.443265 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-ed3bf2-7cbcbd6758-2gkwj"] Oct 29 23:31:40.454446 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:40.454420 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ed3bf2-dd855d495-fj85v"] Oct 29 23:31:40.459603 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:40.459579 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ed3bf2-dd855d495-fj85v"] Oct 29 23:31:40.620723 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:40.620693 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5babc19a-cjh8n"] Oct 29 23:31:40.623102 ip-10-0-140-74 kubenswrapper[5057]: W1029 23:31:40.623075 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cb5a4ef_c702_49a4_83e0_0d1f0ff1a549.slice/crio-591aa13d5ffaeb063181cbd12ecc7487d68c36671d96cbea705609a93a00a8b4 WatchSource:0}: Error finding container 591aa13d5ffaeb063181cbd12ecc7487d68c36671d96cbea705609a93a00a8b4: Status 404 returned error can't find the container with id 591aa13d5ffaeb063181cbd12ecc7487d68c36671d96cbea705609a93a00a8b4 Oct 29 23:31:41.329257 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:41.329218 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ae48191-676a-43fa-9ffd-a872f474e461" path="/var/lib/kubelet/pods/4ae48191-676a-43fa-9ffd-a872f474e461/volumes" Oct 29 23:31:41.329722 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:41.329705 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56962a7e-7f67-4151-b04a-c4161d2de549" path="/var/lib/kubelet/pods/56962a7e-7f67-4151-b04a-c4161d2de549/volumes" Oct 29 23:31:41.330106 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:41.330092 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="695c8042-ace4-420f-bdaf-11ca7e09ef5f" path="/var/lib/kubelet/pods/695c8042-ace4-420f-bdaf-11ca7e09ef5f/volumes" Oct 29 23:31:41.358513 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:41.358480 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5babc19a-cjh8n" event={"ID":"2cb5a4ef-c702-49a4-83e0-0d1f0ff1a549","Type":"ContainerStarted","Data":"2901cb9566fb13e3f66be97cc64404504d8071f8be013dc21010daa13528e33f"} Oct 29 23:31:41.358693 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:41.358538 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5babc19a-cjh8n" event={"ID":"2cb5a4ef-c702-49a4-83e0-0d1f0ff1a549","Type":"ContainerStarted","Data":"591aa13d5ffaeb063181cbd12ecc7487d68c36671d96cbea705609a93a00a8b4"} Oct 29 23:31:41.400491 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:41.400411 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5babc19a-cjh8n" podStartSLOduration=2.400394438 podStartE2EDuration="2.400394438s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:31:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:31:41.397403343 +0000 UTC m=+3482.180616768" watchObservedRunningTime="2026-10-29 23:31:41.400394438 +0000 UTC m=+3482.183607864" Oct 29 23:31:42.213128 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:42.213091 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:31:42.371595 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:42.371555 5057 generic.go:356] "Generic (PLEG): container finished" podID="2cb5a4ef-c702-49a4-83e0-0d1f0ff1a549" containerID="2901cb9566fb13e3f66be97cc64404504d8071f8be013dc21010daa13528e33f" exitCode=0 Oct 29 23:31:42.372081 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:42.371594 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5babc19a-cjh8n" event={"ID":"2cb5a4ef-c702-49a4-83e0-0d1f0ff1a549","Type":"ContainerDied","Data":"2901cb9566fb13e3f66be97cc64404504d8071f8be013dc21010daa13528e33f"} Oct 29 23:31:42.807047 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:42.807008 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-59eefa-bfbc68664-trphs"] Oct 29 23:31:42.840651 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:42.840619 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-59eefa-6867f64dbc-5fxv2"] Oct 29 23:31:42.957338 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:42.957279 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:31:43.068553 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:43.068464 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:31:43.168826 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:43.168794 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7596-450b88"] Oct 29 23:31:43.169513 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:43.169493 5057 cpu_manager.go:414] "containerMap: removing container" podUID="695c8042-ace4-420f-bdaf-11ca7e09ef5f" containerName="payload-pre-processing" Oct 29 23:31:43.169513 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:43.169511 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="695c8042-ace4-420f-bdaf-11ca7e09ef5f" containerName="payload-pre-processing" Oct 29 23:31:43.169651 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:43.169567 5057 cpu_manager.go:414] "containerMap: removing container" podUID="56962a7e-7f67-4151-b04a-c4161d2de549" containerName="payload-processing" Oct 29 23:31:43.169651 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:43.169572 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56962a7e-7f67-4151-b04a-c4161d2de549" containerName="payload-processing" Oct 29 23:31:43.169722 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:43.169677 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="56962a7e-7f67-4151-b04a-c4161d2de549" containerName="payload-processing" Oct 29 23:31:43.169722 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:43.169686 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="695c8042-ace4-420f-bdaf-11ca7e09ef5f" containerName="payload-pre-processing" Oct 29 23:31:43.173375 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:43.173325 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7596-450b88" Oct 29 23:31:43.188423 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:43.188398 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-fvr2l\"" Oct 29 23:31:43.196258 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:43.196231 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7596-450b88"] Oct 29 23:31:43.288559 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:43.288527 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhb9v\" (UniqueName: \"kubernetes.io/projected/6c332db7-e2f9-495d-8dfe-3a9e2474246d-kube-api-access-nhb9v\") pod \"test-curl-7596-450b88\" (UID: \"6c332db7-e2f9-495d-8dfe-3a9e2474246d\") " pod="opendatahub/test-curl-7596-450b88" Oct 29 23:31:43.389966 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:43.389935 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhb9v\" (UniqueName: \"kubernetes.io/projected/6c332db7-e2f9-495d-8dfe-3a9e2474246d-kube-api-access-nhb9v\") pod \"test-curl-7596-450b88\" (UID: \"6c332db7-e2f9-495d-8dfe-3a9e2474246d\") " pod="opendatahub/test-curl-7596-450b88" Oct 29 23:31:43.435227 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:43.434928 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhb9v\" (UniqueName: \"kubernetes.io/projected/6c332db7-e2f9-495d-8dfe-3a9e2474246d-kube-api-access-nhb9v\") pod \"test-curl-7596-450b88\" (UID: \"6c332db7-e2f9-495d-8dfe-3a9e2474246d\") " pod="opendatahub/test-curl-7596-450b88" Oct 29 23:31:43.497376 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:43.497341 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7596-450b88" Oct 29 23:31:43.678087 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:43.678057 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5babc19a-cjh8n" Oct 29 23:31:43.696742 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:43.696703 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2cb5a4ef-c702-49a4-83e0-0d1f0ff1a549-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2cb5a4ef-c702-49a4-83e0-0d1f0ff1a549-maas-api-openshift-ca\") pod \"2cb5a4ef-c702-49a4-83e0-0d1f0ff1a549\" (UID: \"2cb5a4ef-c702-49a4-83e0-0d1f0ff1a549\") " Oct 29 23:31:43.696897 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:43.696763 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2cb5a4ef-c702-49a4-83e0-0d1f0ff1a549-kube-api-access-ckhrn\" (UniqueName: \"kubernetes.io/projected/2cb5a4ef-c702-49a4-83e0-0d1f0ff1a549-kube-api-access-ckhrn\") pod \"2cb5a4ef-c702-49a4-83e0-0d1f0ff1a549\" (UID: \"2cb5a4ef-c702-49a4-83e0-0d1f0ff1a549\") " Oct 29 23:31:43.697020 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:43.697003 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2cb5a4ef-c702-49a4-83e0-0d1f0ff1a549-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2cb5a4ef-c702-49a4-83e0-0d1f0ff1a549-maas-api-xks-ca\") pod \"2cb5a4ef-c702-49a4-83e0-0d1f0ff1a549\" (UID: \"2cb5a4ef-c702-49a4-83e0-0d1f0ff1a549\") " Oct 29 23:31:43.697082 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:43.697040 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2cb5a4ef-c702-49a4-83e0-0d1f0ff1a549-maas-api-openshift-ca" pod "2cb5a4ef-c702-49a4-83e0-0d1f0ff1a549" (UID: "2cb5a4ef-c702-49a4-83e0-0d1f0ff1a549"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:31:43.697548 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:43.697513 5057 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2cb5a4ef-c702-49a4-83e0-0d1f0ff1a549-maas-api-openshift-ca\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:31:43.699237 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:43.699212 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2cb5a4ef-c702-49a4-83e0-0d1f0ff1a549-maas-api-xks-ca" pod "2cb5a4ef-c702-49a4-83e0-0d1f0ff1a549" (UID: "2cb5a4ef-c702-49a4-83e0-0d1f0ff1a549"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:31:43.699460 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:43.699439 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2cb5a4ef-c702-49a4-83e0-0d1f0ff1a549-kube-api-access-ckhrn" pod "2cb5a4ef-c702-49a4-83e0-0d1f0ff1a549" (UID: "2cb5a4ef-c702-49a4-83e0-0d1f0ff1a549"). InnerVolumeSpecName "kube-api-access-ckhrn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:31:43.798902 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:43.798869 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ckhrn\" (UniqueName: \"kubernetes.io/projected/2cb5a4ef-c702-49a4-83e0-0d1f0ff1a549-kube-api-access-ckhrn\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:31:43.798902 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:43.798900 5057 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2cb5a4ef-c702-49a4-83e0-0d1f0ff1a549-maas-api-xks-ca\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:31:43.833905 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:43.833879 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7596-450b88"] Oct 29 23:31:43.834769 ip-10-0-140-74 kubenswrapper[5057]: W1029 23:31:43.834745 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6c332db7_e2f9_495d_8dfe_3a9e2474246d.slice/crio-62a95e1a54459dd96620e7513aceeecd6e1de8526349963cb0bd55f1346eccbc WatchSource:0}: Error finding container 62a95e1a54459dd96620e7513aceeecd6e1de8526349963cb0bd55f1346eccbc: Status 404 returned error can't find the container with id 62a95e1a54459dd96620e7513aceeecd6e1de8526349963cb0bd55f1346eccbc Oct 29 23:31:43.914117 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:43.914085 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:31:44.089152 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:44.089110 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w2-575d59-65c58457dd-c4d2z" podUID="3bb1016d-e1c5-4e72-ab08-a063c94e3079" containerName="payload-processing" containerID="cri-o://859f4b40a2735141c396ace5bb8a76e1e729fbc5ccac7db116b1d72c0087416b" gracePeriod=25 Oct 29 23:31:44.393264 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:44.393232 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5babc19a-cjh8n" Oct 29 23:31:44.393264 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:44.393257 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5babc19a-cjh8n" event={"ID":"2cb5a4ef-c702-49a4-83e0-0d1f0ff1a549","Type":"ContainerDied","Data":"591aa13d5ffaeb063181cbd12ecc7487d68c36671d96cbea705609a93a00a8b4"} Oct 29 23:31:44.393838 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:44.393308 5057 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="591aa13d5ffaeb063181cbd12ecc7487d68c36671d96cbea705609a93a00a8b4" Oct 29 23:31:44.399696 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:44.399662 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7596-450b88" event={"ID":"6c332db7-e2f9-495d-8dfe-3a9e2474246d","Type":"ContainerStarted","Data":"962f3c162c82a853a09ab27afb962997d1e03ea64d550d886daa1ca6ee3bf9a1"} Oct 29 23:31:44.399827 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:44.399709 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7596-450b88" event={"ID":"6c332db7-e2f9-495d-8dfe-3a9e2474246d","Type":"ContainerStarted","Data":"62a95e1a54459dd96620e7513aceeecd6e1de8526349963cb0bd55f1346eccbc"} Oct 29 23:31:44.437041 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:44.436992 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7596-450b88" podStartSLOduration=1.436979864 podStartE2EDuration="1.436979864s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:31:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:31:44.435859908 +0000 UTC m=+3485.219073353" watchObservedRunningTime="2026-10-29 23:31:44.436979864 +0000 UTC m=+3485.220193290" Oct 29 23:31:44.556475 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:44.556440 5057 scope.go:122] "RemoveContainer" containerID="e6f242d686dbf299d854b5a83769ff7f9e7396435f5cad56517d4b2a7ccf7307" Oct 29 23:31:44.568398 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:44.568373 5057 scope.go:122] "RemoveContainer" containerID="4078490a9d95f128f42ba59fe2fed27076cf6d44f3035f5cb99906de470a5bf8" Oct 29 23:31:44.578245 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:44.578223 5057 scope.go:122] "RemoveContainer" containerID="81dec6d33a1cdcb1f0d9abf7e190eb24d1da254eec4ae5de05a3eeb2625f826b" Oct 29 23:31:44.590049 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:44.590028 5057 scope.go:122] "RemoveContainer" containerID="e07f3c463f24e027d9e9e362be1ea0eba9b4d43bc226953a3bd4fa2ee94537b4" Oct 29 23:31:44.605895 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:44.605872 5057 scope.go:122] "RemoveContainer" containerID="065a3eb120a6a150bf4db4a0e291f76138871419a9559ebd0507bca41e8f9a46" Oct 29 23:31:45.348420 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:45.348315 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7596-450b88"] Oct 29 23:31:45.354308 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:45.354276 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7596-450b88"] Oct 29 23:31:45.491884 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:45.491841 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:31:45.506929 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:45.506898 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7596-a53d72"] Oct 29 23:31:45.507653 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:45.507630 5057 cpu_manager.go:414] "containerMap: removing container" podUID="2cb5a4ef-c702-49a4-83e0-0d1f0ff1a549" containerName="revoke-keys" Oct 29 23:31:45.507653 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:45.507656 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2cb5a4ef-c702-49a4-83e0-0d1f0ff1a549" containerName="revoke-keys" Oct 29 23:31:45.507761 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:45.507678 5057 cpu_manager.go:414] "containerMap: removing container" podUID="6c332db7-e2f9-495d-8dfe-3a9e2474246d" containerName="test-curl-7596-450b88" Oct 29 23:31:45.507761 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:45.507683 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c332db7-e2f9-495d-8dfe-3a9e2474246d" containerName="test-curl-7596-450b88" Oct 29 23:31:45.507832 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:45.507783 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c332db7-e2f9-495d-8dfe-3a9e2474246d" containerName="test-curl-7596-450b88" Oct 29 23:31:45.507832 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:45.507803 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="2cb5a4ef-c702-49a4-83e0-0d1f0ff1a549" containerName="revoke-keys" Oct 29 23:31:45.513683 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:45.513660 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7596-a53d72" Oct 29 23:31:45.526746 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:45.526719 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7596-a53d72"] Oct 29 23:31:45.627466 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:45.627378 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjrsm\" (UniqueName: \"kubernetes.io/projected/b93153d1-59cc-4e9a-94c3-dd3c44a33d9c-kube-api-access-qjrsm\") pod \"test-curl-7596-a53d72\" (UID: \"b93153d1-59cc-4e9a-94c3-dd3c44a33d9c\") " pod="opendatahub/test-curl-7596-a53d72" Oct 29 23:31:45.728760 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:45.728728 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjrsm\" (UniqueName: \"kubernetes.io/projected/b93153d1-59cc-4e9a-94c3-dd3c44a33d9c-kube-api-access-qjrsm\") pod \"test-curl-7596-a53d72\" (UID: \"b93153d1-59cc-4e9a-94c3-dd3c44a33d9c\") " pod="opendatahub/test-curl-7596-a53d72" Oct 29 23:31:45.751255 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:45.751228 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjrsm\" (UniqueName: \"kubernetes.io/projected/b93153d1-59cc-4e9a-94c3-dd3c44a33d9c-kube-api-access-qjrsm\") pod \"test-curl-7596-a53d72\" (UID: \"b93153d1-59cc-4e9a-94c3-dd3c44a33d9c\") " pod="opendatahub/test-curl-7596-a53d72" Oct 29 23:31:45.835561 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:45.835528 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7596-a53d72" Oct 29 23:31:46.142616 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:46.142585 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7596-a53d72"] Oct 29 23:31:46.143522 ip-10-0-140-74 kubenswrapper[5057]: W1029 23:31:46.143493 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb93153d1_59cc_4e9a_94c3_dd3c44a33d9c.slice/crio-323edca052641f08a672659a58f093ab2f91e22c29484ea00b10faa18606575b WatchSource:0}: Error finding container 323edca052641f08a672659a58f093ab2f91e22c29484ea00b10faa18606575b: Status 404 returned error can't find the container with id 323edca052641f08a672659a58f093ab2f91e22c29484ea00b10faa18606575b Oct 29 23:31:46.424237 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:46.424139 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7596-a53d72" event={"ID":"b93153d1-59cc-4e9a-94c3-dd3c44a33d9c","Type":"ContainerStarted","Data":"db2dc4ea4f7f74a4c3963f3a45dd15814adc309c277d36b507081e8e1ceef9e4"} Oct 29 23:31:46.424237 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:46.424192 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7596-a53d72" event={"ID":"b93153d1-59cc-4e9a-94c3-dd3c44a33d9c","Type":"ContainerStarted","Data":"323edca052641f08a672659a58f093ab2f91e22c29484ea00b10faa18606575b"} Oct 29 23:31:46.424491 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:46.424397 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7596-450b88" podUID="6c332db7-e2f9-495d-8dfe-3a9e2474246d" containerName="test-curl-7596-450b88" containerID="cri-o://962f3c162c82a853a09ab27afb962997d1e03ea64d550d886daa1ca6ee3bf9a1" gracePeriod=2 Oct 29 23:31:46.462304 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:46.462258 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7596-a53d72" podStartSLOduration=1.462242768 podStartE2EDuration="1.462242768s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:31:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:31:46.460759624 +0000 UTC m=+3487.243973051" watchObservedRunningTime="2026-10-29 23:31:46.462242768 +0000 UTC m=+3487.245456198" Oct 29 23:31:46.473768 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:31:46.473730 5057 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7596-450b88\" is forbidden: User \"system:node:ip-10-0-140-74.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-74.ec2.internal' and this object" podUID="6c332db7-e2f9-495d-8dfe-3a9e2474246d" pod="opendatahub/test-curl-7596-450b88" Oct 29 23:31:47.116132 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:47.116062 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w1-e170ed/rate-test-model-e170ed-kserve-544667c877-hc2sg" podUID="2c19bfd1-9bdf-48e2-9033-29a2f02ef5b2" containerName="main" containerID="cri-o://a7efd1d116ad6b79adbb884f9cd34b050d1882d0d5e230c3fdae27a7272e2378" gracePeriod=45 Oct 29 23:31:47.123806 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:47.123772 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:31:47.216170 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:47.216133 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-575d59-65c58457dd-c4d2z" podUID="3bb1016d-e1c5-4e72-ab08-a063c94e3079" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:31:47.334917 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:47.334845 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7596-a53d72"] Oct 29 23:31:47.336099 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:47.336069 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7596-a53d72"] Oct 29 23:31:47.435378 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:47.435343 5057 generic.go:356] "Generic (PLEG): container finished" podID="2c19bfd1-9bdf-48e2-9033-29a2f02ef5b2" containerID="a7efd1d116ad6b79adbb884f9cd34b050d1882d0d5e230c3fdae27a7272e2378" exitCode=0 Oct 29 23:31:47.435524 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:47.435406 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-e170ed/rate-test-model-e170ed-kserve-544667c877-hc2sg" event={"ID":"2c19bfd1-9bdf-48e2-9033-29a2f02ef5b2","Type":"ContainerDied","Data":"a7efd1d116ad6b79adbb884f9cd34b050d1882d0d5e230c3fdae27a7272e2378"} Oct 29 23:31:47.435524 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:47.435446 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-e170ed/rate-test-model-e170ed-kserve-544667c877-hc2sg" event={"ID":"2c19bfd1-9bdf-48e2-9033-29a2f02ef5b2","Type":"ContainerDied","Data":"d2d8e774dc4ffdb5316eeca6f521385df9258cab28ceb132a73d6f44b85ffd3e"} Oct 29 23:31:47.435524 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:47.435460 5057 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d2d8e774dc4ffdb5316eeca6f521385df9258cab28ceb132a73d6f44b85ffd3e" Oct 29 23:31:47.443512 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:47.443492 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-e170ed/rate-test-model-e170ed-kserve-544667c877-hc2sg" Oct 29 23:31:47.551438 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:47.551400 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2c19bfd1-9bdf-48e2-9033-29a2f02ef5b2-tls-certs\" (UniqueName: \"kubernetes.io/secret/2c19bfd1-9bdf-48e2-9033-29a2f02ef5b2-tls-certs\") pod \"2c19bfd1-9bdf-48e2-9033-29a2f02ef5b2\" (UID: \"2c19bfd1-9bdf-48e2-9033-29a2f02ef5b2\") " Oct 29 23:31:47.551638 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:47.551488 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c19bfd1-9bdf-48e2-9033-29a2f02ef5b2-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c19bfd1-9bdf-48e2-9033-29a2f02ef5b2-dshm\") pod \"2c19bfd1-9bdf-48e2-9033-29a2f02ef5b2\" (UID: \"2c19bfd1-9bdf-48e2-9033-29a2f02ef5b2\") " Oct 29 23:31:47.551638 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:47.551566 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c19bfd1-9bdf-48e2-9033-29a2f02ef5b2-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c19bfd1-9bdf-48e2-9033-29a2f02ef5b2-model-cache\") pod \"2c19bfd1-9bdf-48e2-9033-29a2f02ef5b2\" (UID: \"2c19bfd1-9bdf-48e2-9033-29a2f02ef5b2\") " Oct 29 23:31:47.551638 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:47.551609 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c19bfd1-9bdf-48e2-9033-29a2f02ef5b2-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c19bfd1-9bdf-48e2-9033-29a2f02ef5b2-tmp-dir\") pod \"2c19bfd1-9bdf-48e2-9033-29a2f02ef5b2\" (UID: \"2c19bfd1-9bdf-48e2-9033-29a2f02ef5b2\") " Oct 29 23:31:47.551821 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:47.551691 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c19bfd1-9bdf-48e2-9033-29a2f02ef5b2-home\" (UniqueName: \"kubernetes.io/empty-dir/2c19bfd1-9bdf-48e2-9033-29a2f02ef5b2-home\") pod \"2c19bfd1-9bdf-48e2-9033-29a2f02ef5b2\" (UID: \"2c19bfd1-9bdf-48e2-9033-29a2f02ef5b2\") " Oct 29 23:31:47.551883 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:47.551826 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c19bfd1-9bdf-48e2-9033-29a2f02ef5b2-kube-api-access-pcqrc\" (UniqueName: \"kubernetes.io/projected/2c19bfd1-9bdf-48e2-9033-29a2f02ef5b2-kube-api-access-pcqrc\") pod \"2c19bfd1-9bdf-48e2-9033-29a2f02ef5b2\" (UID: \"2c19bfd1-9bdf-48e2-9033-29a2f02ef5b2\") " Oct 29 23:31:47.551943 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:47.551896 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c19bfd1-9bdf-48e2-9033-29a2f02ef5b2-model-cache" pod "2c19bfd1-9bdf-48e2-9033-29a2f02ef5b2" (UID: "2c19bfd1-9bdf-48e2-9033-29a2f02ef5b2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:31:47.551943 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:47.551921 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c19bfd1-9bdf-48e2-9033-29a2f02ef5b2-tmp-dir" pod "2c19bfd1-9bdf-48e2-9033-29a2f02ef5b2" (UID: "2c19bfd1-9bdf-48e2-9033-29a2f02ef5b2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:31:47.552076 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:47.552043 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c19bfd1-9bdf-48e2-9033-29a2f02ef5b2-home" pod "2c19bfd1-9bdf-48e2-9033-29a2f02ef5b2" (UID: "2c19bfd1-9bdf-48e2-9033-29a2f02ef5b2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:31:47.552396 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:47.552378 5057 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c19bfd1-9bdf-48e2-9033-29a2f02ef5b2-home\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:31:47.552502 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:47.552399 5057 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c19bfd1-9bdf-48e2-9033-29a2f02ef5b2-model-cache\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:31:47.552502 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:47.552414 5057 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c19bfd1-9bdf-48e2-9033-29a2f02ef5b2-tmp-dir\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:31:47.554018 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:47.553987 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c19bfd1-9bdf-48e2-9033-29a2f02ef5b2-kube-api-access-pcqrc" pod "2c19bfd1-9bdf-48e2-9033-29a2f02ef5b2" (UID: "2c19bfd1-9bdf-48e2-9033-29a2f02ef5b2"). InnerVolumeSpecName "kube-api-access-pcqrc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:31:47.554144 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:47.554110 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c19bfd1-9bdf-48e2-9033-29a2f02ef5b2-tls-certs" pod "2c19bfd1-9bdf-48e2-9033-29a2f02ef5b2" (UID: "2c19bfd1-9bdf-48e2-9033-29a2f02ef5b2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:31:47.554144 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:47.554134 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c19bfd1-9bdf-48e2-9033-29a2f02ef5b2-dshm" pod "2c19bfd1-9bdf-48e2-9033-29a2f02ef5b2" (UID: "2c19bfd1-9bdf-48e2-9033-29a2f02ef5b2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:31:47.653794 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:47.653724 5057 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c19bfd1-9bdf-48e2-9033-29a2f02ef5b2-tls-certs\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:31:47.653794 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:47.653751 5057 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c19bfd1-9bdf-48e2-9033-29a2f02ef5b2-dshm\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:31:47.653794 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:47.653761 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pcqrc\" (UniqueName: \"kubernetes.io/projected/2c19bfd1-9bdf-48e2-9033-29a2f02ef5b2-kube-api-access-pcqrc\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:31:47.807510 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:47.807465 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-59eefa-bfbc68664-trphs" podUID="e5000261-99fe-4f46-8056-ffe2298d8610" containerName="payload-processing" containerID="cri-o://ee3303b0e063d310fb05ed74624103eeaa7606cdf2d137b533207e6a4a3b39a2" gracePeriod=25 Oct 29 23:31:47.841561 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:47.841520 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-59eefa-6867f64dbc-5fxv2" podUID="ffe9a7c2-5370-4aaf-99b8-d1d4bdad7f0f" containerName="payload-pre-processing" containerID="cri-o://c42625b42e3ba7dbb9afe9682f109076edf814c1cf1417439ad9f19c7b54d5d1" gracePeriod=25 Oct 29 23:31:48.402801 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:48.402761 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-575d59/distinct-2-w2-575d59-kserve-d4565b76c-hdgbf" podUID="06bb975e-d825-426f-b030-17bd1f99bb65" containerName="main" containerID="cri-o://e704c8a8830b7916eee00fae54a8c9ad8a79e8eb469659a79fab08e751af81a1" gracePeriod=45 Oct 29 23:31:48.444180 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:48.444151 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-e170ed/rate-test-model-e170ed-kserve-544667c877-hc2sg" Oct 29 23:31:48.444308 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:48.444226 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7596-a53d72" podUID="b93153d1-59cc-4e9a-94c3-dd3c44a33d9c" containerName="test-curl-7596-a53d72" containerID="cri-o://db2dc4ea4f7f74a4c3963f3a45dd15814adc309c277d36b507081e8e1ceef9e4" gracePeriod=2 Oct 29 23:31:48.472850 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:31:48.472821 5057 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7596-a53d72\" is forbidden: User \"system:node:ip-10-0-140-74.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-74.ec2.internal' and this object" podUID="b93153d1-59cc-4e9a-94c3-dd3c44a33d9c" pod="opendatahub/test-curl-7596-a53d72" Oct 29 23:31:48.486618 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:31:48.486583 5057 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7596-a53d72\" is forbidden: User \"system:node:ip-10-0-140-74.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-74.ec2.internal' and this object" podUID="b93153d1-59cc-4e9a-94c3-dd3c44a33d9c" pod="opendatahub/test-curl-7596-a53d72" Oct 29 23:31:48.620408 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:48.620249 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-e170ed/rate-test-model-e170ed-kserve-544667c877-hc2sg"] Oct 29 23:31:48.627081 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:48.626117 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w1-e170ed/rate-test-model-e170ed-kserve-544667c877-hc2sg"] Oct 29 23:31:48.628089 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:31:48.627249 5057 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7596-a53d72\" is forbidden: User \"system:node:ip-10-0-140-74.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-74.ec2.internal' and this object" podUID="b93153d1-59cc-4e9a-94c3-dd3c44a33d9c" pod="opendatahub/test-curl-7596-a53d72" Oct 29 23:31:48.696498 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:48.696470 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-575d59/distinct-2-w2-575d59-kserve-d4565b76c-hdgbf" Oct 29 23:31:48.719990 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:48.719967 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7596-450b88" Oct 29 23:31:48.742754 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:48.742722 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:31:48.751640 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:31:48.751604 5057 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7596-a53d72\" is forbidden: User \"system:node:ip-10-0-140-74.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-74.ec2.internal' and this object" podUID="b93153d1-59cc-4e9a-94c3-dd3c44a33d9c" pod="opendatahub/test-curl-7596-a53d72" Oct 29 23:31:48.765247 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:31:48.765189 5057 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7596-a53d72\" is forbidden: User \"system:node:ip-10-0-140-74.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-74.ec2.internal' and this object" podUID="b93153d1-59cc-4e9a-94c3-dd3c44a33d9c" pod="opendatahub/test-curl-7596-a53d72" Oct 29 23:31:48.768735 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:48.768705 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06bb975e-d825-426f-b030-17bd1f99bb65-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06bb975e-d825-426f-b030-17bd1f99bb65-tmp-dir\") pod \"06bb975e-d825-426f-b030-17bd1f99bb65\" (UID: \"06bb975e-d825-426f-b030-17bd1f99bb65\") " Oct 29 23:31:48.768837 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:48.768762 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06bb975e-d825-426f-b030-17bd1f99bb65-dshm\" (UniqueName: \"kubernetes.io/empty-dir/06bb975e-d825-426f-b030-17bd1f99bb65-dshm\") pod \"06bb975e-d825-426f-b030-17bd1f99bb65\" (UID: \"06bb975e-d825-426f-b030-17bd1f99bb65\") " Oct 29 23:31:48.768837 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:48.768810 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06bb975e-d825-426f-b030-17bd1f99bb65-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06bb975e-d825-426f-b030-17bd1f99bb65-model-cache\") pod \"06bb975e-d825-426f-b030-17bd1f99bb65\" (UID: \"06bb975e-d825-426f-b030-17bd1f99bb65\") " Oct 29 23:31:48.768940 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:48.768927 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c332db7-e2f9-495d-8dfe-3a9e2474246d-kube-api-access-nhb9v\" (UniqueName: \"kubernetes.io/projected/6c332db7-e2f9-495d-8dfe-3a9e2474246d-kube-api-access-nhb9v\") pod \"6c332db7-e2f9-495d-8dfe-3a9e2474246d\" (UID: \"6c332db7-e2f9-495d-8dfe-3a9e2474246d\") " Oct 29 23:31:48.768987 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:48.768975 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06bb975e-d825-426f-b030-17bd1f99bb65-kube-api-access-hchnc\" (UniqueName: \"kubernetes.io/projected/06bb975e-d825-426f-b030-17bd1f99bb65-kube-api-access-hchnc\") pod \"06bb975e-d825-426f-b030-17bd1f99bb65\" (UID: \"06bb975e-d825-426f-b030-17bd1f99bb65\") " Oct 29 23:31:48.769032 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:48.768997 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06bb975e-d825-426f-b030-17bd1f99bb65-home\" (UniqueName: \"kubernetes.io/empty-dir/06bb975e-d825-426f-b030-17bd1f99bb65-home\") pod \"06bb975e-d825-426f-b030-17bd1f99bb65\" (UID: \"06bb975e-d825-426f-b030-17bd1f99bb65\") " Oct 29 23:31:48.769032 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:48.769026 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/06bb975e-d825-426f-b030-17bd1f99bb65-tls-certs\" (UniqueName: \"kubernetes.io/secret/06bb975e-d825-426f-b030-17bd1f99bb65-tls-certs\") pod \"06bb975e-d825-426f-b030-17bd1f99bb65\" (UID: \"06bb975e-d825-426f-b030-17bd1f99bb65\") " Oct 29 23:31:48.770079 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:48.770050 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06bb975e-d825-426f-b030-17bd1f99bb65-model-cache" pod "06bb975e-d825-426f-b030-17bd1f99bb65" (UID: "06bb975e-d825-426f-b030-17bd1f99bb65"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:31:48.770189 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:48.770114 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06bb975e-d825-426f-b030-17bd1f99bb65-tmp-dir" pod "06bb975e-d825-426f-b030-17bd1f99bb65" (UID: "06bb975e-d825-426f-b030-17bd1f99bb65"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:31:48.770712 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:48.770671 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06bb975e-d825-426f-b030-17bd1f99bb65-home" pod "06bb975e-d825-426f-b030-17bd1f99bb65" (UID: "06bb975e-d825-426f-b030-17bd1f99bb65"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:31:48.772443 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:48.772420 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06bb975e-d825-426f-b030-17bd1f99bb65-tls-certs" pod "06bb975e-d825-426f-b030-17bd1f99bb65" (UID: "06bb975e-d825-426f-b030-17bd1f99bb65"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:31:48.773684 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:48.773662 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06bb975e-d825-426f-b030-17bd1f99bb65-dshm" pod "06bb975e-d825-426f-b030-17bd1f99bb65" (UID: "06bb975e-d825-426f-b030-17bd1f99bb65"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:31:48.773766 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:48.773727 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06bb975e-d825-426f-b030-17bd1f99bb65-kube-api-access-hchnc" pod "06bb975e-d825-426f-b030-17bd1f99bb65" (UID: "06bb975e-d825-426f-b030-17bd1f99bb65"). InnerVolumeSpecName "kube-api-access-hchnc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:31:48.774920 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:48.774896 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c332db7-e2f9-495d-8dfe-3a9e2474246d-kube-api-access-nhb9v" pod "6c332db7-e2f9-495d-8dfe-3a9e2474246d" (UID: "6c332db7-e2f9-495d-8dfe-3a9e2474246d"). InnerVolumeSpecName "kube-api-access-nhb9v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:31:48.871281 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:48.871247 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nhb9v\" (UniqueName: \"kubernetes.io/projected/6c332db7-e2f9-495d-8dfe-3a9e2474246d-kube-api-access-nhb9v\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:31:48.871281 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:48.871278 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hchnc\" (UniqueName: \"kubernetes.io/projected/06bb975e-d825-426f-b030-17bd1f99bb65-kube-api-access-hchnc\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:31:48.871560 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:48.871293 5057 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06bb975e-d825-426f-b030-17bd1f99bb65-home\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:31:48.871560 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:48.871307 5057 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06bb975e-d825-426f-b030-17bd1f99bb65-tls-certs\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:31:48.871560 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:48.871319 5057 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06bb975e-d825-426f-b030-17bd1f99bb65-tmp-dir\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:31:48.871560 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:48.871362 5057 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06bb975e-d825-426f-b030-17bd1f99bb65-dshm\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:31:48.871560 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:48.871377 5057 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06bb975e-d825-426f-b030-17bd1f99bb65-model-cache\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:31:49.027964 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:31:49.027932 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d2dnn" containerName="extractor" Oct 29 23:31:49.341123 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:49.341042 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c19bfd1-9bdf-48e2-9033-29a2f02ef5b2" path="/var/lib/kubelet/pods/2c19bfd1-9bdf-48e2-9033-29a2f02ef5b2/volumes" Oct 29 23:31:49.341487 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:49.341472 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c332db7-e2f9-495d-8dfe-3a9e2474246d" path="/var/lib/kubelet/pods/6c332db7-e2f9-495d-8dfe-3a9e2474246d/volumes" Oct 29 23:31:49.348323 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:31:49.348290 5057 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7596-a53d72\" is forbidden: User \"system:node:ip-10-0-140-74.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-74.ec2.internal' and this object" podUID="b93153d1-59cc-4e9a-94c3-dd3c44a33d9c" pod="opendatahub/test-curl-7596-a53d72" Oct 29 23:31:49.453013 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:49.452981 5057 generic.go:356] "Generic (PLEG): container finished" podID="6c332db7-e2f9-495d-8dfe-3a9e2474246d" containerID="962f3c162c82a853a09ab27afb962997d1e03ea64d550d886daa1ca6ee3bf9a1" exitCode=137 Oct 29 23:31:49.453433 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:49.453031 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7596-450b88" Oct 29 23:31:49.453433 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:49.453083 5057 scope.go:122] "RemoveContainer" containerID="962f3c162c82a853a09ab27afb962997d1e03ea64d550d886daa1ca6ee3bf9a1" Oct 29 23:31:49.454552 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:49.454530 5057 generic.go:356] "Generic (PLEG): container finished" podID="06bb975e-d825-426f-b030-17bd1f99bb65" containerID="e704c8a8830b7916eee00fae54a8c9ad8a79e8eb469659a79fab08e751af81a1" exitCode=0 Oct 29 23:31:49.454646 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:49.454590 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-575d59/distinct-2-w2-575d59-kserve-d4565b76c-hdgbf" Oct 29 23:31:49.454708 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:49.454631 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-575d59/distinct-2-w2-575d59-kserve-d4565b76c-hdgbf" event={"ID":"06bb975e-d825-426f-b030-17bd1f99bb65","Type":"ContainerDied","Data":"e704c8a8830b7916eee00fae54a8c9ad8a79e8eb469659a79fab08e751af81a1"} Oct 29 23:31:49.454708 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:49.454675 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-575d59/distinct-2-w2-575d59-kserve-d4565b76c-hdgbf" event={"ID":"06bb975e-d825-426f-b030-17bd1f99bb65","Type":"ContainerDied","Data":"9cde8c313d2c28b3d25a153f7c5954231196de2ed7b6f9f267d9dcb2fcf04fe7"} Oct 29 23:31:49.463616 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:49.463596 5057 scope.go:122] "RemoveContainer" containerID="962f3c162c82a853a09ab27afb962997d1e03ea64d550d886daa1ca6ee3bf9a1" Oct 29 23:31:49.463891 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:31:49.463873 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"962f3c162c82a853a09ab27afb962997d1e03ea64d550d886daa1ca6ee3bf9a1\": container with ID starting with 962f3c162c82a853a09ab27afb962997d1e03ea64d550d886daa1ca6ee3bf9a1 not found: ID does not exist" containerID="962f3c162c82a853a09ab27afb962997d1e03ea64d550d886daa1ca6ee3bf9a1" Oct 29 23:31:49.463951 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:49.463899 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"962f3c162c82a853a09ab27afb962997d1e03ea64d550d886daa1ca6ee3bf9a1"} err="failed to get container status \"962f3c162c82a853a09ab27afb962997d1e03ea64d550d886daa1ca6ee3bf9a1\": rpc error: code = NotFound desc = could not find container \"962f3c162c82a853a09ab27afb962997d1e03ea64d550d886daa1ca6ee3bf9a1\": container with ID starting with 962f3c162c82a853a09ab27afb962997d1e03ea64d550d886daa1ca6ee3bf9a1 not found: ID does not exist" Oct 29 23:31:49.463951 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:49.463915 5057 scope.go:122] "RemoveContainer" containerID="e704c8a8830b7916eee00fae54a8c9ad8a79e8eb469659a79fab08e751af81a1" Oct 29 23:31:49.473088 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:49.473066 5057 scope.go:122] "RemoveContainer" containerID="e704c8a8830b7916eee00fae54a8c9ad8a79e8eb469659a79fab08e751af81a1" Oct 29 23:31:49.473373 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:31:49.473323 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e704c8a8830b7916eee00fae54a8c9ad8a79e8eb469659a79fab08e751af81a1\": container with ID starting with e704c8a8830b7916eee00fae54a8c9ad8a79e8eb469659a79fab08e751af81a1 not found: ID does not exist" containerID="e704c8a8830b7916eee00fae54a8c9ad8a79e8eb469659a79fab08e751af81a1" Oct 29 23:31:49.473457 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:49.473383 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e704c8a8830b7916eee00fae54a8c9ad8a79e8eb469659a79fab08e751af81a1"} err="failed to get container status \"e704c8a8830b7916eee00fae54a8c9ad8a79e8eb469659a79fab08e751af81a1\": rpc error: code = NotFound desc = could not find container \"e704c8a8830b7916eee00fae54a8c9ad8a79e8eb469659a79fab08e751af81a1\": container with ID starting with e704c8a8830b7916eee00fae54a8c9ad8a79e8eb469659a79fab08e751af81a1 not found: ID does not exist" Oct 29 23:31:49.507171 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:49.507142 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-575d59/distinct-2-w2-575d59-kserve-d4565b76c-hdgbf"] Oct 29 23:31:49.517967 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:49.517945 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-575d59/distinct-2-w2-575d59-kserve-d4565b76c-hdgbf"] Oct 29 23:31:50.402697 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:50.402658 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:31:50.655109 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:50.655044 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65a313ba-f92ch"] Oct 29 23:31:50.656045 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:50.656027 5057 cpu_manager.go:414] "containerMap: removing container" podUID="2c19bfd1-9bdf-48e2-9033-29a2f02ef5b2" containerName="main" Oct 29 23:31:50.656145 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:50.656047 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c19bfd1-9bdf-48e2-9033-29a2f02ef5b2" containerName="main" Oct 29 23:31:50.656145 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:50.656076 5057 cpu_manager.go:414] "containerMap: removing container" podUID="b93153d1-59cc-4e9a-94c3-dd3c44a33d9c" containerName="test-curl-7596-a53d72" Oct 29 23:31:50.656145 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:50.656084 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b93153d1-59cc-4e9a-94c3-dd3c44a33d9c" containerName="test-curl-7596-a53d72" Oct 29 23:31:50.656145 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:50.656141 5057 cpu_manager.go:414] "containerMap: removing container" podUID="06bb975e-d825-426f-b030-17bd1f99bb65" containerName="main" Oct 29 23:31:50.656270 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:50.656148 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06bb975e-d825-426f-b030-17bd1f99bb65" containerName="main" Oct 29 23:31:50.656354 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:50.656342 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="06bb975e-d825-426f-b030-17bd1f99bb65" containerName="main" Oct 29 23:31:50.656394 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:50.656364 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c19bfd1-9bdf-48e2-9033-29a2f02ef5b2" containerName="main" Oct 29 23:31:50.656394 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:50.656376 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="b93153d1-59cc-4e9a-94c3-dd3c44a33d9c" containerName="test-curl-7596-a53d72" Oct 29 23:31:50.658298 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:50.658282 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7596-a53d72" Oct 29 23:31:50.660293 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:50.660276 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65a313ba-f92ch" Oct 29 23:31:50.687562 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:50.687536 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dmx57\"" Oct 29 23:31:50.687726 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:50.687614 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:31:50.693692 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:50.693668 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65a313ba-f92ch"] Oct 29 23:31:50.801632 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:50.801597 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b93153d1-59cc-4e9a-94c3-dd3c44a33d9c-kube-api-access-qjrsm\" (UniqueName: \"kubernetes.io/projected/b93153d1-59cc-4e9a-94c3-dd3c44a33d9c-kube-api-access-qjrsm\") pod \"b93153d1-59cc-4e9a-94c3-dd3c44a33d9c\" (UID: \"b93153d1-59cc-4e9a-94c3-dd3c44a33d9c\") " Oct 29 23:31:50.801944 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:50.801926 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5be39f9d-8c3a-4450-bd67-b1a17ee56be7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-65a313ba-f92ch\" (UID: \"5be39f9d-8c3a-4450-bd67-b1a17ee56be7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65a313ba-f92ch" Oct 29 23:31:50.802171 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:50.802146 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hs66h\" (UniqueName: \"kubernetes.io/projected/5be39f9d-8c3a-4450-bd67-b1a17ee56be7-kube-api-access-hs66h\") pod \"maas-api-revoke-sub-65a313ba-f92ch\" (UID: \"5be39f9d-8c3a-4450-bd67-b1a17ee56be7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65a313ba-f92ch" Oct 29 23:31:50.802272 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:50.802257 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5be39f9d-8c3a-4450-bd67-b1a17ee56be7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-65a313ba-f92ch\" (UID: \"5be39f9d-8c3a-4450-bd67-b1a17ee56be7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65a313ba-f92ch" Oct 29 23:31:50.803747 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:50.803721 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b93153d1-59cc-4e9a-94c3-dd3c44a33d9c-kube-api-access-qjrsm" pod "b93153d1-59cc-4e9a-94c3-dd3c44a33d9c" (UID: "b93153d1-59cc-4e9a-94c3-dd3c44a33d9c"). InnerVolumeSpecName "kube-api-access-qjrsm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:31:50.903388 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:50.903354 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5be39f9d-8c3a-4450-bd67-b1a17ee56be7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-65a313ba-f92ch\" (UID: \"5be39f9d-8c3a-4450-bd67-b1a17ee56be7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65a313ba-f92ch" Oct 29 23:31:50.903562 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:50.903484 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5be39f9d-8c3a-4450-bd67-b1a17ee56be7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-65a313ba-f92ch\" (UID: \"5be39f9d-8c3a-4450-bd67-b1a17ee56be7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65a313ba-f92ch" Oct 29 23:31:50.903726 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:50.903706 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hs66h\" (UniqueName: \"kubernetes.io/projected/5be39f9d-8c3a-4450-bd67-b1a17ee56be7-kube-api-access-hs66h\") pod \"maas-api-revoke-sub-65a313ba-f92ch\" (UID: \"5be39f9d-8c3a-4450-bd67-b1a17ee56be7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65a313ba-f92ch" Oct 29 23:31:50.903815 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:50.903799 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qjrsm\" (UniqueName: \"kubernetes.io/projected/b93153d1-59cc-4e9a-94c3-dd3c44a33d9c-kube-api-access-qjrsm\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:31:50.904016 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:50.903995 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5be39f9d-8c3a-4450-bd67-b1a17ee56be7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-65a313ba-f92ch\" (UID: \"5be39f9d-8c3a-4450-bd67-b1a17ee56be7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65a313ba-f92ch" Oct 29 23:31:50.905931 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:50.905875 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5be39f9d-8c3a-4450-bd67-b1a17ee56be7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-65a313ba-f92ch\" (UID: \"5be39f9d-8c3a-4450-bd67-b1a17ee56be7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65a313ba-f92ch" Oct 29 23:31:50.926294 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:50.926272 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hs66h\" (UniqueName: \"kubernetes.io/projected/5be39f9d-8c3a-4450-bd67-b1a17ee56be7-kube-api-access-hs66h\") pod \"maas-api-revoke-sub-65a313ba-f92ch\" (UID: \"5be39f9d-8c3a-4450-bd67-b1a17ee56be7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65a313ba-f92ch" Oct 29 23:31:50.974149 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:50.974126 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65a313ba-f92ch" Oct 29 23:31:51.279870 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:51.279841 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65a313ba-f92ch"] Oct 29 23:31:51.281206 ip-10-0-140-74 kubenswrapper[5057]: W1029 23:31:51.281179 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5be39f9d_8c3a_4450_bd67_b1a17ee56be7.slice/crio-0112b0c5b305803c08c5ac2ab387dbceae1f9d9eb22abbedb5c35e3fed1bc3ed WatchSource:0}: Error finding container 0112b0c5b305803c08c5ac2ab387dbceae1f9d9eb22abbedb5c35e3fed1bc3ed: Status 404 returned error can't find the container with id 0112b0c5b305803c08c5ac2ab387dbceae1f9d9eb22abbedb5c35e3fed1bc3ed Oct 29 23:31:51.329567 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:51.329543 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06bb975e-d825-426f-b030-17bd1f99bb65" path="/var/lib/kubelet/pods/06bb975e-d825-426f-b030-17bd1f99bb65/volumes" Oct 29 23:31:51.329966 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:51.329951 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b93153d1-59cc-4e9a-94c3-dd3c44a33d9c" path="/var/lib/kubelet/pods/b93153d1-59cc-4e9a-94c3-dd3c44a33d9c/volumes" Oct 29 23:31:51.497830 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:51.495992 5057 generic.go:356] "Generic (PLEG): container finished" podID="b93153d1-59cc-4e9a-94c3-dd3c44a33d9c" containerID="db2dc4ea4f7f74a4c3963f3a45dd15814adc309c277d36b507081e8e1ceef9e4" exitCode=137 Oct 29 23:31:51.497830 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:51.496207 5057 scope.go:122] "RemoveContainer" containerID="db2dc4ea4f7f74a4c3963f3a45dd15814adc309c277d36b507081e8e1ceef9e4" Oct 29 23:31:51.497830 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:51.496323 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7596-a53d72" Oct 29 23:31:51.500348 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:51.500291 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65a313ba-f92ch" event={"ID":"5be39f9d-8c3a-4450-bd67-b1a17ee56be7","Type":"ContainerStarted","Data":"0112b0c5b305803c08c5ac2ab387dbceae1f9d9eb22abbedb5c35e3fed1bc3ed"} Oct 29 23:31:51.520046 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:51.520020 5057 scope.go:122] "RemoveContainer" containerID="db2dc4ea4f7f74a4c3963f3a45dd15814adc309c277d36b507081e8e1ceef9e4" Oct 29 23:31:51.520389 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:31:51.520365 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db2dc4ea4f7f74a4c3963f3a45dd15814adc309c277d36b507081e8e1ceef9e4\": container with ID starting with db2dc4ea4f7f74a4c3963f3a45dd15814adc309c277d36b507081e8e1ceef9e4 not found: ID does not exist" containerID="db2dc4ea4f7f74a4c3963f3a45dd15814adc309c277d36b507081e8e1ceef9e4" Oct 29 23:31:51.520475 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:51.520397 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db2dc4ea4f7f74a4c3963f3a45dd15814adc309c277d36b507081e8e1ceef9e4"} err="failed to get container status \"db2dc4ea4f7f74a4c3963f3a45dd15814adc309c277d36b507081e8e1ceef9e4\": rpc error: code = NotFound desc = could not find container \"db2dc4ea4f7f74a4c3963f3a45dd15814adc309c277d36b507081e8e1ceef9e4\": container with ID starting with db2dc4ea4f7f74a4c3963f3a45dd15814adc309c277d36b507081e8e1ceef9e4 not found: ID does not exist" Oct 29 23:31:51.749225 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:51.749196 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:31:51.861262 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:51.861229 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:31:52.011740 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:52.011704 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:31:52.509633 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:52.509587 5057 generic.go:356] "Generic (PLEG): container finished" podID="5be39f9d-8c3a-4450-bd67-b1a17ee56be7" containerID="c04d9d5cfd5ffe24fa59b80fdfcaa3eecc53080315f492554c7e18c203c5aaf7" exitCode=0 Oct 29 23:31:52.509798 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:52.509660 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65a313ba-f92ch" event={"ID":"5be39f9d-8c3a-4450-bd67-b1a17ee56be7","Type":"ContainerDied","Data":"c04d9d5cfd5ffe24fa59b80fdfcaa3eecc53080315f492554c7e18c203c5aaf7"} Oct 29 23:31:53.523255 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:53.523224 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:31:53.690174 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:53.690145 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65a313ba-f92ch" Oct 29 23:31:53.844258 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:53.844169 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5be39f9d-8c3a-4450-bd67-b1a17ee56be7-kube-api-access-hs66h\" (UniqueName: \"kubernetes.io/projected/5be39f9d-8c3a-4450-bd67-b1a17ee56be7-kube-api-access-hs66h\") pod \"5be39f9d-8c3a-4450-bd67-b1a17ee56be7\" (UID: \"5be39f9d-8c3a-4450-bd67-b1a17ee56be7\") " Oct 29 23:31:53.844425 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:53.844369 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5be39f9d-8c3a-4450-bd67-b1a17ee56be7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5be39f9d-8c3a-4450-bd67-b1a17ee56be7-maas-api-xks-ca\") pod \"5be39f9d-8c3a-4450-bd67-b1a17ee56be7\" (UID: \"5be39f9d-8c3a-4450-bd67-b1a17ee56be7\") " Oct 29 23:31:53.844482 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:53.844473 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5be39f9d-8c3a-4450-bd67-b1a17ee56be7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5be39f9d-8c3a-4450-bd67-b1a17ee56be7-maas-api-openshift-ca\") pod \"5be39f9d-8c3a-4450-bd67-b1a17ee56be7\" (UID: \"5be39f9d-8c3a-4450-bd67-b1a17ee56be7\") " Oct 29 23:31:53.844861 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:53.844830 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5be39f9d-8c3a-4450-bd67-b1a17ee56be7-maas-api-openshift-ca" pod "5be39f9d-8c3a-4450-bd67-b1a17ee56be7" (UID: "5be39f9d-8c3a-4450-bd67-b1a17ee56be7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:31:53.845178 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:53.845157 5057 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5be39f9d-8c3a-4450-bd67-b1a17ee56be7-maas-api-openshift-ca\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:31:53.846542 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:53.846516 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5be39f9d-8c3a-4450-bd67-b1a17ee56be7-maas-api-xks-ca" pod "5be39f9d-8c3a-4450-bd67-b1a17ee56be7" (UID: "5be39f9d-8c3a-4450-bd67-b1a17ee56be7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:31:53.846733 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:53.846716 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5be39f9d-8c3a-4450-bd67-b1a17ee56be7-kube-api-access-hs66h" pod "5be39f9d-8c3a-4450-bd67-b1a17ee56be7" (UID: "5be39f9d-8c3a-4450-bd67-b1a17ee56be7"). InnerVolumeSpecName "kube-api-access-hs66h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:31:53.946226 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:53.946191 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hs66h\" (UniqueName: \"kubernetes.io/projected/5be39f9d-8c3a-4450-bd67-b1a17ee56be7-kube-api-access-hs66h\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:31:53.946226 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:53.946220 5057 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5be39f9d-8c3a-4450-bd67-b1a17ee56be7-maas-api-xks-ca\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:31:54.529620 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:54.529578 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65a313ba-f92ch" event={"ID":"5be39f9d-8c3a-4450-bd67-b1a17ee56be7","Type":"ContainerDied","Data":"0112b0c5b305803c08c5ac2ab387dbceae1f9d9eb22abbedb5c35e3fed1bc3ed"} Oct 29 23:31:54.530191 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:54.529631 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65a313ba-f92ch" Oct 29 23:31:54.530191 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:54.529645 5057 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0112b0c5b305803c08c5ac2ab387dbceae1f9d9eb22abbedb5c35e3fed1bc3ed" Oct 29 23:31:54.858262 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:54.858168 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:31:55.401651 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:55.401610 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:31:55.423101 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:55.413842 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-59eefa-6867f64dbc-5fxv2" podUID="ffe9a7c2-5370-4aaf-99b8-d1d4bdad7f0f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:31:55.423101 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:55.418318 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-59eefa-bfbc68664-trphs" podUID="e5000261-99fe-4f46-8056-ffe2298d8610" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:31:55.548435 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:55.548392 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:31:55.998728 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:55.998683 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:31:57.215811 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:57.215768 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-575d59-65c58457dd-c4d2z" podUID="3bb1016d-e1c5-4e72-ab08-a063c94e3079" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:31:57.283122 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:57.283078 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:31:58.889021 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:31:58.888986 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:32:00.188260 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:00.188193 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-575d59-55c4d89464-snt8r"] Oct 29 23:32:00.189114 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:00.189050 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-575d59-55c4d89464-snt8r" podUID="90a14eb2-ffff-495b-a714-a6a950d37b7c" containerName="maas-api" containerID="cri-o://1a035f869846e51d38ca36ad761885487601310ea5871d58f4c4da835f1112aa" gracePeriod=30 Oct 29 23:32:00.253246 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:00.253209 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:32:00.532302 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:00.532273 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-575d59-55c4d89464-snt8r" Oct 29 23:32:00.628643 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:00.628604 5057 generic.go:356] "Generic (PLEG): container finished" podID="90a14eb2-ffff-495b-a714-a6a950d37b7c" containerID="1a035f869846e51d38ca36ad761885487601310ea5871d58f4c4da835f1112aa" exitCode=0 Oct 29 23:32:00.628643 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:00.628651 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-575d59-55c4d89464-snt8r" Oct 29 23:32:00.628919 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:00.628689 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-575d59-55c4d89464-snt8r" event={"ID":"90a14eb2-ffff-495b-a714-a6a950d37b7c","Type":"ContainerDied","Data":"1a035f869846e51d38ca36ad761885487601310ea5871d58f4c4da835f1112aa"} Oct 29 23:32:00.628919 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:00.628761 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-575d59-55c4d89464-snt8r" event={"ID":"90a14eb2-ffff-495b-a714-a6a950d37b7c","Type":"ContainerDied","Data":"cdf5d3e275de4435ecce4b717d26e612a50b7bb761ea1317ac822e0a78f507f0"} Oct 29 23:32:00.628919 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:00.628790 5057 scope.go:122] "RemoveContainer" containerID="1a035f869846e51d38ca36ad761885487601310ea5871d58f4c4da835f1112aa" Oct 29 23:32:00.643279 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:00.643256 5057 scope.go:122] "RemoveContainer" containerID="1a035f869846e51d38ca36ad761885487601310ea5871d58f4c4da835f1112aa" Oct 29 23:32:00.643624 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:32:00.643600 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a035f869846e51d38ca36ad761885487601310ea5871d58f4c4da835f1112aa\": container with ID starting with 1a035f869846e51d38ca36ad761885487601310ea5871d58f4c4da835f1112aa not found: ID does not exist" containerID="1a035f869846e51d38ca36ad761885487601310ea5871d58f4c4da835f1112aa" Oct 29 23:32:00.643731 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:00.643635 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a035f869846e51d38ca36ad761885487601310ea5871d58f4c4da835f1112aa"} err="failed to get container status \"1a035f869846e51d38ca36ad761885487601310ea5871d58f4c4da835f1112aa\": rpc error: code = NotFound desc = could not find container \"1a035f869846e51d38ca36ad761885487601310ea5871d58f4c4da835f1112aa\": container with ID starting with 1a035f869846e51d38ca36ad761885487601310ea5871d58f4c4da835f1112aa not found: ID does not exist" Oct 29 23:32:00.645184 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:00.645157 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/90a14eb2-ffff-495b-a714-a6a950d37b7c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/90a14eb2-ffff-495b-a714-a6a950d37b7c-metrics-certs\") pod \"90a14eb2-ffff-495b-a714-a6a950d37b7c\" (UID: \"90a14eb2-ffff-495b-a714-a6a950d37b7c\") " Oct 29 23:32:00.645304 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:00.645218 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/90a14eb2-ffff-495b-a714-a6a950d37b7c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/90a14eb2-ffff-495b-a714-a6a950d37b7c-maas-api-tls\") pod \"90a14eb2-ffff-495b-a714-a6a950d37b7c\" (UID: \"90a14eb2-ffff-495b-a714-a6a950d37b7c\") " Oct 29 23:32:00.645389 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:00.645364 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90a14eb2-ffff-495b-a714-a6a950d37b7c-kube-api-access-5z9vp\" (UniqueName: \"kubernetes.io/projected/90a14eb2-ffff-495b-a714-a6a950d37b7c-kube-api-access-5z9vp\") pod \"90a14eb2-ffff-495b-a714-a6a950d37b7c\" (UID: \"90a14eb2-ffff-495b-a714-a6a950d37b7c\") " Oct 29 23:32:00.645455 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:00.645447 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90a14eb2-ffff-495b-a714-a6a950d37b7c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/90a14eb2-ffff-495b-a714-a6a950d37b7c-openshift-service-ca\") pod \"90a14eb2-ffff-495b-a714-a6a950d37b7c\" (UID: \"90a14eb2-ffff-495b-a714-a6a950d37b7c\") " Oct 29 23:32:00.646099 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:00.646076 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90a14eb2-ffff-495b-a714-a6a950d37b7c-openshift-service-ca" pod "90a14eb2-ffff-495b-a714-a6a950d37b7c" (UID: "90a14eb2-ffff-495b-a714-a6a950d37b7c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:32:00.648436 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:00.648398 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90a14eb2-ffff-495b-a714-a6a950d37b7c-maas-api-tls" pod "90a14eb2-ffff-495b-a714-a6a950d37b7c" (UID: "90a14eb2-ffff-495b-a714-a6a950d37b7c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:32:00.648758 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:00.648738 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90a14eb2-ffff-495b-a714-a6a950d37b7c-metrics-certs" pod "90a14eb2-ffff-495b-a714-a6a950d37b7c" (UID: "90a14eb2-ffff-495b-a714-a6a950d37b7c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:32:00.652509 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:00.652484 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90a14eb2-ffff-495b-a714-a6a950d37b7c-kube-api-access-5z9vp" pod "90a14eb2-ffff-495b-a714-a6a950d37b7c" (UID: "90a14eb2-ffff-495b-a714-a6a950d37b7c"). InnerVolumeSpecName "kube-api-access-5z9vp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:32:00.742634 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:00.742552 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:32:00.747088 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:00.747059 5057 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/90a14eb2-ffff-495b-a714-a6a950d37b7c-maas-api-tls\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:32:00.747237 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:00.747091 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5z9vp\" (UniqueName: \"kubernetes.io/projected/90a14eb2-ffff-495b-a714-a6a950d37b7c-kube-api-access-5z9vp\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:32:00.747237 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:00.747105 5057 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/90a14eb2-ffff-495b-a714-a6a950d37b7c-openshift-service-ca\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:32:00.747237 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:00.747117 5057 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/90a14eb2-ffff-495b-a714-a6a950d37b7c-metrics-certs\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:32:00.866966 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:00.866925 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:32:00.986797 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:00.986760 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:32:00.986961 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:00.986850 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-575d59-55c4d89464-snt8r"] Oct 29 23:32:00.994318 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:00.993134 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-575d59-55c4d89464-snt8r"] Oct 29 23:32:01.331047 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:01.330959 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90a14eb2-ffff-495b-a714-a6a950d37b7c" path="/var/lib/kubelet/pods/90a14eb2-ffff-495b-a714-a6a950d37b7c/volumes" Oct 29 23:32:01.351918 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:01.351882 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:32:01.668785 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:01.668688 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:32:02.809944 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:02.809909 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:32:03.225524 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:03.225484 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:32:03.339422 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:03.339381 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:32:03.563127 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:03.563023 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:32:04.094949 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:04.094919 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-e170ed-z6gqx"] Oct 29 23:32:04.096132 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:04.096104 5057 cpu_manager.go:414] "containerMap: removing container" podUID="90a14eb2-ffff-495b-a714-a6a950d37b7c" containerName="maas-api" Oct 29 23:32:04.096132 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:04.096131 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90a14eb2-ffff-495b-a714-a6a950d37b7c" containerName="maas-api" Oct 29 23:32:04.096284 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:04.096162 5057 cpu_manager.go:414] "containerMap: removing container" podUID="5be39f9d-8c3a-4450-bd67-b1a17ee56be7" containerName="revoke-keys" Oct 29 23:32:04.096284 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:04.096168 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5be39f9d-8c3a-4450-bd67-b1a17ee56be7" containerName="revoke-keys" Oct 29 23:32:04.096387 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:04.096301 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="90a14eb2-ffff-495b-a714-a6a950d37b7c" containerName="maas-api" Oct 29 23:32:04.096387 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:04.096314 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="5be39f9d-8c3a-4450-bd67-b1a17ee56be7" containerName="revoke-keys" Oct 29 23:32:04.100068 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:04.100047 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-e170ed-z6gqx" Oct 29 23:32:04.117903 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:04.117876 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-e170ed-z6gqx"] Oct 29 23:32:04.125377 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:04.124842 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:32:04.125377 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:04.124890 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dmx57\"" Oct 29 23:32:04.199967 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:04.199938 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8htv4\" (UniqueName: \"kubernetes.io/projected/d377f40f-66a1-4844-a9ce-00a5ab8c3e10-kube-api-access-8htv4\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-e170ed-z6gqx\" (UID: \"d377f40f-66a1-4844-a9ce-00a5ab8c3e10\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-e170ed-z6gqx" Oct 29 23:32:04.200142 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:04.200009 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d377f40f-66a1-4844-a9ce-00a5ab8c3e10-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-e170ed-z6gqx\" (UID: \"d377f40f-66a1-4844-a9ce-00a5ab8c3e10\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-e170ed-z6gqx" Oct 29 23:32:04.200217 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:04.200147 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d377f40f-66a1-4844-a9ce-00a5ab8c3e10-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-e170ed-z6gqx\" (UID: \"d377f40f-66a1-4844-a9ce-00a5ab8c3e10\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-e170ed-z6gqx" Oct 29 23:32:04.301326 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:04.301281 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8htv4\" (UniqueName: \"kubernetes.io/projected/d377f40f-66a1-4844-a9ce-00a5ab8c3e10-kube-api-access-8htv4\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-e170ed-z6gqx\" (UID: \"d377f40f-66a1-4844-a9ce-00a5ab8c3e10\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-e170ed-z6gqx" Oct 29 23:32:04.301528 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:04.301453 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d377f40f-66a1-4844-a9ce-00a5ab8c3e10-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-e170ed-z6gqx\" (UID: \"d377f40f-66a1-4844-a9ce-00a5ab8c3e10\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-e170ed-z6gqx" Oct 29 23:32:04.302023 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:04.302000 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d377f40f-66a1-4844-a9ce-00a5ab8c3e10-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-e170ed-z6gqx\" (UID: \"d377f40f-66a1-4844-a9ce-00a5ab8c3e10\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-e170ed-z6gqx" Oct 29 23:32:04.302681 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:04.302655 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d377f40f-66a1-4844-a9ce-00a5ab8c3e10-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-e170ed-z6gqx\" (UID: \"d377f40f-66a1-4844-a9ce-00a5ab8c3e10\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-e170ed-z6gqx" Oct 29 23:32:04.304415 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:04.304382 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d377f40f-66a1-4844-a9ce-00a5ab8c3e10-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-e170ed-z6gqx\" (UID: \"d377f40f-66a1-4844-a9ce-00a5ab8c3e10\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-e170ed-z6gqx" Oct 29 23:32:04.326740 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:04.326686 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8htv4\" (UniqueName: \"kubernetes.io/projected/d377f40f-66a1-4844-a9ce-00a5ab8c3e10-kube-api-access-8htv4\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-e170ed-z6gqx\" (UID: \"d377f40f-66a1-4844-a9ce-00a5ab8c3e10\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-e170ed-z6gqx" Oct 29 23:32:04.390714 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:04.390689 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-575d59-65c58457dd-c4d2z" Oct 29 23:32:04.473913 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:04.473882 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-e170ed-z6gqx" Oct 29 23:32:04.504761 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:04.504731 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3bb1016d-e1c5-4e72-ab08-a063c94e3079-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3bb1016d-e1c5-4e72-ab08-a063c94e3079-plugins-config-volume\") pod \"3bb1016d-e1c5-4e72-ab08-a063c94e3079\" (UID: \"3bb1016d-e1c5-4e72-ab08-a063c94e3079\") " Oct 29 23:32:04.504896 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:04.504808 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3bb1016d-e1c5-4e72-ab08-a063c94e3079-kube-api-access-w7k8g\" (UniqueName: \"kubernetes.io/projected/3bb1016d-e1c5-4e72-ab08-a063c94e3079-kube-api-access-w7k8g\") pod \"3bb1016d-e1c5-4e72-ab08-a063c94e3079\" (UID: \"3bb1016d-e1c5-4e72-ab08-a063c94e3079\") " Oct 29 23:32:04.505868 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:04.505827 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3bb1016d-e1c5-4e72-ab08-a063c94e3079-plugins-config-volume" pod "3bb1016d-e1c5-4e72-ab08-a063c94e3079" (UID: "3bb1016d-e1c5-4e72-ab08-a063c94e3079"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:32:04.507498 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:04.507465 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bb1016d-e1c5-4e72-ab08-a063c94e3079-kube-api-access-w7k8g" pod "3bb1016d-e1c5-4e72-ab08-a063c94e3079" (UID: "3bb1016d-e1c5-4e72-ab08-a063c94e3079"). InnerVolumeSpecName "kube-api-access-w7k8g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:32:04.550506 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:04.550156 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:32:04.606843 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:04.606790 5057 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3bb1016d-e1c5-4e72-ab08-a063c94e3079-plugins-config-volume\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:32:04.607119 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:04.607089 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w7k8g\" (UniqueName: \"kubernetes.io/projected/3bb1016d-e1c5-4e72-ab08-a063c94e3079-kube-api-access-w7k8g\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:32:04.687496 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:04.687407 5057 generic.go:356] "Generic (PLEG): container finished" podID="3bb1016d-e1c5-4e72-ab08-a063c94e3079" containerID="859f4b40a2735141c396ace5bb8a76e1e729fbc5ccac7db116b1d72c0087416b" exitCode=0 Oct 29 23:32:04.687496 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:04.687468 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-575d59-65c58457dd-c4d2z" Oct 29 23:32:04.687496 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:04.687488 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-575d59-65c58457dd-c4d2z" event={"ID":"3bb1016d-e1c5-4e72-ab08-a063c94e3079","Type":"ContainerDied","Data":"859f4b40a2735141c396ace5bb8a76e1e729fbc5ccac7db116b1d72c0087416b"} Oct 29 23:32:04.687751 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:04.687547 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-575d59-65c58457dd-c4d2z" event={"ID":"3bb1016d-e1c5-4e72-ab08-a063c94e3079","Type":"ContainerDied","Data":"00645d9564df01312c0f0618aa9ead63ee22f4a6507f2a9141023f7b531f99e9"} Oct 29 23:32:04.687751 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:04.687575 5057 scope.go:122] "RemoveContainer" containerID="859f4b40a2735141c396ace5bb8a76e1e729fbc5ccac7db116b1d72c0087416b" Oct 29 23:32:04.699052 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:04.699029 5057 scope.go:122] "RemoveContainer" containerID="859f4b40a2735141c396ace5bb8a76e1e729fbc5ccac7db116b1d72c0087416b" Oct 29 23:32:04.699359 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:32:04.699321 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"859f4b40a2735141c396ace5bb8a76e1e729fbc5ccac7db116b1d72c0087416b\": container with ID starting with 859f4b40a2735141c396ace5bb8a76e1e729fbc5ccac7db116b1d72c0087416b not found: ID does not exist" containerID="859f4b40a2735141c396ace5bb8a76e1e729fbc5ccac7db116b1d72c0087416b" Oct 29 23:32:04.699441 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:04.699371 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"859f4b40a2735141c396ace5bb8a76e1e729fbc5ccac7db116b1d72c0087416b"} err="failed to get container status \"859f4b40a2735141c396ace5bb8a76e1e729fbc5ccac7db116b1d72c0087416b\": rpc error: code = NotFound desc = could not find container \"859f4b40a2735141c396ace5bb8a76e1e729fbc5ccac7db116b1d72c0087416b\": container with ID starting with 859f4b40a2735141c396ace5bb8a76e1e729fbc5ccac7db116b1d72c0087416b not found: ID does not exist" Oct 29 23:32:04.742369 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:04.742016 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-575d59-65c58457dd-c4d2z"] Oct 29 23:32:04.747549 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:04.747522 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-575d59-65c58457dd-c4d2z"] Oct 29 23:32:04.792691 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:04.792668 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-e170ed-z6gqx"] Oct 29 23:32:04.794364 ip-10-0-140-74 kubenswrapper[5057]: W1029 23:32:04.794341 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd377f40f_66a1_4844_a9ce_00a5ab8c3e10.slice/crio-be29ee5894d8834e8fda5f55ba04546ae30d2547e0ca1e01da982f9c7f7ba0dc WatchSource:0}: Error finding container be29ee5894d8834e8fda5f55ba04546ae30d2547e0ca1e01da982f9c7f7ba0dc: Status 404 returned error can't find the container with id be29ee5894d8834e8fda5f55ba04546ae30d2547e0ca1e01da982f9c7f7ba0dc Oct 29 23:32:05.330634 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:05.330546 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3bb1016d-e1c5-4e72-ab08-a063c94e3079" path="/var/lib/kubelet/pods/3bb1016d-e1c5-4e72-ab08-a063c94e3079/volumes" Oct 29 23:32:05.411125 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:05.411088 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-59eefa-6867f64dbc-5fxv2" podUID="ffe9a7c2-5370-4aaf-99b8-d1d4bdad7f0f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:32:05.411629 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:05.411592 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-59eefa-bfbc68664-trphs" podUID="e5000261-99fe-4f46-8056-ffe2298d8610" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:32:05.700698 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:05.700658 5057 generic.go:356] "Generic (PLEG): container finished" podID="d377f40f-66a1-4844-a9ce-00a5ab8c3e10" containerID="a1f72ddb1dca328631cd2cb107c95e5ba054b930538a40c0e63d36678e3e1848" exitCode=0 Oct 29 23:32:05.700864 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:05.700720 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-e170ed-z6gqx" event={"ID":"d377f40f-66a1-4844-a9ce-00a5ab8c3e10","Type":"ContainerDied","Data":"a1f72ddb1dca328631cd2cb107c95e5ba054b930538a40c0e63d36678e3e1848"} Oct 29 23:32:05.700864 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:05.700803 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-e170ed-z6gqx" event={"ID":"d377f40f-66a1-4844-a9ce-00a5ab8c3e10","Type":"ContainerStarted","Data":"be29ee5894d8834e8fda5f55ba04546ae30d2547e0ca1e01da982f9c7f7ba0dc"} Oct 29 23:32:06.193671 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:06.193633 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:32:06.889518 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:06.889493 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-e170ed-z6gqx" Oct 29 23:32:06.934705 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:06.934673 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d377f40f-66a1-4844-a9ce-00a5ab8c3e10-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d377f40f-66a1-4844-a9ce-00a5ab8c3e10-maas-api-openshift-ca\") pod \"d377f40f-66a1-4844-a9ce-00a5ab8c3e10\" (UID: \"d377f40f-66a1-4844-a9ce-00a5ab8c3e10\") " Oct 29 23:32:06.934862 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:06.934765 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d377f40f-66a1-4844-a9ce-00a5ab8c3e10-kube-api-access-8htv4\" (UniqueName: \"kubernetes.io/projected/d377f40f-66a1-4844-a9ce-00a5ab8c3e10-kube-api-access-8htv4\") pod \"d377f40f-66a1-4844-a9ce-00a5ab8c3e10\" (UID: \"d377f40f-66a1-4844-a9ce-00a5ab8c3e10\") " Oct 29 23:32:06.934862 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:06.934801 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d377f40f-66a1-4844-a9ce-00a5ab8c3e10-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d377f40f-66a1-4844-a9ce-00a5ab8c3e10-maas-api-xks-ca\") pod \"d377f40f-66a1-4844-a9ce-00a5ab8c3e10\" (UID: \"d377f40f-66a1-4844-a9ce-00a5ab8c3e10\") " Oct 29 23:32:06.935074 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:06.935047 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d377f40f-66a1-4844-a9ce-00a5ab8c3e10-maas-api-openshift-ca" pod "d377f40f-66a1-4844-a9ce-00a5ab8c3e10" (UID: "d377f40f-66a1-4844-a9ce-00a5ab8c3e10"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:32:06.935234 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:06.935214 5057 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d377f40f-66a1-4844-a9ce-00a5ab8c3e10-maas-api-openshift-ca\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:32:06.936924 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:06.936897 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d377f40f-66a1-4844-a9ce-00a5ab8c3e10-maas-api-xks-ca" pod "d377f40f-66a1-4844-a9ce-00a5ab8c3e10" (UID: "d377f40f-66a1-4844-a9ce-00a5ab8c3e10"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:32:06.937029 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:06.936907 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d377f40f-66a1-4844-a9ce-00a5ab8c3e10-kube-api-access-8htv4" pod "d377f40f-66a1-4844-a9ce-00a5ab8c3e10" (UID: "d377f40f-66a1-4844-a9ce-00a5ab8c3e10"). InnerVolumeSpecName "kube-api-access-8htv4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:32:07.036709 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:07.036634 5057 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d377f40f-66a1-4844-a9ce-00a5ab8c3e10-maas-api-xks-ca\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:32:07.036709 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:07.036665 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8htv4\" (UniqueName: \"kubernetes.io/projected/d377f40f-66a1-4844-a9ce-00a5ab8c3e10-kube-api-access-8htv4\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:32:07.719193 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:07.719163 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-e170ed-z6gqx" Oct 29 23:32:07.719428 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:07.719194 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-e170ed-z6gqx" event={"ID":"d377f40f-66a1-4844-a9ce-00a5ab8c3e10","Type":"ContainerDied","Data":"be29ee5894d8834e8fda5f55ba04546ae30d2547e0ca1e01da982f9c7f7ba0dc"} Oct 29 23:32:07.719428 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:07.719245 5057 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="be29ee5894d8834e8fda5f55ba04546ae30d2547e0ca1e01da982f9c7f7ba0dc" Oct 29 23:32:07.801897 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:07.801853 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:32:07.961362 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:07.956876 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:32:08.070896 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:08.070829 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:32:08.157324 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:08.157300 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-59eefa-bfbc68664-trphs" Oct 29 23:32:08.194304 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:08.194280 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-59eefa-6867f64dbc-5fxv2" Oct 29 23:32:08.254586 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:08.254552 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ffe9a7c2-5370-4aaf-99b8-d1d4bdad7f0f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ffe9a7c2-5370-4aaf-99b8-d1d4bdad7f0f-plugins-config-volume\") pod \"ffe9a7c2-5370-4aaf-99b8-d1d4bdad7f0f\" (UID: \"ffe9a7c2-5370-4aaf-99b8-d1d4bdad7f0f\") " Oct 29 23:32:08.254852 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:08.254829 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5000261-99fe-4f46-8056-ffe2298d8610-kube-api-access-xnd4v\" (UniqueName: \"kubernetes.io/projected/e5000261-99fe-4f46-8056-ffe2298d8610-kube-api-access-xnd4v\") pod \"e5000261-99fe-4f46-8056-ffe2298d8610\" (UID: \"e5000261-99fe-4f46-8056-ffe2298d8610\") " Oct 29 23:32:08.255036 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:08.255016 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ffe9a7c2-5370-4aaf-99b8-d1d4bdad7f0f-kube-api-access-p4tp8\" (UniqueName: \"kubernetes.io/projected/ffe9a7c2-5370-4aaf-99b8-d1d4bdad7f0f-kube-api-access-p4tp8\") pod \"ffe9a7c2-5370-4aaf-99b8-d1d4bdad7f0f\" (UID: \"ffe9a7c2-5370-4aaf-99b8-d1d4bdad7f0f\") " Oct 29 23:32:08.255156 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:08.255130 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ffe9a7c2-5370-4aaf-99b8-d1d4bdad7f0f-plugins-config-volume" pod "ffe9a7c2-5370-4aaf-99b8-d1d4bdad7f0f" (UID: "ffe9a7c2-5370-4aaf-99b8-d1d4bdad7f0f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:32:08.255347 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:08.255311 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5000261-99fe-4f46-8056-ffe2298d8610-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5000261-99fe-4f46-8056-ffe2298d8610-plugins-config-volume\") pod \"e5000261-99fe-4f46-8056-ffe2298d8610\" (UID: \"e5000261-99fe-4f46-8056-ffe2298d8610\") " Oct 29 23:32:08.256057 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:08.256028 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5000261-99fe-4f46-8056-ffe2298d8610-plugins-config-volume" pod "e5000261-99fe-4f46-8056-ffe2298d8610" (UID: "e5000261-99fe-4f46-8056-ffe2298d8610"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:32:08.256306 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:08.256272 5057 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ffe9a7c2-5370-4aaf-99b8-d1d4bdad7f0f-plugins-config-volume\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:32:08.279838 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:08.279756 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-k8ws5"] Oct 29 23:32:08.280645 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:08.280598 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-k8ws5" podUID="f53c1835-d1af-41c6-b8b9-e749ed7ef210" containerName="istio-proxy" containerID="cri-o://7351d791569edd66770f39ec0b0b3b6a545aac98c0cfed385b0bbb3a82f07022" gracePeriod=30 Oct 29 23:32:08.286538 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:08.286509 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffe9a7c2-5370-4aaf-99b8-d1d4bdad7f0f-kube-api-access-p4tp8" pod "ffe9a7c2-5370-4aaf-99b8-d1d4bdad7f0f" (UID: "ffe9a7c2-5370-4aaf-99b8-d1d4bdad7f0f"). InnerVolumeSpecName "kube-api-access-p4tp8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:32:08.300424 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:08.300020 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5000261-99fe-4f46-8056-ffe2298d8610-kube-api-access-xnd4v" pod "e5000261-99fe-4f46-8056-ffe2298d8610" (UID: "e5000261-99fe-4f46-8056-ffe2298d8610"). InnerVolumeSpecName "kube-api-access-xnd4v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:32:08.302175 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:08.300884 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:32:08.357772 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:08.357735 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xnd4v\" (UniqueName: \"kubernetes.io/projected/e5000261-99fe-4f46-8056-ffe2298d8610-kube-api-access-xnd4v\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:32:08.357772 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:08.357775 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p4tp8\" (UniqueName: \"kubernetes.io/projected/ffe9a7c2-5370-4aaf-99b8-d1d4bdad7f0f-kube-api-access-p4tp8\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:32:08.357977 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:08.357790 5057 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5000261-99fe-4f46-8056-ffe2298d8610-plugins-config-volume\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:32:08.730143 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:08.730096 5057 generic.go:356] "Generic (PLEG): container finished" podID="ffe9a7c2-5370-4aaf-99b8-d1d4bdad7f0f" containerID="c42625b42e3ba7dbb9afe9682f109076edf814c1cf1417439ad9f19c7b54d5d1" exitCode=0 Oct 29 23:32:08.730296 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:08.730163 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-59eefa-6867f64dbc-5fxv2" Oct 29 23:32:08.730296 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:08.730169 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-59eefa-6867f64dbc-5fxv2" event={"ID":"ffe9a7c2-5370-4aaf-99b8-d1d4bdad7f0f","Type":"ContainerDied","Data":"c42625b42e3ba7dbb9afe9682f109076edf814c1cf1417439ad9f19c7b54d5d1"} Oct 29 23:32:08.730296 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:08.730276 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-59eefa-6867f64dbc-5fxv2" event={"ID":"ffe9a7c2-5370-4aaf-99b8-d1d4bdad7f0f","Type":"ContainerDied","Data":"d64867445d9189f9dd9860236712022bcc7533fad987f7ccf59051915b6a5001"} Oct 29 23:32:08.730468 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:08.730312 5057 scope.go:122] "RemoveContainer" containerID="c42625b42e3ba7dbb9afe9682f109076edf814c1cf1417439ad9f19c7b54d5d1" Oct 29 23:32:08.732059 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:08.732032 5057 generic.go:356] "Generic (PLEG): container finished" podID="e5000261-99fe-4f46-8056-ffe2298d8610" containerID="ee3303b0e063d310fb05ed74624103eeaa7606cdf2d137b533207e6a4a3b39a2" exitCode=0 Oct 29 23:32:08.732203 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:08.732135 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-59eefa-bfbc68664-trphs" Oct 29 23:32:08.732369 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:08.732134 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-59eefa-bfbc68664-trphs" event={"ID":"e5000261-99fe-4f46-8056-ffe2298d8610","Type":"ContainerDied","Data":"ee3303b0e063d310fb05ed74624103eeaa7606cdf2d137b533207e6a4a3b39a2"} Oct 29 23:32:08.732533 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:08.732513 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-59eefa-bfbc68664-trphs" event={"ID":"e5000261-99fe-4f46-8056-ffe2298d8610","Type":"ContainerDied","Data":"38eae6eeca1efa1ec4b9322f56aebb8a2730f9c0b6eb3025b1a2b182a2b8649f"} Oct 29 23:32:08.743968 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:08.743952 5057 scope.go:122] "RemoveContainer" containerID="c42625b42e3ba7dbb9afe9682f109076edf814c1cf1417439ad9f19c7b54d5d1" Oct 29 23:32:08.744723 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:32:08.744259 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c42625b42e3ba7dbb9afe9682f109076edf814c1cf1417439ad9f19c7b54d5d1\": container with ID starting with c42625b42e3ba7dbb9afe9682f109076edf814c1cf1417439ad9f19c7b54d5d1 not found: ID does not exist" containerID="c42625b42e3ba7dbb9afe9682f109076edf814c1cf1417439ad9f19c7b54d5d1" Oct 29 23:32:08.744723 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:08.744571 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c42625b42e3ba7dbb9afe9682f109076edf814c1cf1417439ad9f19c7b54d5d1"} err="failed to get container status \"c42625b42e3ba7dbb9afe9682f109076edf814c1cf1417439ad9f19c7b54d5d1\": rpc error: code = NotFound desc = could not find container \"c42625b42e3ba7dbb9afe9682f109076edf814c1cf1417439ad9f19c7b54d5d1\": container with ID starting with c42625b42e3ba7dbb9afe9682f109076edf814c1cf1417439ad9f19c7b54d5d1 not found: ID does not exist" Oct 29 23:32:08.744723 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:08.744604 5057 scope.go:122] "RemoveContainer" containerID="ee3303b0e063d310fb05ed74624103eeaa7606cdf2d137b533207e6a4a3b39a2" Oct 29 23:32:08.759195 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:08.759174 5057 scope.go:122] "RemoveContainer" containerID="ee3303b0e063d310fb05ed74624103eeaa7606cdf2d137b533207e6a4a3b39a2" Oct 29 23:32:08.759484 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:32:08.759464 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee3303b0e063d310fb05ed74624103eeaa7606cdf2d137b533207e6a4a3b39a2\": container with ID starting with ee3303b0e063d310fb05ed74624103eeaa7606cdf2d137b533207e6a4a3b39a2 not found: ID does not exist" containerID="ee3303b0e063d310fb05ed74624103eeaa7606cdf2d137b533207e6a4a3b39a2" Oct 29 23:32:08.759598 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:08.759491 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee3303b0e063d310fb05ed74624103eeaa7606cdf2d137b533207e6a4a3b39a2"} err="failed to get container status \"ee3303b0e063d310fb05ed74624103eeaa7606cdf2d137b533207e6a4a3b39a2\": rpc error: code = NotFound desc = could not find container \"ee3303b0e063d310fb05ed74624103eeaa7606cdf2d137b533207e6a4a3b39a2\": container with ID starting with ee3303b0e063d310fb05ed74624103eeaa7606cdf2d137b533207e6a4a3b39a2 not found: ID does not exist" Oct 29 23:32:08.800808 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:08.800773 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-59eefa-6867f64dbc-5fxv2"] Oct 29 23:32:08.804511 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:08.804484 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-59eefa-6867f64dbc-5fxv2"] Oct 29 23:32:08.835500 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:08.835468 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-59eefa-bfbc68664-trphs"] Oct 29 23:32:08.838863 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:08.838842 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-59eefa-bfbc68664-trphs"] Oct 29 23:32:08.971280 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:08.971243 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-35e58700-pxt8s"] Oct 29 23:32:08.971911 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:08.971895 5057 cpu_manager.go:414] "containerMap: removing container" podUID="ffe9a7c2-5370-4aaf-99b8-d1d4bdad7f0f" containerName="payload-pre-processing" Oct 29 23:32:08.971956 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:08.971912 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ffe9a7c2-5370-4aaf-99b8-d1d4bdad7f0f" containerName="payload-pre-processing" Oct 29 23:32:08.971956 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:08.971938 5057 cpu_manager.go:414] "containerMap: removing container" podUID="d377f40f-66a1-4844-a9ce-00a5ab8c3e10" containerName="revoke-keys" Oct 29 23:32:08.971956 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:08.971943 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d377f40f-66a1-4844-a9ce-00a5ab8c3e10" containerName="revoke-keys" Oct 29 23:32:08.971956 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:08.971956 5057 cpu_manager.go:414] "containerMap: removing container" podUID="e5000261-99fe-4f46-8056-ffe2298d8610" containerName="payload-processing" Oct 29 23:32:08.972088 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:08.971961 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5000261-99fe-4f46-8056-ffe2298d8610" containerName="payload-processing" Oct 29 23:32:08.972088 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:08.971989 5057 cpu_manager.go:414] "containerMap: removing container" podUID="3bb1016d-e1c5-4e72-ab08-a063c94e3079" containerName="payload-processing" Oct 29 23:32:08.972088 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:08.971994 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3bb1016d-e1c5-4e72-ab08-a063c94e3079" containerName="payload-processing" Oct 29 23:32:08.972177 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:08.972103 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="d377f40f-66a1-4844-a9ce-00a5ab8c3e10" containerName="revoke-keys" Oct 29 23:32:08.972177 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:08.972113 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5000261-99fe-4f46-8056-ffe2298d8610" containerName="payload-processing" Oct 29 23:32:08.972177 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:08.972123 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="3bb1016d-e1c5-4e72-ab08-a063c94e3079" containerName="payload-processing" Oct 29 23:32:08.972177 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:08.972133 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="ffe9a7c2-5370-4aaf-99b8-d1d4bdad7f0f" containerName="payload-pre-processing" Oct 29 23:32:08.977084 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:08.977066 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35e58700-pxt8s" Oct 29 23:32:09.010461 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:09.010433 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-35e58700-pxt8s"] Oct 29 23:32:09.069672 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:09.069637 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5db04267-4c0f-493e-8a0d-f4d37995cd2b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-35e58700-pxt8s\" (UID: \"5db04267-4c0f-493e-8a0d-f4d37995cd2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35e58700-pxt8s" Oct 29 23:32:09.069799 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:09.069773 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5db04267-4c0f-493e-8a0d-f4d37995cd2b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-35e58700-pxt8s\" (UID: \"5db04267-4c0f-493e-8a0d-f4d37995cd2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35e58700-pxt8s" Oct 29 23:32:09.069879 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:09.069864 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdffq\" (UniqueName: \"kubernetes.io/projected/5db04267-4c0f-493e-8a0d-f4d37995cd2b-kube-api-access-hdffq\") pod \"maas-api-revoke-sub-35e58700-pxt8s\" (UID: \"5db04267-4c0f-493e-8a0d-f4d37995cd2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35e58700-pxt8s" Oct 29 23:32:09.076292 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:09.076267 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:32:09.076436 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:09.076366 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dmx57\"" Oct 29 23:32:09.171468 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:09.171437 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdffq\" (UniqueName: \"kubernetes.io/projected/5db04267-4c0f-493e-8a0d-f4d37995cd2b-kube-api-access-hdffq\") pod \"maas-api-revoke-sub-35e58700-pxt8s\" (UID: \"5db04267-4c0f-493e-8a0d-f4d37995cd2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35e58700-pxt8s" Oct 29 23:32:09.171636 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:09.171613 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5db04267-4c0f-493e-8a0d-f4d37995cd2b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-35e58700-pxt8s\" (UID: \"5db04267-4c0f-493e-8a0d-f4d37995cd2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35e58700-pxt8s" Oct 29 23:32:09.171772 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:09.171753 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5db04267-4c0f-493e-8a0d-f4d37995cd2b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-35e58700-pxt8s\" (UID: \"5db04267-4c0f-493e-8a0d-f4d37995cd2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35e58700-pxt8s" Oct 29 23:32:09.172221 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:09.172204 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5db04267-4c0f-493e-8a0d-f4d37995cd2b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-35e58700-pxt8s\" (UID: \"5db04267-4c0f-493e-8a0d-f4d37995cd2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35e58700-pxt8s" Oct 29 23:32:09.173929 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:09.173912 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5db04267-4c0f-493e-8a0d-f4d37995cd2b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-35e58700-pxt8s\" (UID: \"5db04267-4c0f-493e-8a0d-f4d37995cd2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35e58700-pxt8s" Oct 29 23:32:09.194987 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:09.194958 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdffq\" (UniqueName: \"kubernetes.io/projected/5db04267-4c0f-493e-8a0d-f4d37995cd2b-kube-api-access-hdffq\") pod \"maas-api-revoke-sub-35e58700-pxt8s\" (UID: \"5db04267-4c0f-493e-8a0d-f4d37995cd2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35e58700-pxt8s" Oct 29 23:32:09.292224 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:09.292152 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35e58700-pxt8s" Oct 29 23:32:09.332298 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:09.332271 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5000261-99fe-4f46-8056-ffe2298d8610" path="/var/lib/kubelet/pods/e5000261-99fe-4f46-8056-ffe2298d8610/volumes" Oct 29 23:32:09.332729 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:09.332713 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ffe9a7c2-5370-4aaf-99b8-d1d4bdad7f0f" path="/var/lib/kubelet/pods/ffe9a7c2-5370-4aaf-99b8-d1d4bdad7f0f/volumes" Oct 29 23:32:09.499104 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:09.499068 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:32:09.605019 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:09.604992 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-35e58700-pxt8s"] Oct 29 23:32:09.606019 ip-10-0-140-74 kubenswrapper[5057]: W1029 23:32:09.605996 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5db04267_4c0f_493e_8a0d_f4d37995cd2b.slice/crio-3f78991a170696ff9dfb5fbddd54f2806dc75f927d3b5a2bbc80a2ee8198a6d5 WatchSource:0}: Error finding container 3f78991a170696ff9dfb5fbddd54f2806dc75f927d3b5a2bbc80a2ee8198a6d5: Status 404 returned error can't find the container with id 3f78991a170696ff9dfb5fbddd54f2806dc75f927d3b5a2bbc80a2ee8198a6d5 Oct 29 23:32:09.741877 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:09.741840 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35e58700-pxt8s" event={"ID":"5db04267-4c0f-493e-8a0d-f4d37995cd2b","Type":"ContainerStarted","Data":"3f78991a170696ff9dfb5fbddd54f2806dc75f927d3b5a2bbc80a2ee8198a6d5"} Oct 29 23:32:10.754045 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:10.753956 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:32:10.755547 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:10.755516 5057 generic.go:356] "Generic (PLEG): container finished" podID="5db04267-4c0f-493e-8a0d-f4d37995cd2b" containerID="064e8eac6b31ffcaa31135332448ca690bcf50080333e8c0b70621ff1bbf3751" exitCode=0 Oct 29 23:32:10.755745 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:10.755597 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35e58700-pxt8s" event={"ID":"5db04267-4c0f-493e-8a0d-f4d37995cd2b","Type":"ContainerDied","Data":"064e8eac6b31ffcaa31135332448ca690bcf50080333e8c0b70621ff1bbf3751"} Oct 29 23:32:10.867174 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:10.867137 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:32:11.144227 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:11.144127 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:32:11.549512 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:11.549477 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:32:11.942515 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:11.942488 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35e58700-pxt8s" Oct 29 23:32:12.009755 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:12.009727 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5db04267-4c0f-493e-8a0d-f4d37995cd2b-kube-api-access-hdffq\" (UniqueName: \"kubernetes.io/projected/5db04267-4c0f-493e-8a0d-f4d37995cd2b-kube-api-access-hdffq\") pod \"5db04267-4c0f-493e-8a0d-f4d37995cd2b\" (UID: \"5db04267-4c0f-493e-8a0d-f4d37995cd2b\") " Oct 29 23:32:12.009940 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:12.009834 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5db04267-4c0f-493e-8a0d-f4d37995cd2b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5db04267-4c0f-493e-8a0d-f4d37995cd2b-maas-api-xks-ca\") pod \"5db04267-4c0f-493e-8a0d-f4d37995cd2b\" (UID: \"5db04267-4c0f-493e-8a0d-f4d37995cd2b\") " Oct 29 23:32:12.010006 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:12.009963 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5db04267-4c0f-493e-8a0d-f4d37995cd2b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5db04267-4c0f-493e-8a0d-f4d37995cd2b-maas-api-openshift-ca\") pod \"5db04267-4c0f-493e-8a0d-f4d37995cd2b\" (UID: \"5db04267-4c0f-493e-8a0d-f4d37995cd2b\") " Oct 29 23:32:12.010629 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:12.010597 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5db04267-4c0f-493e-8a0d-f4d37995cd2b-maas-api-openshift-ca" pod "5db04267-4c0f-493e-8a0d-f4d37995cd2b" (UID: "5db04267-4c0f-493e-8a0d-f4d37995cd2b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:32:12.011987 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:12.011962 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5db04267-4c0f-493e-8a0d-f4d37995cd2b-kube-api-access-hdffq" pod "5db04267-4c0f-493e-8a0d-f4d37995cd2b" (UID: "5db04267-4c0f-493e-8a0d-f4d37995cd2b"). InnerVolumeSpecName "kube-api-access-hdffq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:32:12.014441 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:12.014416 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5db04267-4c0f-493e-8a0d-f4d37995cd2b-maas-api-xks-ca" pod "5db04267-4c0f-493e-8a0d-f4d37995cd2b" (UID: "5db04267-4c0f-493e-8a0d-f4d37995cd2b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:32:12.112259 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:12.112180 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hdffq\" (UniqueName: \"kubernetes.io/projected/5db04267-4c0f-493e-8a0d-f4d37995cd2b-kube-api-access-hdffq\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:32:12.112259 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:12.112210 5057 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5db04267-4c0f-493e-8a0d-f4d37995cd2b-maas-api-xks-ca\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:32:12.112259 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:12.112221 5057 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5db04267-4c0f-493e-8a0d-f4d37995cd2b-maas-api-openshift-ca\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:32:12.707516 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:12.707459 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:32:12.780491 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:12.780457 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35e58700-pxt8s" Oct 29 23:32:12.780491 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:12.780475 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35e58700-pxt8s" event={"ID":"5db04267-4c0f-493e-8a0d-f4d37995cd2b","Type":"ContainerDied","Data":"3f78991a170696ff9dfb5fbddd54f2806dc75f927d3b5a2bbc80a2ee8198a6d5"} Oct 29 23:32:12.780702 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:12.780527 5057 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3f78991a170696ff9dfb5fbddd54f2806dc75f927d3b5a2bbc80a2ee8198a6d5" Oct 29 23:32:12.987816 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:12.987731 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v2tfm"] Oct 29 23:32:13.789659 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:13.789627 5057 generic.go:356] "Generic (PLEG): container finished" podID="f53c1835-d1af-41c6-b8b9-e749ed7ef210" containerID="7351d791569edd66770f39ec0b0b3b6a545aac98c0cfed385b0bbb3a82f07022" exitCode=0 Oct 29 23:32:13.789838 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:13.789700 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-k8ws5" event={"ID":"f53c1835-d1af-41c6-b8b9-e749ed7ef210","Type":"ContainerDied","Data":"7351d791569edd66770f39ec0b0b3b6a545aac98c0cfed385b0bbb3a82f07022"} Oct 29 23:32:13.967857 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:13.967831 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-k8ws5" Oct 29 23:32:14.052802 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:14.052708 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f53c1835-d1af-41c6-b8b9-e749ed7ef210-kube-api-access-87scs\" (UniqueName: \"kubernetes.io/projected/f53c1835-d1af-41c6-b8b9-e749ed7ef210-kube-api-access-87scs\") pod \"f53c1835-d1af-41c6-b8b9-e749ed7ef210\" (UID: \"f53c1835-d1af-41c6-b8b9-e749ed7ef210\") " Oct 29 23:32:14.052802 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:14.052765 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f53c1835-d1af-41c6-b8b9-e749ed7ef210-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f53c1835-d1af-41c6-b8b9-e749ed7ef210-istio-data\") pod \"f53c1835-d1af-41c6-b8b9-e749ed7ef210\" (UID: \"f53c1835-d1af-41c6-b8b9-e749ed7ef210\") " Oct 29 23:32:14.053301 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:14.052811 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f53c1835-d1af-41c6-b8b9-e749ed7ef210-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f53c1835-d1af-41c6-b8b9-e749ed7ef210-istio-envoy\") pod \"f53c1835-d1af-41c6-b8b9-e749ed7ef210\" (UID: \"f53c1835-d1af-41c6-b8b9-e749ed7ef210\") " Oct 29 23:32:14.053301 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:14.052852 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f53c1835-d1af-41c6-b8b9-e749ed7ef210-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f53c1835-d1af-41c6-b8b9-e749ed7ef210-istio-podinfo\") pod \"f53c1835-d1af-41c6-b8b9-e749ed7ef210\" (UID: \"f53c1835-d1af-41c6-b8b9-e749ed7ef210\") " Oct 29 23:32:14.053301 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:14.052921 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f53c1835-d1af-41c6-b8b9-e749ed7ef210-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f53c1835-d1af-41c6-b8b9-e749ed7ef210-credential-socket\") pod \"f53c1835-d1af-41c6-b8b9-e749ed7ef210\" (UID: \"f53c1835-d1af-41c6-b8b9-e749ed7ef210\") " Oct 29 23:32:14.053301 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:14.052991 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f53c1835-d1af-41c6-b8b9-e749ed7ef210-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f53c1835-d1af-41c6-b8b9-e749ed7ef210-workload-socket\") pod \"f53c1835-d1af-41c6-b8b9-e749ed7ef210\" (UID: \"f53c1835-d1af-41c6-b8b9-e749ed7ef210\") " Oct 29 23:32:14.053301 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:14.053090 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f53c1835-d1af-41c6-b8b9-e749ed7ef210-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f53c1835-d1af-41c6-b8b9-e749ed7ef210-workload-certs\") pod \"f53c1835-d1af-41c6-b8b9-e749ed7ef210\" (UID: \"f53c1835-d1af-41c6-b8b9-e749ed7ef210\") " Oct 29 23:32:14.053301 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:14.053097 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f53c1835-d1af-41c6-b8b9-e749ed7ef210-istio-data" pod "f53c1835-d1af-41c6-b8b9-e749ed7ef210" (UID: "f53c1835-d1af-41c6-b8b9-e749ed7ef210"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:32:14.053301 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:14.053291 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f53c1835-d1af-41c6-b8b9-e749ed7ef210-istio-token\" (UniqueName: \"kubernetes.io/projected/f53c1835-d1af-41c6-b8b9-e749ed7ef210-istio-token\") pod \"f53c1835-d1af-41c6-b8b9-e749ed7ef210\" (UID: \"f53c1835-d1af-41c6-b8b9-e749ed7ef210\") " Oct 29 23:32:14.054010 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:14.053754 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f53c1835-d1af-41c6-b8b9-e749ed7ef210-credential-socket" pod "f53c1835-d1af-41c6-b8b9-e749ed7ef210" (UID: "f53c1835-d1af-41c6-b8b9-e749ed7ef210"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:32:14.054010 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:14.053756 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f53c1835-d1af-41c6-b8b9-e749ed7ef210-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f53c1835-d1af-41c6-b8b9-e749ed7ef210-istiod-ca-cert\") pod \"f53c1835-d1af-41c6-b8b9-e749ed7ef210\" (UID: \"f53c1835-d1af-41c6-b8b9-e749ed7ef210\") " Oct 29 23:32:14.054010 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:14.053931 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f53c1835-d1af-41c6-b8b9-e749ed7ef210-workload-socket" pod "f53c1835-d1af-41c6-b8b9-e749ed7ef210" (UID: "f53c1835-d1af-41c6-b8b9-e749ed7ef210"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:32:14.054261 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:14.054073 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f53c1835-d1af-41c6-b8b9-e749ed7ef210-istiod-ca-cert" pod "f53c1835-d1af-41c6-b8b9-e749ed7ef210" (UID: "f53c1835-d1af-41c6-b8b9-e749ed7ef210"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:32:14.054261 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:14.054213 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f53c1835-d1af-41c6-b8b9-e749ed7ef210-workload-certs" pod "f53c1835-d1af-41c6-b8b9-e749ed7ef210" (UID: "f53c1835-d1af-41c6-b8b9-e749ed7ef210"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:32:14.057049 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:14.057022 5057 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f53c1835-d1af-41c6-b8b9-e749ed7ef210-credential-socket\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:32:14.057147 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:14.057063 5057 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f53c1835-d1af-41c6-b8b9-e749ed7ef210-workload-socket\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:32:14.057147 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:14.057079 5057 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f53c1835-d1af-41c6-b8b9-e749ed7ef210-workload-certs\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:32:14.057147 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:14.057116 5057 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f53c1835-d1af-41c6-b8b9-e749ed7ef210-istiod-ca-cert\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:32:14.057147 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:14.057131 5057 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f53c1835-d1af-41c6-b8b9-e749ed7ef210-istio-data\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:32:14.057343 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:14.057232 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f53c1835-d1af-41c6-b8b9-e749ed7ef210-istio-envoy" pod "f53c1835-d1af-41c6-b8b9-e749ed7ef210" (UID: "f53c1835-d1af-41c6-b8b9-e749ed7ef210"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:32:14.057343 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:14.057251 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f53c1835-d1af-41c6-b8b9-e749ed7ef210-kube-api-access-87scs" pod "f53c1835-d1af-41c6-b8b9-e749ed7ef210" (UID: "f53c1835-d1af-41c6-b8b9-e749ed7ef210"). InnerVolumeSpecName "kube-api-access-87scs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:32:14.057343 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:14.057277 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f53c1835-d1af-41c6-b8b9-e749ed7ef210-istio-token" pod "f53c1835-d1af-41c6-b8b9-e749ed7ef210" (UID: "f53c1835-d1af-41c6-b8b9-e749ed7ef210"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:32:14.058803 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:14.058778 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f53c1835-d1af-41c6-b8b9-e749ed7ef210-istio-podinfo" pod "f53c1835-d1af-41c6-b8b9-e749ed7ef210" (UID: "f53c1835-d1af-41c6-b8b9-e749ed7ef210"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:32:14.158237 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:14.158201 5057 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f53c1835-d1af-41c6-b8b9-e749ed7ef210-istio-token\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:32:14.158237 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:14.158234 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-87scs\" (UniqueName: \"kubernetes.io/projected/f53c1835-d1af-41c6-b8b9-e749ed7ef210-kube-api-access-87scs\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:32:14.158482 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:14.158248 5057 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f53c1835-d1af-41c6-b8b9-e749ed7ef210-istio-envoy\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:32:14.158482 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:14.158262 5057 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f53c1835-d1af-41c6-b8b9-e749ed7ef210-istio-podinfo\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:32:14.422816 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:14.422721 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:32:14.753637 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:14.753603 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:32:14.798311 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:14.798281 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-k8ws5" Oct 29 23:32:14.798487 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:14.798313 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-k8ws5" event={"ID":"f53c1835-d1af-41c6-b8b9-e749ed7ef210","Type":"ContainerDied","Data":"f90015118699f9baa22bde68e99fb77de0bb2cd118777d21ce62d47dc1d7ba8b"} Oct 29 23:32:14.798487 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:14.798385 5057 scope.go:122] "RemoveContainer" containerID="7351d791569edd66770f39ec0b0b3b6a545aac98c0cfed385b0bbb3a82f07022" Oct 29 23:32:14.853917 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:14.853870 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-k8ws5"] Oct 29 23:32:14.862682 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:14.862644 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:32:14.862826 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:14.862739 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-k8ws5"] Oct 29 23:32:15.329740 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:15.329707 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f53c1835-d1af-41c6-b8b9-e749ed7ef210" path="/var/lib/kubelet/pods/f53c1835-d1af-41c6-b8b9-e749ed7ef210/volumes" Oct 29 23:32:16.206908 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:16.206869 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:32:17.802872 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:17.802813 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:32:17.864586 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:17.864547 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-65e2d3/sub-test-model-65e2d3-kserve-554d948579-9m65w"] Oct 29 23:32:17.924124 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:17.924080 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:32:18.130831 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:18.130738 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-af237d-4kzbg"] Oct 29 23:32:18.131533 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:18.131510 5057 cpu_manager.go:414] "containerMap: removing container" podUID="5db04267-4c0f-493e-8a0d-f4d37995cd2b" containerName="revoke-keys" Oct 29 23:32:18.131533 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:18.131532 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5db04267-4c0f-493e-8a0d-f4d37995cd2b" containerName="revoke-keys" Oct 29 23:32:18.131749 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:18.131568 5057 cpu_manager.go:414] "containerMap: removing container" podUID="f53c1835-d1af-41c6-b8b9-e749ed7ef210" containerName="istio-proxy" Oct 29 23:32:18.131749 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:18.131575 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f53c1835-d1af-41c6-b8b9-e749ed7ef210" containerName="istio-proxy" Oct 29 23:32:18.131869 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:18.131780 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="f53c1835-d1af-41c6-b8b9-e749ed7ef210" containerName="istio-proxy" Oct 29 23:32:18.131869 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:18.131812 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="5db04267-4c0f-493e-8a0d-f4d37995cd2b" containerName="revoke-keys" Oct 29 23:32:18.136431 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:18.136396 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-af237d-4kzbg" Oct 29 23:32:18.151433 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:18.151408 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:32:18.151572 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:18.151518 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dmx57\"" Oct 29 23:32:18.154232 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:18.154208 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-af237d-4kzbg"] Oct 29 23:32:18.312068 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:18.312029 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/176dd45c-5cd1-452e-bbb9-3cb0ae30205e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-af237d-4kzbg\" (UID: \"176dd45c-5cd1-452e-bbb9-3cb0ae30205e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-af237d-4kzbg" Oct 29 23:32:18.312068 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:18.312074 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/176dd45c-5cd1-452e-bbb9-3cb0ae30205e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-af237d-4kzbg\" (UID: \"176dd45c-5cd1-452e-bbb9-3cb0ae30205e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-af237d-4kzbg" Oct 29 23:32:18.312321 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:18.312166 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fl889\" (UniqueName: \"kubernetes.io/projected/176dd45c-5cd1-452e-bbb9-3cb0ae30205e-kube-api-access-fl889\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-af237d-4kzbg\" (UID: \"176dd45c-5cd1-452e-bbb9-3cb0ae30205e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-af237d-4kzbg" Oct 29 23:32:18.413261 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:18.413174 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/176dd45c-5cd1-452e-bbb9-3cb0ae30205e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-af237d-4kzbg\" (UID: \"176dd45c-5cd1-452e-bbb9-3cb0ae30205e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-af237d-4kzbg" Oct 29 23:32:18.413261 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:18.413215 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/176dd45c-5cd1-452e-bbb9-3cb0ae30205e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-af237d-4kzbg\" (UID: \"176dd45c-5cd1-452e-bbb9-3cb0ae30205e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-af237d-4kzbg" Oct 29 23:32:18.413261 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:18.413258 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fl889\" (UniqueName: \"kubernetes.io/projected/176dd45c-5cd1-452e-bbb9-3cb0ae30205e-kube-api-access-fl889\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-af237d-4kzbg\" (UID: \"176dd45c-5cd1-452e-bbb9-3cb0ae30205e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-af237d-4kzbg" Oct 29 23:32:18.414044 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:18.414020 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/176dd45c-5cd1-452e-bbb9-3cb0ae30205e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-af237d-4kzbg\" (UID: \"176dd45c-5cd1-452e-bbb9-3cb0ae30205e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-af237d-4kzbg" Oct 29 23:32:18.416143 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:18.416110 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/176dd45c-5cd1-452e-bbb9-3cb0ae30205e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-af237d-4kzbg\" (UID: \"176dd45c-5cd1-452e-bbb9-3cb0ae30205e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-af237d-4kzbg" Oct 29 23:32:18.436424 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:18.436397 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fl889\" (UniqueName: \"kubernetes.io/projected/176dd45c-5cd1-452e-bbb9-3cb0ae30205e-kube-api-access-fl889\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-af237d-4kzbg\" (UID: \"176dd45c-5cd1-452e-bbb9-3cb0ae30205e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-af237d-4kzbg" Oct 29 23:32:18.451296 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:18.451264 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-af237d-4kzbg" Oct 29 23:32:18.767684 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:18.767653 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-af237d-4kzbg"] Oct 29 23:32:18.769601 ip-10-0-140-74 kubenswrapper[5057]: W1029 23:32:18.769572 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod176dd45c_5cd1_452e_bbb9_3cb0ae30205e.slice/crio-d81c1516ead5029d0147ddec7eab144b827ef4da8bb8765171b81c4b9cfac64e WatchSource:0}: Error finding container d81c1516ead5029d0147ddec7eab144b827ef4da8bb8765171b81c4b9cfac64e: Status 404 returned error can't find the container with id d81c1516ead5029d0147ddec7eab144b827ef4da8bb8765171b81c4b9cfac64e Oct 29 23:32:18.835310 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:18.835280 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-af237d-4kzbg" event={"ID":"176dd45c-5cd1-452e-bbb9-3cb0ae30205e","Type":"ContainerStarted","Data":"d81c1516ead5029d0147ddec7eab144b827ef4da8bb8765171b81c4b9cfac64e"} Oct 29 23:32:19.028184 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:32:19.028112 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d2dnn" containerName="extractor" Oct 29 23:32:19.510047 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:19.510013 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:32:19.845785 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:19.845707 5057 generic.go:356] "Generic (PLEG): container finished" podID="176dd45c-5cd1-452e-bbb9-3cb0ae30205e" containerID="bde860da99db81642553e4b2ac9715e89ead5d7b47603477d76fd62a4366de61" exitCode=0 Oct 29 23:32:19.846145 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:19.845792 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-af237d-4kzbg" event={"ID":"176dd45c-5cd1-452e-bbb9-3cb0ae30205e","Type":"ContainerDied","Data":"bde860da99db81642553e4b2ac9715e89ead5d7b47603477d76fd62a4366de61"} Oct 29 23:32:21.022823 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:21.022796 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-af237d-4kzbg" Oct 29 23:32:21.151172 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:21.151092 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/176dd45c-5cd1-452e-bbb9-3cb0ae30205e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/176dd45c-5cd1-452e-bbb9-3cb0ae30205e-maas-api-openshift-ca\") pod \"176dd45c-5cd1-452e-bbb9-3cb0ae30205e\" (UID: \"176dd45c-5cd1-452e-bbb9-3cb0ae30205e\") " Oct 29 23:32:21.151364 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:21.151249 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/176dd45c-5cd1-452e-bbb9-3cb0ae30205e-kube-api-access-fl889\" (UniqueName: \"kubernetes.io/projected/176dd45c-5cd1-452e-bbb9-3cb0ae30205e-kube-api-access-fl889\") pod \"176dd45c-5cd1-452e-bbb9-3cb0ae30205e\" (UID: \"176dd45c-5cd1-452e-bbb9-3cb0ae30205e\") " Oct 29 23:32:21.151364 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:21.151326 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/176dd45c-5cd1-452e-bbb9-3cb0ae30205e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/176dd45c-5cd1-452e-bbb9-3cb0ae30205e-maas-api-xks-ca\") pod \"176dd45c-5cd1-452e-bbb9-3cb0ae30205e\" (UID: \"176dd45c-5cd1-452e-bbb9-3cb0ae30205e\") " Oct 29 23:32:21.151534 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:21.151506 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/176dd45c-5cd1-452e-bbb9-3cb0ae30205e-maas-api-openshift-ca" pod "176dd45c-5cd1-452e-bbb9-3cb0ae30205e" (UID: "176dd45c-5cd1-452e-bbb9-3cb0ae30205e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:32:21.151843 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:21.151825 5057 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/176dd45c-5cd1-452e-bbb9-3cb0ae30205e-maas-api-openshift-ca\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:32:21.153412 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:21.153388 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/176dd45c-5cd1-452e-bbb9-3cb0ae30205e-maas-api-xks-ca" pod "176dd45c-5cd1-452e-bbb9-3cb0ae30205e" (UID: "176dd45c-5cd1-452e-bbb9-3cb0ae30205e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:32:21.153511 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:21.153491 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/176dd45c-5cd1-452e-bbb9-3cb0ae30205e-kube-api-access-fl889" pod "176dd45c-5cd1-452e-bbb9-3cb0ae30205e" (UID: "176dd45c-5cd1-452e-bbb9-3cb0ae30205e"). InnerVolumeSpecName "kube-api-access-fl889". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:32:21.209874 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:21.209845 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:32:21.253684 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:21.253638 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fl889\" (UniqueName: \"kubernetes.io/projected/176dd45c-5cd1-452e-bbb9-3cb0ae30205e-kube-api-access-fl889\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:32:21.253684 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:21.253688 5057 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/176dd45c-5cd1-452e-bbb9-3cb0ae30205e-maas-api-xks-ca\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:32:21.863059 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:21.863029 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-af237d-4kzbg" Oct 29 23:32:21.863251 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:21.863025 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-af237d-4kzbg" event={"ID":"176dd45c-5cd1-452e-bbb9-3cb0ae30205e","Type":"ContainerDied","Data":"d81c1516ead5029d0147ddec7eab144b827ef4da8bb8765171b81c4b9cfac64e"} Oct 29 23:32:21.863251 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:21.863154 5057 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d81c1516ead5029d0147ddec7eab144b827ef4da8bb8765171b81c4b9cfac64e" Oct 29 23:32:22.897743 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:22.897681 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:32:24.400559 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:24.400519 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:32:25.768196 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:25.768147 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:32:26.273415 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:26.273378 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:32:26.390200 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:26.390167 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-e170ed-z6gqx"] Oct 29 23:32:26.418650 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:26.418610 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:32:26.426956 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:26.426924 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-e170ed-z6gqx"] Oct 29 23:32:26.545096 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:26.545007 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:32:26.657478 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:26.657440 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:32:26.871004 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:26.870910 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:32:27.329353 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:27.329311 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d377f40f-66a1-4844-a9ce-00a5ab8c3e10" path="/var/lib/kubelet/pods/d377f40f-66a1-4844-a9ce-00a5ab8c3e10/volumes" Oct 29 23:32:28.104286 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:28.104231 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:32:28.746737 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:28.746703 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:32:28.872571 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:28.872534 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:32:29.731926 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:29.731894 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:32:30.950454 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:30.950422 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-4504f1/sub-test-model-4504f1-kserve-657f694cf7-lqr5k"] Oct 29 23:32:31.001595 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:31.001564 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:32:31.116044 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:31.116003 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:32:31.736901 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:31.736864 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:32:31.830571 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:31.830533 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-af237d-4kzbg"] Oct 29 23:32:31.842080 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:31.842049 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-af237d-4kzbg"] Oct 29 23:32:31.920595 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:31.920551 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:32:32.307584 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:32.307533 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:32:32.885893 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:32.885854 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w4-65e2d3/sub-test-model-65e2d3-kserve-554d948579-9m65w" podUID="aa71d8ce-5a91-4d79-bd0a-8f9997a3c9c4" containerName="main" containerID="cri-o://2251a2a475e5968d2edc5ceabbe001d7c7896943fec54c739ac2df776dc22161" gracePeriod=45 Oct 29 23:32:33.175088 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:33.175063 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-65e2d3/sub-test-model-65e2d3-kserve-554d948579-9m65w" Oct 29 23:32:33.214151 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:33.214118 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa71d8ce-5a91-4d79-bd0a-8f9997a3c9c4-home\" (UniqueName: \"kubernetes.io/empty-dir/aa71d8ce-5a91-4d79-bd0a-8f9997a3c9c4-home\") pod \"aa71d8ce-5a91-4d79-bd0a-8f9997a3c9c4\" (UID: \"aa71d8ce-5a91-4d79-bd0a-8f9997a3c9c4\") " Oct 29 23:32:33.214304 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:33.214180 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa71d8ce-5a91-4d79-bd0a-8f9997a3c9c4-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa71d8ce-5a91-4d79-bd0a-8f9997a3c9c4-tmp-dir\") pod \"aa71d8ce-5a91-4d79-bd0a-8f9997a3c9c4\" (UID: \"aa71d8ce-5a91-4d79-bd0a-8f9997a3c9c4\") " Oct 29 23:32:33.214304 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:33.214217 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa71d8ce-5a91-4d79-bd0a-8f9997a3c9c4-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa71d8ce-5a91-4d79-bd0a-8f9997a3c9c4-model-cache\") pod \"aa71d8ce-5a91-4d79-bd0a-8f9997a3c9c4\" (UID: \"aa71d8ce-5a91-4d79-bd0a-8f9997a3c9c4\") " Oct 29 23:32:33.214304 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:33.214288 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa71d8ce-5a91-4d79-bd0a-8f9997a3c9c4-kube-api-access-5rqd7\" (UniqueName: \"kubernetes.io/projected/aa71d8ce-5a91-4d79-bd0a-8f9997a3c9c4-kube-api-access-5rqd7\") pod \"aa71d8ce-5a91-4d79-bd0a-8f9997a3c9c4\" (UID: \"aa71d8ce-5a91-4d79-bd0a-8f9997a3c9c4\") " Oct 29 23:32:33.214527 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:33.214393 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aa71d8ce-5a91-4d79-bd0a-8f9997a3c9c4-tls-certs\" (UniqueName: \"kubernetes.io/secret/aa71d8ce-5a91-4d79-bd0a-8f9997a3c9c4-tls-certs\") pod \"aa71d8ce-5a91-4d79-bd0a-8f9997a3c9c4\" (UID: \"aa71d8ce-5a91-4d79-bd0a-8f9997a3c9c4\") " Oct 29 23:32:33.214527 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:33.214424 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa71d8ce-5a91-4d79-bd0a-8f9997a3c9c4-dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa71d8ce-5a91-4d79-bd0a-8f9997a3c9c4-dshm\") pod \"aa71d8ce-5a91-4d79-bd0a-8f9997a3c9c4\" (UID: \"aa71d8ce-5a91-4d79-bd0a-8f9997a3c9c4\") " Oct 29 23:32:33.214527 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:33.214416 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa71d8ce-5a91-4d79-bd0a-8f9997a3c9c4-home" pod "aa71d8ce-5a91-4d79-bd0a-8f9997a3c9c4" (UID: "aa71d8ce-5a91-4d79-bd0a-8f9997a3c9c4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:32:33.214527 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:33.214432 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa71d8ce-5a91-4d79-bd0a-8f9997a3c9c4-tmp-dir" pod "aa71d8ce-5a91-4d79-bd0a-8f9997a3c9c4" (UID: "aa71d8ce-5a91-4d79-bd0a-8f9997a3c9c4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:32:33.214527 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:33.214477 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa71d8ce-5a91-4d79-bd0a-8f9997a3c9c4-model-cache" pod "aa71d8ce-5a91-4d79-bd0a-8f9997a3c9c4" (UID: "aa71d8ce-5a91-4d79-bd0a-8f9997a3c9c4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:32:33.214953 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:33.214932 5057 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa71d8ce-5a91-4d79-bd0a-8f9997a3c9c4-home\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:32:33.214953 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:33.214951 5057 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa71d8ce-5a91-4d79-bd0a-8f9997a3c9c4-tmp-dir\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:32:33.215119 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:33.214960 5057 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa71d8ce-5a91-4d79-bd0a-8f9997a3c9c4-model-cache\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:32:33.217060 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:33.216998 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa71d8ce-5a91-4d79-bd0a-8f9997a3c9c4-tls-certs" pod "aa71d8ce-5a91-4d79-bd0a-8f9997a3c9c4" (UID: "aa71d8ce-5a91-4d79-bd0a-8f9997a3c9c4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:32:33.217210 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:33.217189 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa71d8ce-5a91-4d79-bd0a-8f9997a3c9c4-dshm" pod "aa71d8ce-5a91-4d79-bd0a-8f9997a3c9c4" (UID: "aa71d8ce-5a91-4d79-bd0a-8f9997a3c9c4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:32:33.217281 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:33.217220 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa71d8ce-5a91-4d79-bd0a-8f9997a3c9c4-kube-api-access-5rqd7" pod "aa71d8ce-5a91-4d79-bd0a-8f9997a3c9c4" (UID: "aa71d8ce-5a91-4d79-bd0a-8f9997a3c9c4"). InnerVolumeSpecName "kube-api-access-5rqd7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:32:33.332360 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:33.316778 5057 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa71d8ce-5a91-4d79-bd0a-8f9997a3c9c4-tls-certs\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:32:33.332360 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:33.316821 5057 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa71d8ce-5a91-4d79-bd0a-8f9997a3c9c4-dshm\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:32:33.332360 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:33.316837 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5rqd7\" (UniqueName: \"kubernetes.io/projected/aa71d8ce-5a91-4d79-bd0a-8f9997a3c9c4-kube-api-access-5rqd7\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:32:33.338088 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:33.338050 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="176dd45c-5cd1-452e-bbb9-3cb0ae30205e" path="/var/lib/kubelet/pods/176dd45c-5cd1-452e-bbb9-3cb0ae30205e/volumes" Oct 29 23:32:33.619350 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:33.619070 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:32:33.882081 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:33.881996 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:32:33.971785 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:33.971744 5057 generic.go:356] "Generic (PLEG): container finished" podID="aa71d8ce-5a91-4d79-bd0a-8f9997a3c9c4" containerID="2251a2a475e5968d2edc5ceabbe001d7c7896943fec54c739ac2df776dc22161" exitCode=0 Oct 29 23:32:33.971987 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:33.971789 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-65e2d3/sub-test-model-65e2d3-kserve-554d948579-9m65w" Oct 29 23:32:33.971987 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:33.971830 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-65e2d3/sub-test-model-65e2d3-kserve-554d948579-9m65w" event={"ID":"aa71d8ce-5a91-4d79-bd0a-8f9997a3c9c4","Type":"ContainerDied","Data":"2251a2a475e5968d2edc5ceabbe001d7c7896943fec54c739ac2df776dc22161"} Oct 29 23:32:33.971987 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:33.971876 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-65e2d3/sub-test-model-65e2d3-kserve-554d948579-9m65w" event={"ID":"aa71d8ce-5a91-4d79-bd0a-8f9997a3c9c4","Type":"ContainerDied","Data":"3093d202840bfa750b1df20c96977d7d18ab94cac37e0b297bc102947dc66df1"} Oct 29 23:32:33.971987 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:33.971894 5057 scope.go:122] "RemoveContainer" containerID="2251a2a475e5968d2edc5ceabbe001d7c7896943fec54c739ac2df776dc22161" Oct 29 23:32:33.984222 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:33.984169 5057 scope.go:122] "RemoveContainer" containerID="2251a2a475e5968d2edc5ceabbe001d7c7896943fec54c739ac2df776dc22161" Oct 29 23:32:33.984560 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:32:33.984528 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2251a2a475e5968d2edc5ceabbe001d7c7896943fec54c739ac2df776dc22161\": container with ID starting with 2251a2a475e5968d2edc5ceabbe001d7c7896943fec54c739ac2df776dc22161 not found: ID does not exist" containerID="2251a2a475e5968d2edc5ceabbe001d7c7896943fec54c739ac2df776dc22161" Oct 29 23:32:33.984655 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:33.984568 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2251a2a475e5968d2edc5ceabbe001d7c7896943fec54c739ac2df776dc22161"} err="failed to get container status \"2251a2a475e5968d2edc5ceabbe001d7c7896943fec54c739ac2df776dc22161\": rpc error: code = NotFound desc = could not find container \"2251a2a475e5968d2edc5ceabbe001d7c7896943fec54c739ac2df776dc22161\": container with ID starting with 2251a2a475e5968d2edc5ceabbe001d7c7896943fec54c739ac2df776dc22161 not found: ID does not exist" Oct 29 23:32:33.992916 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:33.992888 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:32:34.026184 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:34.026157 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-65e2d3/sub-test-model-65e2d3-kserve-554d948579-9m65w"] Oct 29 23:32:34.036892 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:34.036353 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w4-65e2d3/sub-test-model-65e2d3-kserve-554d948579-9m65w"] Oct 29 23:32:34.040136 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:34.040109 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dcc10992-4pss8"] Oct 29 23:32:34.043166 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:34.043145 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dcc10992-4pss8"] Oct 29 23:32:34.220583 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:34.220543 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:32:34.721002 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:34.720967 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:32:35.333456 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:35.333419 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5a9fd37-8a4a-4703-9114-ca13cac40108" path="/var/lib/kubelet/pods/a5a9fd37-8a4a-4703-9114-ca13cac40108/volumes" Oct 29 23:32:35.334078 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:35.334058 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa71d8ce-5a91-4d79-bd0a-8f9997a3c9c4" path="/var/lib/kubelet/pods/aa71d8ce-5a91-4d79-bd0a-8f9997a3c9c4/volumes" Oct 29 23:32:35.430677 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:35.430640 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:32:35.879038 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:35.879003 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-67363b/sub-test-model-67363b-kserve-54bb84cd5f-dpl7n"] Oct 29 23:32:35.917348 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:35.916901 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:32:36.897513 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:36.897461 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:32:37.104518 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:37.104465 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:32:38.614326 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:38.614282 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:32:39.416240 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:39.416204 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:32:39.534249 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:39.534213 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:32:39.823839 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:39.823805 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-qs6cg"] Oct 29 23:32:39.824236 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:39.824008 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-qs6cg" podUID="b9d46ee5-61dd-4a34-93bc-bcf127bb983a" containerName="istio-proxy" containerID="cri-o://593966f5ad6e4a9a0b51acf162b3273a294c35c1175ba5f00d1e84541724fee7" gracePeriod=30 Oct 29 23:32:39.902495 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:39.902433 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:32:40.111825 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:40.111709 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:32:40.409221 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:40.409127 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:32:42.127324 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:42.127276 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:32:43.801912 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:43.801872 5057 patch_prober.go:29] interesting pod/e2e-shared-a-w4-af237d-openshift-default-64f945b797-qs6cg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 23:32:43.802401 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:43.801945 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-qs6cg" podUID="b9d46ee5-61dd-4a34-93bc-bcf127bb983a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 23:32:43.862933 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:43.862904 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:32:44.714050 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:44.714006 5057 scope.go:122] "RemoveContainer" containerID="1c9205531c5cdc0460fb0cd069a46026bdb0d8e482c58a02ff1dcd97d2996f1b" Oct 29 23:32:44.724959 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:44.724924 5057 scope.go:122] "RemoveContainer" containerID="3dd80ee3e0a5d80a630f55b79ec459e16e2a890babd9b1c6368bc8cb1360d9b6" Oct 29 23:32:44.734780 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:44.734749 5057 scope.go:122] "RemoveContainer" containerID="2aa3748d9c17668fdf2b746b7c8bd3a199c63187f15fa925fd200885e2de4744" Oct 29 23:32:44.743825 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:44.743797 5057 scope.go:122] "RemoveContainer" containerID="36739b33ecc51a349147d6192bb8cf0f89e26939c04d2a125105ec2d1b6027a8" Oct 29 23:32:44.753216 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:44.753200 5057 scope.go:122] "RemoveContainer" containerID="11461d8af346eecf8f180f312bb29405a39f5fc1a63f366784be0cb441e5f349" Oct 29 23:32:44.762503 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:44.762487 5057 scope.go:122] "RemoveContainer" containerID="9fc0bb2728b43e98c9e96557107c448d5860a2b2fd14e4c5a91e72db5ac4c269" Oct 29 23:32:45.081819 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:45.081786 5057 generic.go:356] "Generic (PLEG): container finished" podID="b9d46ee5-61dd-4a34-93bc-bcf127bb983a" containerID="593966f5ad6e4a9a0b51acf162b3273a294c35c1175ba5f00d1e84541724fee7" exitCode=0 Oct 29 23:32:45.082272 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:45.081856 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-qs6cg" event={"ID":"b9d46ee5-61dd-4a34-93bc-bcf127bb983a","Type":"ContainerDied","Data":"593966f5ad6e4a9a0b51acf162b3273a294c35c1175ba5f00d1e84541724fee7"} Oct 29 23:32:45.111206 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:45.111183 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-qs6cg" Oct 29 23:32:45.198383 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:45.198345 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9d46ee5-61dd-4a34-93bc-bcf127bb983a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9d46ee5-61dd-4a34-93bc-bcf127bb983a-istio-data\") pod \"b9d46ee5-61dd-4a34-93bc-bcf127bb983a\" (UID: \"b9d46ee5-61dd-4a34-93bc-bcf127bb983a\") " Oct 29 23:32:45.198904 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:45.198423 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9d46ee5-61dd-4a34-93bc-bcf127bb983a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9d46ee5-61dd-4a34-93bc-bcf127bb983a-credential-socket\") pod \"b9d46ee5-61dd-4a34-93bc-bcf127bb983a\" (UID: \"b9d46ee5-61dd-4a34-93bc-bcf127bb983a\") " Oct 29 23:32:45.198904 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:45.198470 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9d46ee5-61dd-4a34-93bc-bcf127bb983a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9d46ee5-61dd-4a34-93bc-bcf127bb983a-workload-socket\") pod \"b9d46ee5-61dd-4a34-93bc-bcf127bb983a\" (UID: \"b9d46ee5-61dd-4a34-93bc-bcf127bb983a\") " Oct 29 23:32:45.198904 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:45.198547 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9d46ee5-61dd-4a34-93bc-bcf127bb983a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9d46ee5-61dd-4a34-93bc-bcf127bb983a-istio-envoy\") pod \"b9d46ee5-61dd-4a34-93bc-bcf127bb983a\" (UID: \"b9d46ee5-61dd-4a34-93bc-bcf127bb983a\") " Oct 29 23:32:45.198904 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:45.198611 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9d46ee5-61dd-4a34-93bc-bcf127bb983a-istio-data" pod "b9d46ee5-61dd-4a34-93bc-bcf127bb983a" (UID: "b9d46ee5-61dd-4a34-93bc-bcf127bb983a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:32:45.198904 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:45.198606 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b9d46ee5-61dd-4a34-93bc-bcf127bb983a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9d46ee5-61dd-4a34-93bc-bcf127bb983a-istio-podinfo\") pod \"b9d46ee5-61dd-4a34-93bc-bcf127bb983a\" (UID: \"b9d46ee5-61dd-4a34-93bc-bcf127bb983a\") " Oct 29 23:32:45.199194 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:45.199046 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9d46ee5-61dd-4a34-93bc-bcf127bb983a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9d46ee5-61dd-4a34-93bc-bcf127bb983a-workload-certs\") pod \"b9d46ee5-61dd-4a34-93bc-bcf127bb983a\" (UID: \"b9d46ee5-61dd-4a34-93bc-bcf127bb983a\") " Oct 29 23:32:45.199194 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:45.198767 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9d46ee5-61dd-4a34-93bc-bcf127bb983a-credential-socket" pod "b9d46ee5-61dd-4a34-93bc-bcf127bb983a" (UID: "b9d46ee5-61dd-4a34-93bc-bcf127bb983a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:32:45.199194 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:45.198801 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9d46ee5-61dd-4a34-93bc-bcf127bb983a-workload-socket" pod "b9d46ee5-61dd-4a34-93bc-bcf127bb983a" (UID: "b9d46ee5-61dd-4a34-93bc-bcf127bb983a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:32:45.199444 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:45.199306 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9d46ee5-61dd-4a34-93bc-bcf127bb983a-workload-certs" pod "b9d46ee5-61dd-4a34-93bc-bcf127bb983a" (UID: "b9d46ee5-61dd-4a34-93bc-bcf127bb983a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:32:45.199444 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:45.199363 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9d46ee5-61dd-4a34-93bc-bcf127bb983a-kube-api-access-gw879\" (UniqueName: \"kubernetes.io/projected/b9d46ee5-61dd-4a34-93bc-bcf127bb983a-kube-api-access-gw879\") pod \"b9d46ee5-61dd-4a34-93bc-bcf127bb983a\" (UID: \"b9d46ee5-61dd-4a34-93bc-bcf127bb983a\") " Oct 29 23:32:45.199575 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:45.199446 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9d46ee5-61dd-4a34-93bc-bcf127bb983a-istio-token\" (UniqueName: \"kubernetes.io/projected/b9d46ee5-61dd-4a34-93bc-bcf127bb983a-istio-token\") pod \"b9d46ee5-61dd-4a34-93bc-bcf127bb983a\" (UID: \"b9d46ee5-61dd-4a34-93bc-bcf127bb983a\") " Oct 29 23:32:45.199887 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:45.199863 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b9d46ee5-61dd-4a34-93bc-bcf127bb983a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9d46ee5-61dd-4a34-93bc-bcf127bb983a-istiod-ca-cert\") pod \"b9d46ee5-61dd-4a34-93bc-bcf127bb983a\" (UID: \"b9d46ee5-61dd-4a34-93bc-bcf127bb983a\") " Oct 29 23:32:45.200238 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:45.200217 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9d46ee5-61dd-4a34-93bc-bcf127bb983a-istiod-ca-cert" pod "b9d46ee5-61dd-4a34-93bc-bcf127bb983a" (UID: "b9d46ee5-61dd-4a34-93bc-bcf127bb983a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:32:45.200756 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:45.200732 5057 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9d46ee5-61dd-4a34-93bc-bcf127bb983a-workload-certs\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:32:45.200860 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:45.200763 5057 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9d46ee5-61dd-4a34-93bc-bcf127bb983a-istiod-ca-cert\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:32:45.200860 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:45.200777 5057 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9d46ee5-61dd-4a34-93bc-bcf127bb983a-istio-data\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:32:45.200860 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:45.200790 5057 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9d46ee5-61dd-4a34-93bc-bcf127bb983a-credential-socket\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:32:45.200860 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:45.200803 5057 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9d46ee5-61dd-4a34-93bc-bcf127bb983a-workload-socket\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:32:45.202367 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:45.201579 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9d46ee5-61dd-4a34-93bc-bcf127bb983a-istio-envoy" pod "b9d46ee5-61dd-4a34-93bc-bcf127bb983a" (UID: "b9d46ee5-61dd-4a34-93bc-bcf127bb983a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:32:45.204192 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:45.204166 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9d46ee5-61dd-4a34-93bc-bcf127bb983a-istio-token" pod "b9d46ee5-61dd-4a34-93bc-bcf127bb983a" (UID: "b9d46ee5-61dd-4a34-93bc-bcf127bb983a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:32:45.204462 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:45.204438 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9d46ee5-61dd-4a34-93bc-bcf127bb983a-kube-api-access-gw879" pod "b9d46ee5-61dd-4a34-93bc-bcf127bb983a" (UID: "b9d46ee5-61dd-4a34-93bc-bcf127bb983a"). InnerVolumeSpecName "kube-api-access-gw879". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:32:45.208565 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:45.208538 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b9d46ee5-61dd-4a34-93bc-bcf127bb983a-istio-podinfo" pod "b9d46ee5-61dd-4a34-93bc-bcf127bb983a" (UID: "b9d46ee5-61dd-4a34-93bc-bcf127bb983a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:32:45.301915 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:45.301828 5057 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9d46ee5-61dd-4a34-93bc-bcf127bb983a-istio-envoy\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:32:45.301915 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:45.301863 5057 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9d46ee5-61dd-4a34-93bc-bcf127bb983a-istio-podinfo\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:32:45.301915 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:45.301876 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gw879\" (UniqueName: \"kubernetes.io/projected/b9d46ee5-61dd-4a34-93bc-bcf127bb983a-kube-api-access-gw879\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:32:45.301915 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:45.301890 5057 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9d46ee5-61dd-4a34-93bc-bcf127bb983a-istio-token\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:32:45.512323 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:45.512290 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:32:45.628201 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:45.628041 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:32:45.979771 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:45.979728 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w0-4504f1/sub-test-model-4504f1-kserve-657f694cf7-lqr5k" podUID="630519d9-72be-441c-b013-eff2d5f360cc" containerName="main" containerID="cri-o://6adf4ca5f058014e45234e783c824b561ea789ec1c95e6b88930111388b332e6" gracePeriod=45 Oct 29 23:32:46.093597 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:46.093551 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-qs6cg" event={"ID":"b9d46ee5-61dd-4a34-93bc-bcf127bb983a","Type":"ContainerDied","Data":"7f2774f799d6cca60c8cbf02bf4f03fce9423905204d738a3788c876737671b0"} Oct 29 23:32:46.094082 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:46.093630 5057 scope.go:122] "RemoveContainer" containerID="593966f5ad6e4a9a0b51acf162b3273a294c35c1175ba5f00d1e84541724fee7" Oct 29 23:32:46.094082 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:46.093570 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-qs6cg" Oct 29 23:32:46.148144 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:46.148095 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-qs6cg"] Oct 29 23:32:46.160573 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:46.160504 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-qs6cg"] Oct 29 23:32:46.263197 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:46.263169 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-4504f1/sub-test-model-4504f1-kserve-657f694cf7-lqr5k" Oct 29 23:32:46.417058 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:46.417025 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/630519d9-72be-441c-b013-eff2d5f360cc-dshm\" (UniqueName: \"kubernetes.io/empty-dir/630519d9-72be-441c-b013-eff2d5f360cc-dshm\") pod \"630519d9-72be-441c-b013-eff2d5f360cc\" (UID: \"630519d9-72be-441c-b013-eff2d5f360cc\") " Oct 29 23:32:46.417251 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:46.417088 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/630519d9-72be-441c-b013-eff2d5f360cc-home\" (UniqueName: \"kubernetes.io/empty-dir/630519d9-72be-441c-b013-eff2d5f360cc-home\") pod \"630519d9-72be-441c-b013-eff2d5f360cc\" (UID: \"630519d9-72be-441c-b013-eff2d5f360cc\") " Oct 29 23:32:46.417251 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:46.417131 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/630519d9-72be-441c-b013-eff2d5f360cc-kube-api-access-5d8q5\" (UniqueName: \"kubernetes.io/projected/630519d9-72be-441c-b013-eff2d5f360cc-kube-api-access-5d8q5\") pod \"630519d9-72be-441c-b013-eff2d5f360cc\" (UID: \"630519d9-72be-441c-b013-eff2d5f360cc\") " Oct 29 23:32:46.417426 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:46.417256 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/630519d9-72be-441c-b013-eff2d5f360cc-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/630519d9-72be-441c-b013-eff2d5f360cc-model-cache\") pod \"630519d9-72be-441c-b013-eff2d5f360cc\" (UID: \"630519d9-72be-441c-b013-eff2d5f360cc\") " Oct 29 23:32:46.417426 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:46.417299 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/630519d9-72be-441c-b013-eff2d5f360cc-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/630519d9-72be-441c-b013-eff2d5f360cc-tmp-dir\") pod \"630519d9-72be-441c-b013-eff2d5f360cc\" (UID: \"630519d9-72be-441c-b013-eff2d5f360cc\") " Oct 29 23:32:46.417562 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:46.417435 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/630519d9-72be-441c-b013-eff2d5f360cc-tls-certs\" (UniqueName: \"kubernetes.io/secret/630519d9-72be-441c-b013-eff2d5f360cc-tls-certs\") pod \"630519d9-72be-441c-b013-eff2d5f360cc\" (UID: \"630519d9-72be-441c-b013-eff2d5f360cc\") " Oct 29 23:32:46.417622 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:46.417598 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/630519d9-72be-441c-b013-eff2d5f360cc-home" pod "630519d9-72be-441c-b013-eff2d5f360cc" (UID: "630519d9-72be-441c-b013-eff2d5f360cc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:32:46.417687 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:46.417639 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/630519d9-72be-441c-b013-eff2d5f360cc-model-cache" pod "630519d9-72be-441c-b013-eff2d5f360cc" (UID: "630519d9-72be-441c-b013-eff2d5f360cc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:32:46.418180 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:46.418117 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/630519d9-72be-441c-b013-eff2d5f360cc-tmp-dir" pod "630519d9-72be-441c-b013-eff2d5f360cc" (UID: "630519d9-72be-441c-b013-eff2d5f360cc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:32:46.418180 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:46.418161 5057 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/630519d9-72be-441c-b013-eff2d5f360cc-model-cache\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:32:46.418180 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:46.418183 5057 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/630519d9-72be-441c-b013-eff2d5f360cc-home\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:32:46.419821 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:46.419795 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/630519d9-72be-441c-b013-eff2d5f360cc-dshm" pod "630519d9-72be-441c-b013-eff2d5f360cc" (UID: "630519d9-72be-441c-b013-eff2d5f360cc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:32:46.421132 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:46.421103 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/630519d9-72be-441c-b013-eff2d5f360cc-tls-certs" pod "630519d9-72be-441c-b013-eff2d5f360cc" (UID: "630519d9-72be-441c-b013-eff2d5f360cc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:32:46.422984 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:46.422961 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/630519d9-72be-441c-b013-eff2d5f360cc-kube-api-access-5d8q5" pod "630519d9-72be-441c-b013-eff2d5f360cc" (UID: "630519d9-72be-441c-b013-eff2d5f360cc"). InnerVolumeSpecName "kube-api-access-5d8q5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:32:46.520668 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:46.520585 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5d8q5\" (UniqueName: \"kubernetes.io/projected/630519d9-72be-441c-b013-eff2d5f360cc-kube-api-access-5d8q5\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:32:46.520668 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:46.520618 5057 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/630519d9-72be-441c-b013-eff2d5f360cc-tmp-dir\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:32:46.520668 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:46.520632 5057 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/630519d9-72be-441c-b013-eff2d5f360cc-tls-certs\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:32:46.520668 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:46.520645 5057 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/630519d9-72be-441c-b013-eff2d5f360cc-dshm\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:32:46.779340 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:46.779228 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:32:46.890017 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:46.889984 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:32:47.042087 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:47.042007 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v2tfm"] Oct 29 23:32:47.105587 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:47.105550 5057 generic.go:356] "Generic (PLEG): container finished" podID="630519d9-72be-441c-b013-eff2d5f360cc" containerID="6adf4ca5f058014e45234e783c824b561ea789ec1c95e6b88930111388b332e6" exitCode=0 Oct 29 23:32:47.106016 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:47.105597 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-4504f1/sub-test-model-4504f1-kserve-657f694cf7-lqr5k" Oct 29 23:32:47.106016 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:47.105628 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-4504f1/sub-test-model-4504f1-kserve-657f694cf7-lqr5k" event={"ID":"630519d9-72be-441c-b013-eff2d5f360cc","Type":"ContainerDied","Data":"6adf4ca5f058014e45234e783c824b561ea789ec1c95e6b88930111388b332e6"} Oct 29 23:32:47.106016 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:47.105685 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-4504f1/sub-test-model-4504f1-kserve-657f694cf7-lqr5k" event={"ID":"630519d9-72be-441c-b013-eff2d5f360cc","Type":"ContainerDied","Data":"567052a2e252eadb6d89cf6a21bf838e78f9b8dad58ebb7b364cd1e0e8f517a8"} Oct 29 23:32:47.106016 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:47.105707 5057 scope.go:122] "RemoveContainer" containerID="6adf4ca5f058014e45234e783c824b561ea789ec1c95e6b88930111388b332e6" Oct 29 23:32:47.117148 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:47.117130 5057 scope.go:122] "RemoveContainer" containerID="6adf4ca5f058014e45234e783c824b561ea789ec1c95e6b88930111388b332e6" Oct 29 23:32:47.117433 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:32:47.117414 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6adf4ca5f058014e45234e783c824b561ea789ec1c95e6b88930111388b332e6\": container with ID starting with 6adf4ca5f058014e45234e783c824b561ea789ec1c95e6b88930111388b332e6 not found: ID does not exist" containerID="6adf4ca5f058014e45234e783c824b561ea789ec1c95e6b88930111388b332e6" Oct 29 23:32:47.117483 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:47.117441 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6adf4ca5f058014e45234e783c824b561ea789ec1c95e6b88930111388b332e6"} err="failed to get container status \"6adf4ca5f058014e45234e783c824b561ea789ec1c95e6b88930111388b332e6\": rpc error: code = NotFound desc = could not find container \"6adf4ca5f058014e45234e783c824b561ea789ec1c95e6b88930111388b332e6\": container with ID starting with 6adf4ca5f058014e45234e783c824b561ea789ec1c95e6b88930111388b332e6 not found: ID does not exist" Oct 29 23:32:47.155506 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:47.155476 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:32:47.163415 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:47.163391 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-4504f1/sub-test-model-4504f1-kserve-657f694cf7-lqr5k"] Oct 29 23:32:47.170206 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:47.170181 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w0-4504f1/sub-test-model-4504f1-kserve-657f694cf7-lqr5k"] Oct 29 23:32:47.329395 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:47.329300 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="630519d9-72be-441c-b013-eff2d5f360cc" path="/var/lib/kubelet/pods/630519d9-72be-441c-b013-eff2d5f360cc/volumes" Oct 29 23:32:47.329867 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:47.329845 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9d46ee5-61dd-4a34-93bc-bcf127bb983a" path="/var/lib/kubelet/pods/b9d46ee5-61dd-4a34-93bc-bcf127bb983a/volumes" Oct 29 23:32:48.830076 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:48.830037 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:32:49.028428 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:32:49.028382 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d2dnn" containerName="extractor" Oct 29 23:32:49.624263 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:49.624218 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:32:49.752782 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:49.752743 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:32:50.323160 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:50.323123 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:32:50.869768 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:50.869731 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-65e2d3-b8f79d5d4-5f2dp"] Oct 29 23:32:50.902283 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:50.902250 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-65e2d3-5f9cb8568c-qqh8t"] Oct 29 23:32:50.906252 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:50.906224 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w0-67363b/sub-test-model-67363b-kserve-54bb84cd5f-dpl7n" podUID="63e3a68d-90a1-4e09-8967-a1fb5d99b87a" containerName="main" containerID="cri-o://7a393b1e5357e8959f70b848e1b094f27bc7302c338e45443b05e5b5ad409157" gracePeriod=45 Oct 29 23:32:51.026807 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:51.026776 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:32:51.147962 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:51.147871 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:32:51.148166 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:51.148144 5057 generic.go:356] "Generic (PLEG): container finished" podID="63e3a68d-90a1-4e09-8967-a1fb5d99b87a" containerID="7a393b1e5357e8959f70b848e1b094f27bc7302c338e45443b05e5b5ad409157" exitCode=0 Oct 29 23:32:51.148250 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:51.148223 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-67363b/sub-test-model-67363b-kserve-54bb84cd5f-dpl7n" event={"ID":"63e3a68d-90a1-4e09-8967-a1fb5d99b87a","Type":"ContainerDied","Data":"7a393b1e5357e8959f70b848e1b094f27bc7302c338e45443b05e5b5ad409157"} Oct 29 23:32:51.199714 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:51.199688 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-67363b/sub-test-model-67363b-kserve-54bb84cd5f-dpl7n" Oct 29 23:32:51.277236 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:51.277194 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63e3a68d-90a1-4e09-8967-a1fb5d99b87a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/63e3a68d-90a1-4e09-8967-a1fb5d99b87a-dshm\") pod \"63e3a68d-90a1-4e09-8967-a1fb5d99b87a\" (UID: \"63e3a68d-90a1-4e09-8967-a1fb5d99b87a\") " Oct 29 23:32:51.277480 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:51.277260 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63e3a68d-90a1-4e09-8967-a1fb5d99b87a-home\" (UniqueName: \"kubernetes.io/empty-dir/63e3a68d-90a1-4e09-8967-a1fb5d99b87a-home\") pod \"63e3a68d-90a1-4e09-8967-a1fb5d99b87a\" (UID: \"63e3a68d-90a1-4e09-8967-a1fb5d99b87a\") " Oct 29 23:32:51.277480 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:51.277307 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63e3a68d-90a1-4e09-8967-a1fb5d99b87a-kube-api-access-glrcr\" (UniqueName: \"kubernetes.io/projected/63e3a68d-90a1-4e09-8967-a1fb5d99b87a-kube-api-access-glrcr\") pod \"63e3a68d-90a1-4e09-8967-a1fb5d99b87a\" (UID: \"63e3a68d-90a1-4e09-8967-a1fb5d99b87a\") " Oct 29 23:32:51.277480 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:51.277464 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63e3a68d-90a1-4e09-8967-a1fb5d99b87a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/63e3a68d-90a1-4e09-8967-a1fb5d99b87a-tmp-dir\") pod \"63e3a68d-90a1-4e09-8967-a1fb5d99b87a\" (UID: \"63e3a68d-90a1-4e09-8967-a1fb5d99b87a\") " Oct 29 23:32:51.277687 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:51.277513 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63e3a68d-90a1-4e09-8967-a1fb5d99b87a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/63e3a68d-90a1-4e09-8967-a1fb5d99b87a-model-cache\") pod \"63e3a68d-90a1-4e09-8967-a1fb5d99b87a\" (UID: \"63e3a68d-90a1-4e09-8967-a1fb5d99b87a\") " Oct 29 23:32:51.277687 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:51.277581 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/63e3a68d-90a1-4e09-8967-a1fb5d99b87a-tls-certs\" (UniqueName: \"kubernetes.io/secret/63e3a68d-90a1-4e09-8967-a1fb5d99b87a-tls-certs\") pod \"63e3a68d-90a1-4e09-8967-a1fb5d99b87a\" (UID: \"63e3a68d-90a1-4e09-8967-a1fb5d99b87a\") " Oct 29 23:32:51.277687 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:51.277603 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63e3a68d-90a1-4e09-8967-a1fb5d99b87a-home" pod "63e3a68d-90a1-4e09-8967-a1fb5d99b87a" (UID: "63e3a68d-90a1-4e09-8967-a1fb5d99b87a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:32:51.277872 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:51.277847 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63e3a68d-90a1-4e09-8967-a1fb5d99b87a-model-cache" pod "63e3a68d-90a1-4e09-8967-a1fb5d99b87a" (UID: "63e3a68d-90a1-4e09-8967-a1fb5d99b87a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:32:51.277986 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:51.277899 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63e3a68d-90a1-4e09-8967-a1fb5d99b87a-tmp-dir" pod "63e3a68d-90a1-4e09-8967-a1fb5d99b87a" (UID: "63e3a68d-90a1-4e09-8967-a1fb5d99b87a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:32:51.278126 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:51.278105 5057 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/63e3a68d-90a1-4e09-8967-a1fb5d99b87a-tmp-dir\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:32:51.278199 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:51.278134 5057 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/63e3a68d-90a1-4e09-8967-a1fb5d99b87a-model-cache\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:32:51.278199 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:51.278147 5057 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/63e3a68d-90a1-4e09-8967-a1fb5d99b87a-home\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:32:51.280218 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:51.280177 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63e3a68d-90a1-4e09-8967-a1fb5d99b87a-kube-api-access-glrcr" pod "63e3a68d-90a1-4e09-8967-a1fb5d99b87a" (UID: "63e3a68d-90a1-4e09-8967-a1fb5d99b87a"). InnerVolumeSpecName "kube-api-access-glrcr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:32:51.280359 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:51.280263 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63e3a68d-90a1-4e09-8967-a1fb5d99b87a-dshm" pod "63e3a68d-90a1-4e09-8967-a1fb5d99b87a" (UID: "63e3a68d-90a1-4e09-8967-a1fb5d99b87a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:32:51.280425 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:51.280377 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63e3a68d-90a1-4e09-8967-a1fb5d99b87a-tls-certs" pod "63e3a68d-90a1-4e09-8967-a1fb5d99b87a" (UID: "63e3a68d-90a1-4e09-8967-a1fb5d99b87a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:32:51.379539 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:51.379509 5057 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/63e3a68d-90a1-4e09-8967-a1fb5d99b87a-dshm\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:32:51.379539 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:51.379540 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-glrcr\" (UniqueName: \"kubernetes.io/projected/63e3a68d-90a1-4e09-8967-a1fb5d99b87a-kube-api-access-glrcr\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:32:51.380033 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:51.379556 5057 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/63e3a68d-90a1-4e09-8967-a1fb5d99b87a-tls-certs\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:32:52.109101 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:52.109062 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:32:52.167379 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:52.167318 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-67363b/sub-test-model-67363b-kserve-54bb84cd5f-dpl7n" Oct 29 23:32:52.167379 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:52.167362 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-67363b/sub-test-model-67363b-kserve-54bb84cd5f-dpl7n" event={"ID":"63e3a68d-90a1-4e09-8967-a1fb5d99b87a","Type":"ContainerDied","Data":"28c10fb6a882fd395ce6c43e14fdbb405e93ca0f288257b1702e1e296ce552b3"} Oct 29 23:32:52.167697 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:52.167431 5057 scope.go:122] "RemoveContainer" containerID="7a393b1e5357e8959f70b848e1b094f27bc7302c338e45443b05e5b5ad409157" Oct 29 23:32:52.221274 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:52.221237 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-67363b/sub-test-model-67363b-kserve-54bb84cd5f-dpl7n"] Oct 29 23:32:52.231569 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:52.231542 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w0-67363b/sub-test-model-67363b-kserve-54bb84cd5f-dpl7n"] Oct 29 23:32:53.328448 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:53.328414 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63e3a68d-90a1-4e09-8967-a1fb5d99b87a" path="/var/lib/kubelet/pods/63e3a68d-90a1-4e09-8967-a1fb5d99b87a/volumes" Oct 29 23:32:53.474998 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:53.474962 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:32:53.995388 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:53.995349 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:32:54.111255 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:54.111200 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:32:54.225098 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:54.225043 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:32:54.646550 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:54.646514 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:32:55.819620 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:55.819577 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:32:55.870115 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:55.870078 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-65e2d3-b8f79d5d4-5f2dp" podUID="05cd9ea1-8a8c-4434-a32d-540ccf6bfd4c" containerName="payload-processing" containerID="cri-o://f8bc163c52a340bd35c7edafa3089678729d3c00c4ef0bbeaaa8595222e730be" gracePeriod=25 Oct 29 23:32:55.903201 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:55.903166 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-65e2d3-5f9cb8568c-qqh8t" podUID="862fe3ac-f842-48d3-a722-ab1689671efd" containerName="payload-pre-processing" containerID="cri-o://9f448b248d837ec5c3071d5f5417d480c6a30bf44e29c77b3a8df4d5c28ae83d" gracePeriod=25 Oct 29 23:32:56.445846 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:56.445803 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:32:56.567757 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:56.567726 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:32:57.034317 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:57.034280 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:32:58.098611 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:58.098572 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:32:59.697866 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:32:59.697827 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:33:01.158794 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:01.158755 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:33:01.272512 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:01.272480 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:33:01.403891 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:01.403853 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:33:01.553203 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:01.553161 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:33:01.835897 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:01.835804 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-65e2d3-5f9cb8568c-qqh8t" podUID="862fe3ac-f842-48d3-a722-ab1689671efd" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:33:01.835897 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:01.835824 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-65e2d3-b8f79d5d4-5f2dp" podUID="05cd9ea1-8a8c-4434-a32d-540ccf6bfd4c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:33:02.206165 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:02.206129 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v2tfm"] Oct 29 23:33:03.220265 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:03.220219 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:33:04.549574 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:04.549536 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:33:04.657599 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:04.657559 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:33:04.894284 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:04.894191 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:33:05.011031 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:05.010384 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:33:05.376965 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:05.376915 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:33:06.457505 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:06.457468 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-e131af-58475b998f-bx6vp"] Oct 29 23:33:06.458053 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:06.457710 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e131af-58475b998f-bx6vp" podUID="3c334eeb-957b-47ba-9944-c363e0f6b35e" containerName="maas-api" containerID="cri-o://6c6ca1333e07a3e01388e13583348997f943c971b4f9de8b303868fe38113c7b" gracePeriod=30 Oct 29 23:33:06.534874 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:06.534839 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:33:06.768478 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:06.768450 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e131af-58475b998f-bx6vp" Oct 29 23:33:06.895067 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:06.895033 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c334eeb-957b-47ba-9944-c363e0f6b35e-kube-api-access-7d2j6\" (UniqueName: \"kubernetes.io/projected/3c334eeb-957b-47ba-9944-c363e0f6b35e-kube-api-access-7d2j6\") pod \"3c334eeb-957b-47ba-9944-c363e0f6b35e\" (UID: \"3c334eeb-957b-47ba-9944-c363e0f6b35e\") " Oct 29 23:33:06.895260 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:06.895107 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c334eeb-957b-47ba-9944-c363e0f6b35e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c334eeb-957b-47ba-9944-c363e0f6b35e-openshift-service-ca\") pod \"3c334eeb-957b-47ba-9944-c363e0f6b35e\" (UID: \"3c334eeb-957b-47ba-9944-c363e0f6b35e\") " Oct 29 23:33:06.895260 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:06.895152 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3c334eeb-957b-47ba-9944-c363e0f6b35e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3c334eeb-957b-47ba-9944-c363e0f6b35e-maas-api-tls\") pod \"3c334eeb-957b-47ba-9944-c363e0f6b35e\" (UID: \"3c334eeb-957b-47ba-9944-c363e0f6b35e\") " Oct 29 23:33:06.895458 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:06.895284 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3c334eeb-957b-47ba-9944-c363e0f6b35e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c334eeb-957b-47ba-9944-c363e0f6b35e-metrics-certs\") pod \"3c334eeb-957b-47ba-9944-c363e0f6b35e\" (UID: \"3c334eeb-957b-47ba-9944-c363e0f6b35e\") " Oct 29 23:33:06.895527 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:06.895499 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c334eeb-957b-47ba-9944-c363e0f6b35e-openshift-service-ca" pod "3c334eeb-957b-47ba-9944-c363e0f6b35e" (UID: "3c334eeb-957b-47ba-9944-c363e0f6b35e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:33:06.895803 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:06.895779 5057 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c334eeb-957b-47ba-9944-c363e0f6b35e-openshift-service-ca\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:33:06.897309 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:06.897284 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c334eeb-957b-47ba-9944-c363e0f6b35e-maas-api-tls" pod "3c334eeb-957b-47ba-9944-c363e0f6b35e" (UID: "3c334eeb-957b-47ba-9944-c363e0f6b35e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:33:06.897430 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:06.897347 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c334eeb-957b-47ba-9944-c363e0f6b35e-metrics-certs" pod "3c334eeb-957b-47ba-9944-c363e0f6b35e" (UID: "3c334eeb-957b-47ba-9944-c363e0f6b35e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:33:06.897430 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:06.897396 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c334eeb-957b-47ba-9944-c363e0f6b35e-kube-api-access-7d2j6" pod "3c334eeb-957b-47ba-9944-c363e0f6b35e" (UID: "3c334eeb-957b-47ba-9944-c363e0f6b35e"). InnerVolumeSpecName "kube-api-access-7d2j6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:33:06.996782 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:06.996703 5057 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3c334eeb-957b-47ba-9944-c363e0f6b35e-maas-api-tls\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:33:06.996782 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:06.996733 5057 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c334eeb-957b-47ba-9944-c363e0f6b35e-metrics-certs\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:33:06.996782 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:06.996743 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7d2j6\" (UniqueName: \"kubernetes.io/projected/3c334eeb-957b-47ba-9944-c363e0f6b35e-kube-api-access-7d2j6\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:33:07.116303 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:07.116263 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:33:07.260264 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:07.260177 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:33:07.304535 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:07.304498 5057 generic.go:356] "Generic (PLEG): container finished" podID="3c334eeb-957b-47ba-9944-c363e0f6b35e" containerID="6c6ca1333e07a3e01388e13583348997f943c971b4f9de8b303868fe38113c7b" exitCode=0 Oct 29 23:33:07.304704 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:07.304558 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e131af-58475b998f-bx6vp" Oct 29 23:33:07.304704 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:07.304581 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e131af-58475b998f-bx6vp" event={"ID":"3c334eeb-957b-47ba-9944-c363e0f6b35e","Type":"ContainerDied","Data":"6c6ca1333e07a3e01388e13583348997f943c971b4f9de8b303868fe38113c7b"} Oct 29 23:33:07.304704 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:07.304664 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e131af-58475b998f-bx6vp" event={"ID":"3c334eeb-957b-47ba-9944-c363e0f6b35e","Type":"ContainerDied","Data":"645e44971e30420b4c43d5295b253c00739bf855ca24d1b754bf1e2f3d9ef625"} Oct 29 23:33:07.304704 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:07.304692 5057 scope.go:122] "RemoveContainer" containerID="6c6ca1333e07a3e01388e13583348997f943c971b4f9de8b303868fe38113c7b" Oct 29 23:33:07.315891 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:07.315874 5057 scope.go:122] "RemoveContainer" containerID="6c6ca1333e07a3e01388e13583348997f943c971b4f9de8b303868fe38113c7b" Oct 29 23:33:07.316163 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:33:07.316145 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c6ca1333e07a3e01388e13583348997f943c971b4f9de8b303868fe38113c7b\": container with ID starting with 6c6ca1333e07a3e01388e13583348997f943c971b4f9de8b303868fe38113c7b not found: ID does not exist" containerID="6c6ca1333e07a3e01388e13583348997f943c971b4f9de8b303868fe38113c7b" Oct 29 23:33:07.316211 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:07.316175 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c6ca1333e07a3e01388e13583348997f943c971b4f9de8b303868fe38113c7b"} err="failed to get container status \"6c6ca1333e07a3e01388e13583348997f943c971b4f9de8b303868fe38113c7b\": rpc error: code = NotFound desc = could not find container \"6c6ca1333e07a3e01388e13583348997f943c971b4f9de8b303868fe38113c7b\": container with ID starting with 6c6ca1333e07a3e01388e13583348997f943c971b4f9de8b303868fe38113c7b not found: ID does not exist" Oct 29 23:33:07.385708 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:07.374640 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-e131af-58475b998f-bx6vp"] Oct 29 23:33:07.385708 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:07.384560 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:33:07.386029 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:07.385998 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-e131af-58475b998f-bx6vp"] Oct 29 23:33:07.733869 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:07.733830 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:33:09.060969 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:09.060936 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:33:09.330305 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:09.330228 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c334eeb-957b-47ba-9944-c363e0f6b35e" path="/var/lib/kubelet/pods/3c334eeb-957b-47ba-9944-c363e0f6b35e/volumes" Oct 29 23:33:10.457186 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:10.457147 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:33:10.795135 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:10.795006 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:33:11.835495 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:11.835451 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-65e2d3-5f9cb8568c-qqh8t" podUID="862fe3ac-f842-48d3-a722-ab1689671efd" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:33:11.835893 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:11.835455 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-65e2d3-b8f79d5d4-5f2dp" podUID="05cd9ea1-8a8c-4434-a32d-540ccf6bfd4c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:33:12.415810 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:12.415766 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:33:12.537919 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:12.537877 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-67363b-5d8768c84d-sgxgm"] Oct 29 23:33:12.573279 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:12.573245 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-67363b-745447b99f-jsvb4"] Oct 29 23:33:12.692450 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:12.692309 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:33:12.822511 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:12.822472 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:33:13.958477 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:13.958441 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v2tfm"] Oct 29 23:33:14.115911 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:14.115854 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:33:14.391121 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:14.391024 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:33:14.511979 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:14.511941 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:33:14.757857 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:14.757818 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:33:15.801748 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:15.801709 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:33:16.175170 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:16.175145 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-65e2d3-b8f79d5d4-5f2dp" Oct 29 23:33:16.216325 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:16.215571 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-65e2d3-5f9cb8568c-qqh8t" Oct 29 23:33:16.313027 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:16.312994 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/05cd9ea1-8a8c-4434-a32d-540ccf6bfd4c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05cd9ea1-8a8c-4434-a32d-540ccf6bfd4c-plugins-config-volume\") pod \"05cd9ea1-8a8c-4434-a32d-540ccf6bfd4c\" (UID: \"05cd9ea1-8a8c-4434-a32d-540ccf6bfd4c\") " Oct 29 23:33:16.313194 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:16.313157 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/862fe3ac-f842-48d3-a722-ab1689671efd-kube-api-access-6v6pp\" (UniqueName: \"kubernetes.io/projected/862fe3ac-f842-48d3-a722-ab1689671efd-kube-api-access-6v6pp\") pod \"862fe3ac-f842-48d3-a722-ab1689671efd\" (UID: \"862fe3ac-f842-48d3-a722-ab1689671efd\") " Oct 29 23:33:16.313194 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:16.313182 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05cd9ea1-8a8c-4434-a32d-540ccf6bfd4c-kube-api-access-52bc9\" (UniqueName: \"kubernetes.io/projected/05cd9ea1-8a8c-4434-a32d-540ccf6bfd4c-kube-api-access-52bc9\") pod \"05cd9ea1-8a8c-4434-a32d-540ccf6bfd4c\" (UID: \"05cd9ea1-8a8c-4434-a32d-540ccf6bfd4c\") " Oct 29 23:33:16.313470 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:16.313214 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/862fe3ac-f842-48d3-a722-ab1689671efd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/862fe3ac-f842-48d3-a722-ab1689671efd-plugins-config-volume\") pod \"862fe3ac-f842-48d3-a722-ab1689671efd\" (UID: \"862fe3ac-f842-48d3-a722-ab1689671efd\") " Oct 29 23:33:16.313470 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:16.313456 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05cd9ea1-8a8c-4434-a32d-540ccf6bfd4c-plugins-config-volume" pod "05cd9ea1-8a8c-4434-a32d-540ccf6bfd4c" (UID: "05cd9ea1-8a8c-4434-a32d-540ccf6bfd4c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:33:16.313869 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:16.313825 5057 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05cd9ea1-8a8c-4434-a32d-540ccf6bfd4c-plugins-config-volume\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:33:16.313869 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:16.313830 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/862fe3ac-f842-48d3-a722-ab1689671efd-plugins-config-volume" pod "862fe3ac-f842-48d3-a722-ab1689671efd" (UID: "862fe3ac-f842-48d3-a722-ab1689671efd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:33:16.315686 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:16.315663 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/862fe3ac-f842-48d3-a722-ab1689671efd-kube-api-access-6v6pp" pod "862fe3ac-f842-48d3-a722-ab1689671efd" (UID: "862fe3ac-f842-48d3-a722-ab1689671efd"). InnerVolumeSpecName "kube-api-access-6v6pp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:33:16.315774 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:16.315692 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05cd9ea1-8a8c-4434-a32d-540ccf6bfd4c-kube-api-access-52bc9" pod "05cd9ea1-8a8c-4434-a32d-540ccf6bfd4c" (UID: "05cd9ea1-8a8c-4434-a32d-540ccf6bfd4c"). InnerVolumeSpecName "kube-api-access-52bc9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:33:16.317497 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:16.317471 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:33:16.371869 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:16.371804 5057 generic.go:356] "Generic (PLEG): container finished" podID="862fe3ac-f842-48d3-a722-ab1689671efd" containerID="9f448b248d837ec5c3071d5f5417d480c6a30bf44e29c77b3a8df4d5c28ae83d" exitCode=0 Oct 29 23:33:16.371869 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:16.371847 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-65e2d3-5f9cb8568c-qqh8t" Oct 29 23:33:16.372047 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:16.371889 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-65e2d3-5f9cb8568c-qqh8t" event={"ID":"862fe3ac-f842-48d3-a722-ab1689671efd","Type":"ContainerDied","Data":"9f448b248d837ec5c3071d5f5417d480c6a30bf44e29c77b3a8df4d5c28ae83d"} Oct 29 23:33:16.372047 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:16.371943 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-65e2d3-5f9cb8568c-qqh8t" event={"ID":"862fe3ac-f842-48d3-a722-ab1689671efd","Type":"ContainerDied","Data":"0fd6930ee21e4670446c526403a4a59f4d24c7d06f434cd7ac7dc57ae4f2d897"} Oct 29 23:33:16.372047 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:16.371966 5057 scope.go:122] "RemoveContainer" containerID="9f448b248d837ec5c3071d5f5417d480c6a30bf44e29c77b3a8df4d5c28ae83d" Oct 29 23:33:16.373444 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:16.373419 5057 generic.go:356] "Generic (PLEG): container finished" podID="05cd9ea1-8a8c-4434-a32d-540ccf6bfd4c" containerID="f8bc163c52a340bd35c7edafa3089678729d3c00c4ef0bbeaaa8595222e730be" exitCode=0 Oct 29 23:33:16.373563 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:16.373461 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-65e2d3-b8f79d5d4-5f2dp" event={"ID":"05cd9ea1-8a8c-4434-a32d-540ccf6bfd4c","Type":"ContainerDied","Data":"f8bc163c52a340bd35c7edafa3089678729d3c00c4ef0bbeaaa8595222e730be"} Oct 29 23:33:16.373563 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:16.373467 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-65e2d3-b8f79d5d4-5f2dp" Oct 29 23:33:16.373563 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:16.373490 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-65e2d3-b8f79d5d4-5f2dp" event={"ID":"05cd9ea1-8a8c-4434-a32d-540ccf6bfd4c","Type":"ContainerDied","Data":"7ad93ccee587571bbc568b875a5c35a1b7bfc62248b5cc832859f84dfc94170b"} Oct 29 23:33:16.385206 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:16.385190 5057 scope.go:122] "RemoveContainer" containerID="9f448b248d837ec5c3071d5f5417d480c6a30bf44e29c77b3a8df4d5c28ae83d" Oct 29 23:33:16.385488 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:33:16.385466 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f448b248d837ec5c3071d5f5417d480c6a30bf44e29c77b3a8df4d5c28ae83d\": container with ID starting with 9f448b248d837ec5c3071d5f5417d480c6a30bf44e29c77b3a8df4d5c28ae83d not found: ID does not exist" containerID="9f448b248d837ec5c3071d5f5417d480c6a30bf44e29c77b3a8df4d5c28ae83d" Oct 29 23:33:16.385581 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:16.385497 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f448b248d837ec5c3071d5f5417d480c6a30bf44e29c77b3a8df4d5c28ae83d"} err="failed to get container status \"9f448b248d837ec5c3071d5f5417d480c6a30bf44e29c77b3a8df4d5c28ae83d\": rpc error: code = NotFound desc = could not find container \"9f448b248d837ec5c3071d5f5417d480c6a30bf44e29c77b3a8df4d5c28ae83d\": container with ID starting with 9f448b248d837ec5c3071d5f5417d480c6a30bf44e29c77b3a8df4d5c28ae83d not found: ID does not exist" Oct 29 23:33:16.385581 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:16.385521 5057 scope.go:122] "RemoveContainer" containerID="f8bc163c52a340bd35c7edafa3089678729d3c00c4ef0bbeaaa8595222e730be" Oct 29 23:33:16.395870 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:16.395853 5057 scope.go:122] "RemoveContainer" containerID="f8bc163c52a340bd35c7edafa3089678729d3c00c4ef0bbeaaa8595222e730be" Oct 29 23:33:16.396139 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:33:16.396116 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8bc163c52a340bd35c7edafa3089678729d3c00c4ef0bbeaaa8595222e730be\": container with ID starting with f8bc163c52a340bd35c7edafa3089678729d3c00c4ef0bbeaaa8595222e730be not found: ID does not exist" containerID="f8bc163c52a340bd35c7edafa3089678729d3c00c4ef0bbeaaa8595222e730be" Oct 29 23:33:16.396228 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:16.396144 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8bc163c52a340bd35c7edafa3089678729d3c00c4ef0bbeaaa8595222e730be"} err="failed to get container status \"f8bc163c52a340bd35c7edafa3089678729d3c00c4ef0bbeaaa8595222e730be\": rpc error: code = NotFound desc = could not find container \"f8bc163c52a340bd35c7edafa3089678729d3c00c4ef0bbeaaa8595222e730be\": container with ID starting with f8bc163c52a340bd35c7edafa3089678729d3c00c4ef0bbeaaa8595222e730be not found: ID does not exist" Oct 29 23:33:16.430101 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:16.430065 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6v6pp\" (UniqueName: \"kubernetes.io/projected/862fe3ac-f842-48d3-a722-ab1689671efd-kube-api-access-6v6pp\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:33:16.430101 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:16.430102 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-52bc9\" (UniqueName: \"kubernetes.io/projected/05cd9ea1-8a8c-4434-a32d-540ccf6bfd4c-kube-api-access-52bc9\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:33:16.430440 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:16.430119 5057 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/862fe3ac-f842-48d3-a722-ab1689671efd-plugins-config-volume\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:33:16.432428 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:16.432400 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-65e2d3-5f9cb8568c-qqh8t"] Oct 29 23:33:16.437456 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:16.437426 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-65e2d3-5f9cb8568c-qqh8t"] Oct 29 23:33:16.442106 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:16.442075 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:33:16.463815 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:16.463787 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-65e2d3-b8f79d5d4-5f2dp"] Oct 29 23:33:16.466863 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:16.466839 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-65e2d3-b8f79d5d4-5f2dp"] Oct 29 23:33:17.194806 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:17.194709 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:33:17.306376 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:17.306323 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:33:17.328640 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:17.328605 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05cd9ea1-8a8c-4434-a32d-540ccf6bfd4c" path="/var/lib/kubelet/pods/05cd9ea1-8a8c-4434-a32d-540ccf6bfd4c/volumes" Oct 29 23:33:17.329115 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:17.329100 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="862fe3ac-f842-48d3-a722-ab1689671efd" path="/var/lib/kubelet/pods/862fe3ac-f842-48d3-a722-ab1689671efd/volumes" Oct 29 23:33:17.540379 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:17.538706 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-67363b-5d8768c84d-sgxgm" podUID="f15821b0-cb92-4759-aba2-a4fe6f8e62f9" containerName="payload-processing" containerID="cri-o://a8626ca70853027be2afbd044099cce1be94a720312d395c38ab234d25fa908d" gracePeriod=25 Oct 29 23:33:17.574048 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:17.573993 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-67363b-745447b99f-jsvb4" podUID="59602ea3-b37d-479e-96f3-9f72411920b4" containerName="payload-pre-processing" containerID="cri-o://b4700e6140c401d453d38a5d21981223b3514db88f89cc8d87eeaf127e8f19b0" gracePeriod=25 Oct 29 23:33:17.678526 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:17.678488 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:33:17.800647 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:17.800562 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:33:17.925944 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:17.925905 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:33:18.456672 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:18.456636 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:33:19.027546 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:33:19.027508 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d2dnn" containerName="extractor" Oct 29 23:33:19.810739 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:19.810694 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:33:19.921298 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:19.921257 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:33:21.524740 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:21.524706 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:33:23.172236 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:23.172130 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:33:24.825462 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:24.825418 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:33:25.035141 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:25.035093 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:33:25.148213 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:25.148121 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:33:25.335595 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:25.335544 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:33:25.824238 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:25.824194 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:33:26.589461 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:26.589422 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:33:26.610134 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:26.610089 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-67363b-745447b99f-jsvb4" podUID="59602ea3-b37d-479e-96f3-9f72411920b4" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:33:26.610134 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:26.610120 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-67363b-5d8768c84d-sgxgm" podUID="f15821b0-cb92-4759-aba2-a4fe6f8e62f9" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:33:26.712110 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:26.712073 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:33:28.391807 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:28.391770 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:33:29.552172 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:29.552135 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v2tfm"] Oct 29 23:33:30.008202 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:30.008160 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:33:30.787109 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:30.787071 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-08e15d/distinct-w0-08e15d-kserve-85695c78bb-p67gk"] Oct 29 23:33:30.982681 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:30.982645 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:33:31.698122 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:31.698089 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:33:33.219290 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:33.219255 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:33:34.805393 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:34.805351 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:33:36.420293 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:36.420257 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:33:36.540695 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:36.540658 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-08e15d-6bd798d867-mnq4t"] Oct 29 23:33:36.610210 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:36.610165 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-67363b-5d8768c84d-sgxgm" podUID="f15821b0-cb92-4759-aba2-a4fe6f8e62f9" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:33:36.611146 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:36.611112 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-67363b-745447b99f-jsvb4" podUID="59602ea3-b37d-479e-96f3-9f72411920b4" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:33:36.666690 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:36.666650 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:33:36.877135 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:36.877096 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:33:37.033660 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:37.033608 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:33:37.764796 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:37.764752 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-063b46-5895cc45b-jqqdh"] Oct 29 23:33:37.765595 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:37.765550 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-063b46-5895cc45b-jqqdh" podUID="8508fdad-a139-4803-affb-b63a6d52bd13" containerName="maas-api" containerID="cri-o://22aa141fdbcfb4ff64996355576208767fc6947554c48afe4e4134b76164a9c2" gracePeriod=30 Oct 29 23:33:37.795863 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:37.795722 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-67363b-5d8768c84d-sgxgm" Oct 29 23:33:37.832774 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:37.832620 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:33:37.898098 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:37.897729 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-67363b-745447b99f-jsvb4" Oct 29 23:33:37.913149 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:37.910742 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f15821b0-cb92-4759-aba2-a4fe6f8e62f9-kube-api-access-z9gj4\" (UniqueName: \"kubernetes.io/projected/f15821b0-cb92-4759-aba2-a4fe6f8e62f9-kube-api-access-z9gj4\") pod \"f15821b0-cb92-4759-aba2-a4fe6f8e62f9\" (UID: \"f15821b0-cb92-4759-aba2-a4fe6f8e62f9\") " Oct 29 23:33:37.913149 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:37.910987 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f15821b0-cb92-4759-aba2-a4fe6f8e62f9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f15821b0-cb92-4759-aba2-a4fe6f8e62f9-plugins-config-volume\") pod \"f15821b0-cb92-4759-aba2-a4fe6f8e62f9\" (UID: \"f15821b0-cb92-4759-aba2-a4fe6f8e62f9\") " Oct 29 23:33:37.913149 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:37.911868 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f15821b0-cb92-4759-aba2-a4fe6f8e62f9-plugins-config-volume" pod "f15821b0-cb92-4759-aba2-a4fe6f8e62f9" (UID: "f15821b0-cb92-4759-aba2-a4fe6f8e62f9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:33:37.917607 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:37.917583 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f15821b0-cb92-4759-aba2-a4fe6f8e62f9-kube-api-access-z9gj4" pod "f15821b0-cb92-4759-aba2-a4fe6f8e62f9" (UID: "f15821b0-cb92-4759-aba2-a4fe6f8e62f9"). InnerVolumeSpecName "kube-api-access-z9gj4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:33:38.014399 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:38.014150 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/59602ea3-b37d-479e-96f3-9f72411920b4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59602ea3-b37d-479e-96f3-9f72411920b4-plugins-config-volume\") pod \"59602ea3-b37d-479e-96f3-9f72411920b4\" (UID: \"59602ea3-b37d-479e-96f3-9f72411920b4\") " Oct 29 23:33:38.014399 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:38.014318 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59602ea3-b37d-479e-96f3-9f72411920b4-kube-api-access-r69mw\" (UniqueName: \"kubernetes.io/projected/59602ea3-b37d-479e-96f3-9f72411920b4-kube-api-access-r69mw\") pod \"59602ea3-b37d-479e-96f3-9f72411920b4\" (UID: \"59602ea3-b37d-479e-96f3-9f72411920b4\") " Oct 29 23:33:38.015906 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:38.015269 5057 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f15821b0-cb92-4759-aba2-a4fe6f8e62f9-plugins-config-volume\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:33:38.015906 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:38.015305 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z9gj4\" (UniqueName: \"kubernetes.io/projected/f15821b0-cb92-4759-aba2-a4fe6f8e62f9-kube-api-access-z9gj4\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:33:38.016303 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:38.016258 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59602ea3-b37d-479e-96f3-9f72411920b4-plugins-config-volume" pod "59602ea3-b37d-479e-96f3-9f72411920b4" (UID: "59602ea3-b37d-479e-96f3-9f72411920b4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:33:38.019273 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:38.019232 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59602ea3-b37d-479e-96f3-9f72411920b4-kube-api-access-r69mw" pod "59602ea3-b37d-479e-96f3-9f72411920b4" (UID: "59602ea3-b37d-479e-96f3-9f72411920b4"). InnerVolumeSpecName "kube-api-access-r69mw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:33:38.116580 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:38.116544 5057 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59602ea3-b37d-479e-96f3-9f72411920b4-plugins-config-volume\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:33:38.116580 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:38.116573 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r69mw\" (UniqueName: \"kubernetes.io/projected/59602ea3-b37d-479e-96f3-9f72411920b4-kube-api-access-r69mw\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:33:38.148847 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:38.148818 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-063b46-5895cc45b-jqqdh" Oct 29 23:33:38.318576 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:38.318483 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8508fdad-a139-4803-affb-b63a6d52bd13-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8508fdad-a139-4803-affb-b63a6d52bd13-metrics-certs\") pod \"8508fdad-a139-4803-affb-b63a6d52bd13\" (UID: \"8508fdad-a139-4803-affb-b63a6d52bd13\") " Oct 29 23:33:38.318744 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:38.318655 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8508fdad-a139-4803-affb-b63a6d52bd13-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8508fdad-a139-4803-affb-b63a6d52bd13-openshift-service-ca\") pod \"8508fdad-a139-4803-affb-b63a6d52bd13\" (UID: \"8508fdad-a139-4803-affb-b63a6d52bd13\") " Oct 29 23:33:38.318744 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:38.318700 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8508fdad-a139-4803-affb-b63a6d52bd13-kube-api-access-t9zqv\" (UniqueName: \"kubernetes.io/projected/8508fdad-a139-4803-affb-b63a6d52bd13-kube-api-access-t9zqv\") pod \"8508fdad-a139-4803-affb-b63a6d52bd13\" (UID: \"8508fdad-a139-4803-affb-b63a6d52bd13\") " Oct 29 23:33:38.318744 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:38.318727 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8508fdad-a139-4803-affb-b63a6d52bd13-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8508fdad-a139-4803-affb-b63a6d52bd13-maas-api-tls\") pod \"8508fdad-a139-4803-affb-b63a6d52bd13\" (UID: \"8508fdad-a139-4803-affb-b63a6d52bd13\") " Oct 29 23:33:38.319498 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:38.319471 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8508fdad-a139-4803-affb-b63a6d52bd13-openshift-service-ca" pod "8508fdad-a139-4803-affb-b63a6d52bd13" (UID: "8508fdad-a139-4803-affb-b63a6d52bd13"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:33:38.320709 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:38.319994 5057 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8508fdad-a139-4803-affb-b63a6d52bd13-openshift-service-ca\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:33:38.322248 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:38.322222 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8508fdad-a139-4803-affb-b63a6d52bd13-kube-api-access-t9zqv" pod "8508fdad-a139-4803-affb-b63a6d52bd13" (UID: "8508fdad-a139-4803-affb-b63a6d52bd13"). InnerVolumeSpecName "kube-api-access-t9zqv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:33:38.322404 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:38.322313 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8508fdad-a139-4803-affb-b63a6d52bd13-maas-api-tls" pod "8508fdad-a139-4803-affb-b63a6d52bd13" (UID: "8508fdad-a139-4803-affb-b63a6d52bd13"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:33:38.324028 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:38.323999 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8508fdad-a139-4803-affb-b63a6d52bd13-metrics-certs" pod "8508fdad-a139-4803-affb-b63a6d52bd13" (UID: "8508fdad-a139-4803-affb-b63a6d52bd13"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:33:38.381039 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:38.381010 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:33:38.421356 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:38.420986 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t9zqv\" (UniqueName: \"kubernetes.io/projected/8508fdad-a139-4803-affb-b63a6d52bd13-kube-api-access-t9zqv\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:33:38.421356 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:38.421017 5057 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8508fdad-a139-4803-affb-b63a6d52bd13-maas-api-tls\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:33:38.421356 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:38.421033 5057 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8508fdad-a139-4803-affb-b63a6d52bd13-metrics-certs\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:33:38.493629 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:38.493590 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:33:38.579624 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:38.579550 5057 generic.go:356] "Generic (PLEG): container finished" podID="8508fdad-a139-4803-affb-b63a6d52bd13" containerID="22aa141fdbcfb4ff64996355576208767fc6947554c48afe4e4134b76164a9c2" exitCode=0 Oct 29 23:33:38.579833 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:38.579596 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-063b46-5895cc45b-jqqdh" event={"ID":"8508fdad-a139-4803-affb-b63a6d52bd13","Type":"ContainerDied","Data":"22aa141fdbcfb4ff64996355576208767fc6947554c48afe4e4134b76164a9c2"} Oct 29 23:33:38.580027 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:38.579610 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-063b46-5895cc45b-jqqdh" Oct 29 23:33:38.580027 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:38.579884 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-063b46-5895cc45b-jqqdh" event={"ID":"8508fdad-a139-4803-affb-b63a6d52bd13","Type":"ContainerDied","Data":"b7431cd94d1d89632ab3ba4449c4e76976d635f8f1d92042d30a360e403ec39f"} Oct 29 23:33:38.580027 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:38.579915 5057 scope.go:122] "RemoveContainer" containerID="22aa141fdbcfb4ff64996355576208767fc6947554c48afe4e4134b76164a9c2" Oct 29 23:33:38.582513 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:38.582485 5057 generic.go:356] "Generic (PLEG): container finished" podID="59602ea3-b37d-479e-96f3-9f72411920b4" containerID="b4700e6140c401d453d38a5d21981223b3514db88f89cc8d87eeaf127e8f19b0" exitCode=0 Oct 29 23:33:38.582660 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:38.582545 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-67363b-745447b99f-jsvb4" Oct 29 23:33:38.582727 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:38.582658 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-67363b-745447b99f-jsvb4" event={"ID":"59602ea3-b37d-479e-96f3-9f72411920b4","Type":"ContainerDied","Data":"b4700e6140c401d453d38a5d21981223b3514db88f89cc8d87eeaf127e8f19b0"} Oct 29 23:33:38.582727 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:38.582699 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-67363b-745447b99f-jsvb4" event={"ID":"59602ea3-b37d-479e-96f3-9f72411920b4","Type":"ContainerDied","Data":"73c3f36e7e8c5454693b045da0b65f798aced1ac46537126506718e4f00dac9c"} Oct 29 23:33:38.584656 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:38.584624 5057 generic.go:356] "Generic (PLEG): container finished" podID="f15821b0-cb92-4759-aba2-a4fe6f8e62f9" containerID="a8626ca70853027be2afbd044099cce1be94a720312d395c38ab234d25fa908d" exitCode=0 Oct 29 23:33:38.584770 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:38.584730 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-67363b-5d8768c84d-sgxgm" event={"ID":"f15821b0-cb92-4759-aba2-a4fe6f8e62f9","Type":"ContainerDied","Data":"a8626ca70853027be2afbd044099cce1be94a720312d395c38ab234d25fa908d"} Oct 29 23:33:38.584770 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:38.584764 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-67363b-5d8768c84d-sgxgm" event={"ID":"f15821b0-cb92-4759-aba2-a4fe6f8e62f9","Type":"ContainerDied","Data":"5f1308eb98a7b994261e7d61d774565cbda997c5e70dfe4f455ec298930d3d8e"} Oct 29 23:33:38.584887 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:38.584824 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-67363b-5d8768c84d-sgxgm" Oct 29 23:33:38.596925 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:38.596899 5057 scope.go:122] "RemoveContainer" containerID="22aa141fdbcfb4ff64996355576208767fc6947554c48afe4e4134b76164a9c2" Oct 29 23:33:38.597436 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:33:38.597407 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22aa141fdbcfb4ff64996355576208767fc6947554c48afe4e4134b76164a9c2\": container with ID starting with 22aa141fdbcfb4ff64996355576208767fc6947554c48afe4e4134b76164a9c2 not found: ID does not exist" containerID="22aa141fdbcfb4ff64996355576208767fc6947554c48afe4e4134b76164a9c2" Oct 29 23:33:38.598391 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:38.598354 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22aa141fdbcfb4ff64996355576208767fc6947554c48afe4e4134b76164a9c2"} err="failed to get container status \"22aa141fdbcfb4ff64996355576208767fc6947554c48afe4e4134b76164a9c2\": rpc error: code = NotFound desc = could not find container \"22aa141fdbcfb4ff64996355576208767fc6947554c48afe4e4134b76164a9c2\": container with ID starting with 22aa141fdbcfb4ff64996355576208767fc6947554c48afe4e4134b76164a9c2 not found: ID does not exist" Oct 29 23:33:38.598974 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:38.598394 5057 scope.go:122] "RemoveContainer" containerID="b4700e6140c401d453d38a5d21981223b3514db88f89cc8d87eeaf127e8f19b0" Oct 29 23:33:38.613715 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:38.613694 5057 scope.go:122] "RemoveContainer" containerID="b4700e6140c401d453d38a5d21981223b3514db88f89cc8d87eeaf127e8f19b0" Oct 29 23:33:38.614029 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:33:38.614008 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4700e6140c401d453d38a5d21981223b3514db88f89cc8d87eeaf127e8f19b0\": container with ID starting with b4700e6140c401d453d38a5d21981223b3514db88f89cc8d87eeaf127e8f19b0 not found: ID does not exist" containerID="b4700e6140c401d453d38a5d21981223b3514db88f89cc8d87eeaf127e8f19b0" Oct 29 23:33:38.614110 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:38.614039 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4700e6140c401d453d38a5d21981223b3514db88f89cc8d87eeaf127e8f19b0"} err="failed to get container status \"b4700e6140c401d453d38a5d21981223b3514db88f89cc8d87eeaf127e8f19b0\": rpc error: code = NotFound desc = could not find container \"b4700e6140c401d453d38a5d21981223b3514db88f89cc8d87eeaf127e8f19b0\": container with ID starting with b4700e6140c401d453d38a5d21981223b3514db88f89cc8d87eeaf127e8f19b0 not found: ID does not exist" Oct 29 23:33:38.614110 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:38.614064 5057 scope.go:122] "RemoveContainer" containerID="a8626ca70853027be2afbd044099cce1be94a720312d395c38ab234d25fa908d" Oct 29 23:33:38.623938 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:38.623917 5057 scope.go:122] "RemoveContainer" containerID="a8626ca70853027be2afbd044099cce1be94a720312d395c38ab234d25fa908d" Oct 29 23:33:38.624209 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:33:38.624187 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8626ca70853027be2afbd044099cce1be94a720312d395c38ab234d25fa908d\": container with ID starting with a8626ca70853027be2afbd044099cce1be94a720312d395c38ab234d25fa908d not found: ID does not exist" containerID="a8626ca70853027be2afbd044099cce1be94a720312d395c38ab234d25fa908d" Oct 29 23:33:38.624290 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:38.624218 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8626ca70853027be2afbd044099cce1be94a720312d395c38ab234d25fa908d"} err="failed to get container status \"a8626ca70853027be2afbd044099cce1be94a720312d395c38ab234d25fa908d\": rpc error: code = NotFound desc = could not find container \"a8626ca70853027be2afbd044099cce1be94a720312d395c38ab234d25fa908d\": container with ID starting with a8626ca70853027be2afbd044099cce1be94a720312d395c38ab234d25fa908d not found: ID does not exist" Oct 29 23:33:38.685142 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:38.685108 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:33:38.699826 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:38.699797 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-67363b-5d8768c84d-sgxgm"] Oct 29 23:33:38.722689 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:38.722661 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-67363b-5d8768c84d-sgxgm"] Oct 29 23:33:38.756681 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:38.756652 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-063b46-5895cc45b-jqqdh"] Oct 29 23:33:38.759262 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:38.759244 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-063b46-5895cc45b-jqqdh"] Oct 29 23:33:38.789816 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:38.789793 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-67363b-745447b99f-jsvb4"] Oct 29 23:33:38.795222 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:38.795198 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-67363b-745447b99f-jsvb4"] Oct 29 23:33:39.102056 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:39.102019 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:33:39.328944 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:39.328911 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59602ea3-b37d-479e-96f3-9f72411920b4" path="/var/lib/kubelet/pods/59602ea3-b37d-479e-96f3-9f72411920b4/volumes" Oct 29 23:33:39.329448 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:39.329426 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8508fdad-a139-4803-affb-b63a6d52bd13" path="/var/lib/kubelet/pods/8508fdad-a139-4803-affb-b63a6d52bd13/volumes" Oct 29 23:33:39.329867 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:39.329848 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f15821b0-cb92-4759-aba2-a4fe6f8e62f9" path="/var/lib/kubelet/pods/f15821b0-cb92-4759-aba2-a4fe6f8e62f9/volumes" Oct 29 23:33:40.139961 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:40.139930 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-vj6md_aa6a190e-3283-458a-9dfd-2de7cb193fc5/console-operator/2.log" Oct 29 23:33:40.163004 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:40.162976 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-vj6md_aa6a190e-3283-458a-9dfd-2de7cb193fc5/console-operator/2.log" Oct 29 23:33:41.085233 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:41.085196 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v2tfm"] Oct 29 23:33:41.541797 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:41.541758 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-08e15d-6bd798d867-mnq4t" podUID="684cec70-0bb2-4eff-917e-6a04c8734d74" containerName="payload-pre-processing" containerID="cri-o://ba185bb1c59c6d13d01637228fd6f3dc866f5ce40dff3d4253fe06e110a9e2c9" gracePeriod=25 Oct 29 23:33:41.803945 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:41.803877 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-08e15d-6bd798d867-mnq4t" Oct 29 23:33:41.853299 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:41.853270 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/684cec70-0bb2-4eff-917e-6a04c8734d74-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/684cec70-0bb2-4eff-917e-6a04c8734d74-plugins-config-volume\") pod \"684cec70-0bb2-4eff-917e-6a04c8734d74\" (UID: \"684cec70-0bb2-4eff-917e-6a04c8734d74\") " Oct 29 23:33:41.853496 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:41.853389 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/684cec70-0bb2-4eff-917e-6a04c8734d74-kube-api-access-wpvjw\" (UniqueName: \"kubernetes.io/projected/684cec70-0bb2-4eff-917e-6a04c8734d74-kube-api-access-wpvjw\") pod \"684cec70-0bb2-4eff-917e-6a04c8734d74\" (UID: \"684cec70-0bb2-4eff-917e-6a04c8734d74\") " Oct 29 23:33:41.853746 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:41.853714 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/684cec70-0bb2-4eff-917e-6a04c8734d74-plugins-config-volume" pod "684cec70-0bb2-4eff-917e-6a04c8734d74" (UID: "684cec70-0bb2-4eff-917e-6a04c8734d74"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:33:41.855552 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:41.855519 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/684cec70-0bb2-4eff-917e-6a04c8734d74-kube-api-access-wpvjw" pod "684cec70-0bb2-4eff-917e-6a04c8734d74" (UID: "684cec70-0bb2-4eff-917e-6a04c8734d74"). InnerVolumeSpecName "kube-api-access-wpvjw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:33:41.954663 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:41.954633 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wpvjw\" (UniqueName: \"kubernetes.io/projected/684cec70-0bb2-4eff-917e-6a04c8734d74-kube-api-access-wpvjw\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:33:41.954663 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:41.954660 5057 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/684cec70-0bb2-4eff-917e-6a04c8734d74-plugins-config-volume\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:33:42.624723 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:42.624685 5057 generic.go:356] "Generic (PLEG): container finished" podID="684cec70-0bb2-4eff-917e-6a04c8734d74" containerID="ba185bb1c59c6d13d01637228fd6f3dc866f5ce40dff3d4253fe06e110a9e2c9" exitCode=0 Oct 29 23:33:42.625196 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:42.624741 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-08e15d-6bd798d867-mnq4t" Oct 29 23:33:42.625196 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:42.624769 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-08e15d-6bd798d867-mnq4t" event={"ID":"684cec70-0bb2-4eff-917e-6a04c8734d74","Type":"ContainerDied","Data":"ba185bb1c59c6d13d01637228fd6f3dc866f5ce40dff3d4253fe06e110a9e2c9"} Oct 29 23:33:42.625196 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:42.624852 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-08e15d-6bd798d867-mnq4t" event={"ID":"684cec70-0bb2-4eff-917e-6a04c8734d74","Type":"ContainerDied","Data":"4b6783b1565491a31c0c50932649a4bf0a8bdaf5d20d818ee295c0d2842b06c5"} Oct 29 23:33:42.625196 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:42.624885 5057 scope.go:122] "RemoveContainer" containerID="ba185bb1c59c6d13d01637228fd6f3dc866f5ce40dff3d4253fe06e110a9e2c9" Oct 29 23:33:42.638043 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:42.638024 5057 scope.go:122] "RemoveContainer" containerID="ba185bb1c59c6d13d01637228fd6f3dc866f5ce40dff3d4253fe06e110a9e2c9" Oct 29 23:33:42.638557 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:33:42.638533 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba185bb1c59c6d13d01637228fd6f3dc866f5ce40dff3d4253fe06e110a9e2c9\": container with ID starting with ba185bb1c59c6d13d01637228fd6f3dc866f5ce40dff3d4253fe06e110a9e2c9 not found: ID does not exist" containerID="ba185bb1c59c6d13d01637228fd6f3dc866f5ce40dff3d4253fe06e110a9e2c9" Oct 29 23:33:42.638711 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:42.638685 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba185bb1c59c6d13d01637228fd6f3dc866f5ce40dff3d4253fe06e110a9e2c9"} err="failed to get container status \"ba185bb1c59c6d13d01637228fd6f3dc866f5ce40dff3d4253fe06e110a9e2c9\": rpc error: code = NotFound desc = could not find container \"ba185bb1c59c6d13d01637228fd6f3dc866f5ce40dff3d4253fe06e110a9e2c9\": container with ID starting with ba185bb1c59c6d13d01637228fd6f3dc866f5ce40dff3d4253fe06e110a9e2c9 not found: ID does not exist" Oct 29 23:33:42.675881 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:42.675858 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-08e15d-6bd798d867-mnq4t"] Oct 29 23:33:42.686870 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:42.686843 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-08e15d-6bd798d867-mnq4t"] Oct 29 23:33:43.328977 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:43.328939 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="684cec70-0bb2-4eff-917e-6a04c8734d74" path="/var/lib/kubelet/pods/684cec70-0bb2-4eff-917e-6a04c8734d74/volumes" Oct 29 23:33:44.956173 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:44.956143 5057 scope.go:122] "RemoveContainer" containerID="9d7d1d93c53e6b0dd2e85cbfbb91e3fe81dc4305337936c9f8a3cc8e554d1f9a" Oct 29 23:33:44.967018 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:44.966998 5057 scope.go:122] "RemoveContainer" containerID="7aa13ef41f18e9c3406f2adbcde5d172261970c2edbe179c1b8191531624bcb2" Oct 29 23:33:44.976837 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:44.976811 5057 scope.go:122] "RemoveContainer" containerID="a362653d5a9b4c999fa6c9862f67171d87c2483245fbf5446675f0caf135a30e" Oct 29 23:33:44.985513 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:44.985494 5057 scope.go:122] "RemoveContainer" containerID="0df22865870a0255fc37a1e98900b2ebe6a64cf652d48a94c7b571d21032fb99" Oct 29 23:33:45.233805 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:45.233730 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:33:45.366380 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:45.366326 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:33:45.544976 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:45.544887 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-r8g84"] Oct 29 23:33:45.545200 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:45.545150 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-r8g84" podUID="f6c2cdc5-4b87-43a0-a1a0-258e925d849c" containerName="istio-proxy" containerID="cri-o://076e34911feb1ca961414be82bbb939094b236e50500069ffa6215f2a5e68dd1" gracePeriod=30 Oct 29 23:33:45.589351 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:45.587894 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:33:45.739099 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:45.739061 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:33:45.808636 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:45.808561 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-08e15d/distinct-w0-08e15d-kserve-85695c78bb-p67gk" podUID="e664f012-3b80-4a8a-ac19-d767d92c0a33" containerName="main" containerID="cri-o://90918fdd096be65fde5ed90d07b1582c4de7dffa60bed26cc6a010a2bdaf19ba" gracePeriod=45 Oct 29 23:33:45.850560 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:45.850532 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:33:46.112433 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:46.112395 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-08e15d/distinct-w0-08e15d-kserve-85695c78bb-p67gk" Oct 29 23:33:46.207244 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:46.207210 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e664f012-3b80-4a8a-ac19-d767d92c0a33-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e664f012-3b80-4a8a-ac19-d767d92c0a33-dshm\") pod \"e664f012-3b80-4a8a-ac19-d767d92c0a33\" (UID: \"e664f012-3b80-4a8a-ac19-d767d92c0a33\") " Oct 29 23:33:46.207434 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:46.207294 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e664f012-3b80-4a8a-ac19-d767d92c0a33-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e664f012-3b80-4a8a-ac19-d767d92c0a33-model-cache\") pod \"e664f012-3b80-4a8a-ac19-d767d92c0a33\" (UID: \"e664f012-3b80-4a8a-ac19-d767d92c0a33\") " Oct 29 23:33:46.207434 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:46.207403 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e664f012-3b80-4a8a-ac19-d767d92c0a33-kube-api-access-v7rc9\" (UniqueName: \"kubernetes.io/projected/e664f012-3b80-4a8a-ac19-d767d92c0a33-kube-api-access-v7rc9\") pod \"e664f012-3b80-4a8a-ac19-d767d92c0a33\" (UID: \"e664f012-3b80-4a8a-ac19-d767d92c0a33\") " Oct 29 23:33:46.207512 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:46.207470 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e664f012-3b80-4a8a-ac19-d767d92c0a33-tls-certs\" (UniqueName: \"kubernetes.io/secret/e664f012-3b80-4a8a-ac19-d767d92c0a33-tls-certs\") pod \"e664f012-3b80-4a8a-ac19-d767d92c0a33\" (UID: \"e664f012-3b80-4a8a-ac19-d767d92c0a33\") " Oct 29 23:33:46.207512 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:46.207498 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e664f012-3b80-4a8a-ac19-d767d92c0a33-home\" (UniqueName: \"kubernetes.io/empty-dir/e664f012-3b80-4a8a-ac19-d767d92c0a33-home\") pod \"e664f012-3b80-4a8a-ac19-d767d92c0a33\" (UID: \"e664f012-3b80-4a8a-ac19-d767d92c0a33\") " Oct 29 23:33:46.207838 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:46.207623 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e664f012-3b80-4a8a-ac19-d767d92c0a33-model-cache" pod "e664f012-3b80-4a8a-ac19-d767d92c0a33" (UID: "e664f012-3b80-4a8a-ac19-d767d92c0a33"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:33:46.207838 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:46.207822 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e664f012-3b80-4a8a-ac19-d767d92c0a33-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e664f012-3b80-4a8a-ac19-d767d92c0a33-tmp-dir\") pod \"e664f012-3b80-4a8a-ac19-d767d92c0a33\" (UID: \"e664f012-3b80-4a8a-ac19-d767d92c0a33\") " Oct 29 23:33:46.208109 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:46.208065 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e664f012-3b80-4a8a-ac19-d767d92c0a33-tmp-dir" pod "e664f012-3b80-4a8a-ac19-d767d92c0a33" (UID: "e664f012-3b80-4a8a-ac19-d767d92c0a33"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:33:46.208109 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:46.208084 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e664f012-3b80-4a8a-ac19-d767d92c0a33-home" pod "e664f012-3b80-4a8a-ac19-d767d92c0a33" (UID: "e664f012-3b80-4a8a-ac19-d767d92c0a33"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:33:46.208423 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:46.208399 5057 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e664f012-3b80-4a8a-ac19-d767d92c0a33-tmp-dir\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:33:46.208511 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:46.208429 5057 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e664f012-3b80-4a8a-ac19-d767d92c0a33-model-cache\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:33:46.208511 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:46.208444 5057 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e664f012-3b80-4a8a-ac19-d767d92c0a33-home\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:33:46.210217 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:46.210189 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e664f012-3b80-4a8a-ac19-d767d92c0a33-kube-api-access-v7rc9" pod "e664f012-3b80-4a8a-ac19-d767d92c0a33" (UID: "e664f012-3b80-4a8a-ac19-d767d92c0a33"). InnerVolumeSpecName "kube-api-access-v7rc9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:33:46.210354 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:46.210277 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e664f012-3b80-4a8a-ac19-d767d92c0a33-tls-certs" pod "e664f012-3b80-4a8a-ac19-d767d92c0a33" (UID: "e664f012-3b80-4a8a-ac19-d767d92c0a33"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:33:46.210627 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:46.210605 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e664f012-3b80-4a8a-ac19-d767d92c0a33-dshm" pod "e664f012-3b80-4a8a-ac19-d767d92c0a33" (UID: "e664f012-3b80-4a8a-ac19-d767d92c0a33"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:33:46.309145 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:46.309112 5057 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e664f012-3b80-4a8a-ac19-d767d92c0a33-dshm\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:33:46.309145 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:46.309143 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v7rc9\" (UniqueName: \"kubernetes.io/projected/e664f012-3b80-4a8a-ac19-d767d92c0a33-kube-api-access-v7rc9\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:33:46.309390 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:46.309154 5057 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e664f012-3b80-4a8a-ac19-d767d92c0a33-tls-certs\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:33:46.660210 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:46.660173 5057 generic.go:356] "Generic (PLEG): container finished" podID="e664f012-3b80-4a8a-ac19-d767d92c0a33" containerID="90918fdd096be65fde5ed90d07b1582c4de7dffa60bed26cc6a010a2bdaf19ba" exitCode=0 Oct 29 23:33:46.660459 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:46.660225 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-08e15d/distinct-w0-08e15d-kserve-85695c78bb-p67gk" Oct 29 23:33:46.660459 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:46.660231 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-08e15d/distinct-w0-08e15d-kserve-85695c78bb-p67gk" event={"ID":"e664f012-3b80-4a8a-ac19-d767d92c0a33","Type":"ContainerDied","Data":"90918fdd096be65fde5ed90d07b1582c4de7dffa60bed26cc6a010a2bdaf19ba"} Oct 29 23:33:46.660459 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:46.660359 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-08e15d/distinct-w0-08e15d-kserve-85695c78bb-p67gk" event={"ID":"e664f012-3b80-4a8a-ac19-d767d92c0a33","Type":"ContainerDied","Data":"bd3bb9de238db28a11e1e2b25ce393dc7e441c105ec3d05a75e783dfc770afad"} Oct 29 23:33:46.660459 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:46.660387 5057 scope.go:122] "RemoveContainer" containerID="90918fdd096be65fde5ed90d07b1582c4de7dffa60bed26cc6a010a2bdaf19ba" Oct 29 23:33:46.671496 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:46.671476 5057 scope.go:122] "RemoveContainer" containerID="90918fdd096be65fde5ed90d07b1582c4de7dffa60bed26cc6a010a2bdaf19ba" Oct 29 23:33:46.671790 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:33:46.671772 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90918fdd096be65fde5ed90d07b1582c4de7dffa60bed26cc6a010a2bdaf19ba\": container with ID starting with 90918fdd096be65fde5ed90d07b1582c4de7dffa60bed26cc6a010a2bdaf19ba not found: ID does not exist" containerID="90918fdd096be65fde5ed90d07b1582c4de7dffa60bed26cc6a010a2bdaf19ba" Oct 29 23:33:46.671834 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:46.671806 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90918fdd096be65fde5ed90d07b1582c4de7dffa60bed26cc6a010a2bdaf19ba"} err="failed to get container status \"90918fdd096be65fde5ed90d07b1582c4de7dffa60bed26cc6a010a2bdaf19ba\": rpc error: code = NotFound desc = could not find container \"90918fdd096be65fde5ed90d07b1582c4de7dffa60bed26cc6a010a2bdaf19ba\": container with ID starting with 90918fdd096be65fde5ed90d07b1582c4de7dffa60bed26cc6a010a2bdaf19ba not found: ID does not exist" Oct 29 23:33:46.711586 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:46.711557 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-08e15d/distinct-w0-08e15d-kserve-85695c78bb-p67gk"] Oct 29 23:33:46.723188 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:46.723161 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-08e15d/distinct-w0-08e15d-kserve-85695c78bb-p67gk"] Oct 29 23:33:47.330157 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:47.330112 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e664f012-3b80-4a8a-ac19-d767d92c0a33" path="/var/lib/kubelet/pods/e664f012-3b80-4a8a-ac19-d767d92c0a33/volumes" Oct 29 23:33:48.485416 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:48.485375 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:33:48.596138 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:48.596105 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:33:49.028853 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:33:49.028810 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d2dnn" containerName="extractor" Oct 29 23:33:49.147892 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:49.147847 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:33:50.693405 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:50.693375 5057 generic.go:356] "Generic (PLEG): container finished" podID="f6c2cdc5-4b87-43a0-a1a0-258e925d849c" containerID="076e34911feb1ca961414be82bbb939094b236e50500069ffa6215f2a5e68dd1" exitCode=0 Oct 29 23:33:50.693776 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:50.693425 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-r8g84" event={"ID":"f6c2cdc5-4b87-43a0-a1a0-258e925d849c","Type":"ContainerDied","Data":"076e34911feb1ca961414be82bbb939094b236e50500069ffa6215f2a5e68dd1"} Oct 29 23:33:51.230399 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:51.230377 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-r8g84" Oct 29 23:33:51.268824 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:51.268795 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6c2cdc5-4b87-43a0-a1a0-258e925d849c-istio-token\" (UniqueName: \"kubernetes.io/projected/f6c2cdc5-4b87-43a0-a1a0-258e925d849c-istio-token\") pod \"f6c2cdc5-4b87-43a0-a1a0-258e925d849c\" (UID: \"f6c2cdc5-4b87-43a0-a1a0-258e925d849c\") " Oct 29 23:33:51.268950 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:51.268839 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6c2cdc5-4b87-43a0-a1a0-258e925d849c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6c2cdc5-4b87-43a0-a1a0-258e925d849c-workload-socket\") pod \"f6c2cdc5-4b87-43a0-a1a0-258e925d849c\" (UID: \"f6c2cdc5-4b87-43a0-a1a0-258e925d849c\") " Oct 29 23:33:51.268950 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:51.268895 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6c2cdc5-4b87-43a0-a1a0-258e925d849c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6c2cdc5-4b87-43a0-a1a0-258e925d849c-istio-envoy\") pod \"f6c2cdc5-4b87-43a0-a1a0-258e925d849c\" (UID: \"f6c2cdc5-4b87-43a0-a1a0-258e925d849c\") " Oct 29 23:33:51.268950 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:51.268945 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f6c2cdc5-4b87-43a0-a1a0-258e925d849c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6c2cdc5-4b87-43a0-a1a0-258e925d849c-istio-podinfo\") pod \"f6c2cdc5-4b87-43a0-a1a0-258e925d849c\" (UID: \"f6c2cdc5-4b87-43a0-a1a0-258e925d849c\") " Oct 29 23:33:51.269125 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:51.268993 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6c2cdc5-4b87-43a0-a1a0-258e925d849c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6c2cdc5-4b87-43a0-a1a0-258e925d849c-istio-data\") pod \"f6c2cdc5-4b87-43a0-a1a0-258e925d849c\" (UID: \"f6c2cdc5-4b87-43a0-a1a0-258e925d849c\") " Oct 29 23:33:51.269125 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:51.269027 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6c2cdc5-4b87-43a0-a1a0-258e925d849c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6c2cdc5-4b87-43a0-a1a0-258e925d849c-credential-socket\") pod \"f6c2cdc5-4b87-43a0-a1a0-258e925d849c\" (UID: \"f6c2cdc5-4b87-43a0-a1a0-258e925d849c\") " Oct 29 23:33:51.269125 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:51.269090 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6c2cdc5-4b87-43a0-a1a0-258e925d849c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6c2cdc5-4b87-43a0-a1a0-258e925d849c-workload-certs\") pod \"f6c2cdc5-4b87-43a0-a1a0-258e925d849c\" (UID: \"f6c2cdc5-4b87-43a0-a1a0-258e925d849c\") " Oct 29 23:33:51.269300 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:51.269164 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6c2cdc5-4b87-43a0-a1a0-258e925d849c-kube-api-access-2bv8f\" (UniqueName: \"kubernetes.io/projected/f6c2cdc5-4b87-43a0-a1a0-258e925d849c-kube-api-access-2bv8f\") pod \"f6c2cdc5-4b87-43a0-a1a0-258e925d849c\" (UID: \"f6c2cdc5-4b87-43a0-a1a0-258e925d849c\") " Oct 29 23:33:51.269300 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:51.269184 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6c2cdc5-4b87-43a0-a1a0-258e925d849c-workload-socket" pod "f6c2cdc5-4b87-43a0-a1a0-258e925d849c" (UID: "f6c2cdc5-4b87-43a0-a1a0-258e925d849c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:33:51.269300 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:51.269195 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f6c2cdc5-4b87-43a0-a1a0-258e925d849c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6c2cdc5-4b87-43a0-a1a0-258e925d849c-istiod-ca-cert\") pod \"f6c2cdc5-4b87-43a0-a1a0-258e925d849c\" (UID: \"f6c2cdc5-4b87-43a0-a1a0-258e925d849c\") " Oct 29 23:33:51.269300 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:51.269284 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6c2cdc5-4b87-43a0-a1a0-258e925d849c-istio-data" pod "f6c2cdc5-4b87-43a0-a1a0-258e925d849c" (UID: "f6c2cdc5-4b87-43a0-a1a0-258e925d849c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:33:51.269563 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:51.269483 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6c2cdc5-4b87-43a0-a1a0-258e925d849c-workload-certs" pod "f6c2cdc5-4b87-43a0-a1a0-258e925d849c" (UID: "f6c2cdc5-4b87-43a0-a1a0-258e925d849c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:33:51.269563 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:51.269548 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6c2cdc5-4b87-43a0-a1a0-258e925d849c-credential-socket" pod "f6c2cdc5-4b87-43a0-a1a0-258e925d849c" (UID: "f6c2cdc5-4b87-43a0-a1a0-258e925d849c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:33:51.269680 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:51.269548 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6c2cdc5-4b87-43a0-a1a0-258e925d849c-istiod-ca-cert" pod "f6c2cdc5-4b87-43a0-a1a0-258e925d849c" (UID: "f6c2cdc5-4b87-43a0-a1a0-258e925d849c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:33:51.269885 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:51.269863 5057 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6c2cdc5-4b87-43a0-a1a0-258e925d849c-workload-socket\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:33:51.269953 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:51.269890 5057 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6c2cdc5-4b87-43a0-a1a0-258e925d849c-istio-data\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:33:51.269953 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:51.269904 5057 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6c2cdc5-4b87-43a0-a1a0-258e925d849c-credential-socket\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:33:51.269953 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:51.269917 5057 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6c2cdc5-4b87-43a0-a1a0-258e925d849c-workload-certs\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:33:51.269953 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:51.269930 5057 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6c2cdc5-4b87-43a0-a1a0-258e925d849c-istiod-ca-cert\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:33:51.271075 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:51.271049 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6c2cdc5-4b87-43a0-a1a0-258e925d849c-istio-token" pod "f6c2cdc5-4b87-43a0-a1a0-258e925d849c" (UID: "f6c2cdc5-4b87-43a0-a1a0-258e925d849c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:33:51.271193 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:51.271102 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6c2cdc5-4b87-43a0-a1a0-258e925d849c-istio-envoy" pod "f6c2cdc5-4b87-43a0-a1a0-258e925d849c" (UID: "f6c2cdc5-4b87-43a0-a1a0-258e925d849c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:33:51.271470 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:51.271453 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f6c2cdc5-4b87-43a0-a1a0-258e925d849c-istio-podinfo" pod "f6c2cdc5-4b87-43a0-a1a0-258e925d849c" (UID: "f6c2cdc5-4b87-43a0-a1a0-258e925d849c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:33:51.271573 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:51.271550 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6c2cdc5-4b87-43a0-a1a0-258e925d849c-kube-api-access-2bv8f" pod "f6c2cdc5-4b87-43a0-a1a0-258e925d849c" (UID: "f6c2cdc5-4b87-43a0-a1a0-258e925d849c"). InnerVolumeSpecName "kube-api-access-2bv8f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:33:51.370594 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:51.370556 5057 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6c2cdc5-4b87-43a0-a1a0-258e925d849c-istio-token\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:33:51.370594 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:51.370587 5057 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6c2cdc5-4b87-43a0-a1a0-258e925d849c-istio-envoy\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:33:51.370594 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:51.370599 5057 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6c2cdc5-4b87-43a0-a1a0-258e925d849c-istio-podinfo\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:33:51.370802 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:51.370612 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2bv8f\" (UniqueName: \"kubernetes.io/projected/f6c2cdc5-4b87-43a0-a1a0-258e925d849c-kube-api-access-2bv8f\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:33:51.703270 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:51.703237 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-r8g84" Oct 29 23:33:51.703729 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:51.703237 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-r8g84" event={"ID":"f6c2cdc5-4b87-43a0-a1a0-258e925d849c","Type":"ContainerDied","Data":"628cfdc341f9d3aa564fff85404c12c8215f9da56c6a17b1338bf627903e28b4"} Oct 29 23:33:51.703729 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:51.703399 5057 scope.go:122] "RemoveContainer" containerID="076e34911feb1ca961414be82bbb939094b236e50500069ffa6215f2a5e68dd1" Oct 29 23:33:51.753952 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:51.753920 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-r8g84"] Oct 29 23:33:51.765301 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:51.765274 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-r8g84"] Oct 29 23:33:53.330252 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:53.330196 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f6c2cdc5-4b87-43a0-a1a0-258e925d849c" path="/var/lib/kubelet/pods/f6c2cdc5-4b87-43a0-a1a0-258e925d849c/volumes" Oct 29 23:33:53.961021 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:53.960978 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v2tfm"] Oct 29 23:33:55.031791 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:55.031729 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:33:55.448123 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:55.448090 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:33:55.563825 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:55.563772 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:33:55.680621 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:55.680569 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:33:56.108311 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:56.108274 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:33:59.744569 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:33:59.744526 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:34:00.071908 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:34:00.071817 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:34:00.184008 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:34:00.183973 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:34:01.160640 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:34:01.160607 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:34:02.443582 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:34:02.443546 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:34:02.556913 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:34:02.556870 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:34:02.764633 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:34:02.764595 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-dmw4m"] Oct 29 23:34:02.764832 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:34:02.764807 5057 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-dmw4m" podUID="e49752e2-a116-4dd9-b6c6-712153a196f9" containerName="istio-proxy" containerID="cri-o://64e48b5dcfb6775775fef1fe42be0e24654acb0887fa1144196a619c0ff1c256" gracePeriod=30 Oct 29 23:34:02.808292 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:34:02.808260 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:34:04.444291 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:34:04.444258 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v2tfm"] Oct 29 23:34:05.298085 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:34:05.298041 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-2hpht_bd4e5443-0ff9-4a71-b6d1-bbddbbe457ab/manager/0.log" Oct 29 23:34:05.499204 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:34:05.499171 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-v2tfm_5e0f62f7-e32b-45f1-ba71-88bbb389039b/limitador/0.log" Oct 29 23:34:06.464590 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:34:06.464556 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-59d4d9f4d8-v7rk9_bb8ee125-7b3a-48c1-8a10-0d2bf17a4eb0/manager/0.log" Oct 29 23:34:06.759161 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:34:06.759072 5057 patch_prober.go:29] interesting pod/e2e-worker-w0-08e15d-openshift-default-d579fdc75-dmw4m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 23:34:06.759161 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:34:06.759147 5057 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-dmw4m" podUID="e49752e2-a116-4dd9-b6c6-712153a196f9" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 23:34:06.980087 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:34:06.980032 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:34:07.887781 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:34:07.887745 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-5k5vp_494d8090-9803-4be0-a442-6108c1ebc201/manager/0.log" Oct 29 23:34:08.041423 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:34:08.041370 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-dmw4m" Oct 29 23:34:08.156032 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:34:08.156002 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e49752e2-a116-4dd9-b6c6-712153a196f9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e49752e2-a116-4dd9-b6c6-712153a196f9-istio-podinfo\") pod \"e49752e2-a116-4dd9-b6c6-712153a196f9\" (UID: \"e49752e2-a116-4dd9-b6c6-712153a196f9\") " Oct 29 23:34:08.156221 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:34:08.156055 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e49752e2-a116-4dd9-b6c6-712153a196f9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e49752e2-a116-4dd9-b6c6-712153a196f9-istio-data\") pod \"e49752e2-a116-4dd9-b6c6-712153a196f9\" (UID: \"e49752e2-a116-4dd9-b6c6-712153a196f9\") " Oct 29 23:34:08.156221 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:34:08.156077 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e49752e2-a116-4dd9-b6c6-712153a196f9-istio-token\" (UniqueName: \"kubernetes.io/projected/e49752e2-a116-4dd9-b6c6-712153a196f9-istio-token\") pod \"e49752e2-a116-4dd9-b6c6-712153a196f9\" (UID: \"e49752e2-a116-4dd9-b6c6-712153a196f9\") " Oct 29 23:34:08.156221 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:34:08.156104 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e49752e2-a116-4dd9-b6c6-712153a196f9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e49752e2-a116-4dd9-b6c6-712153a196f9-istio-envoy\") pod \"e49752e2-a116-4dd9-b6c6-712153a196f9\" (UID: \"e49752e2-a116-4dd9-b6c6-712153a196f9\") " Oct 29 23:34:08.156221 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:34:08.156129 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e49752e2-a116-4dd9-b6c6-712153a196f9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e49752e2-a116-4dd9-b6c6-712153a196f9-workload-socket\") pod \"e49752e2-a116-4dd9-b6c6-712153a196f9\" (UID: \"e49752e2-a116-4dd9-b6c6-712153a196f9\") " Oct 29 23:34:08.156504 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:34:08.156241 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e49752e2-a116-4dd9-b6c6-712153a196f9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e49752e2-a116-4dd9-b6c6-712153a196f9-credential-socket\") pod \"e49752e2-a116-4dd9-b6c6-712153a196f9\" (UID: \"e49752e2-a116-4dd9-b6c6-712153a196f9\") " Oct 29 23:34:08.156504 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:34:08.156305 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e49752e2-a116-4dd9-b6c6-712153a196f9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e49752e2-a116-4dd9-b6c6-712153a196f9-workload-certs\") pod \"e49752e2-a116-4dd9-b6c6-712153a196f9\" (UID: \"e49752e2-a116-4dd9-b6c6-712153a196f9\") " Oct 29 23:34:08.156504 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:34:08.156307 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e49752e2-a116-4dd9-b6c6-712153a196f9-istio-data" pod "e49752e2-a116-4dd9-b6c6-712153a196f9" (UID: "e49752e2-a116-4dd9-b6c6-712153a196f9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:34:08.156504 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:34:08.156353 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e49752e2-a116-4dd9-b6c6-712153a196f9-kube-api-access-4kt26\" (UniqueName: \"kubernetes.io/projected/e49752e2-a116-4dd9-b6c6-712153a196f9-kube-api-access-4kt26\") pod \"e49752e2-a116-4dd9-b6c6-712153a196f9\" (UID: \"e49752e2-a116-4dd9-b6c6-712153a196f9\") " Oct 29 23:34:08.156504 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:34:08.156407 5057 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e49752e2-a116-4dd9-b6c6-712153a196f9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e49752e2-a116-4dd9-b6c6-712153a196f9-istiod-ca-cert\") pod \"e49752e2-a116-4dd9-b6c6-712153a196f9\" (UID: \"e49752e2-a116-4dd9-b6c6-712153a196f9\") " Oct 29 23:34:08.156799 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:34:08.156517 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e49752e2-a116-4dd9-b6c6-712153a196f9-workload-socket" pod "e49752e2-a116-4dd9-b6c6-712153a196f9" (UID: "e49752e2-a116-4dd9-b6c6-712153a196f9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:34:08.156799 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:34:08.156588 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e49752e2-a116-4dd9-b6c6-712153a196f9-workload-certs" pod "e49752e2-a116-4dd9-b6c6-712153a196f9" (UID: "e49752e2-a116-4dd9-b6c6-712153a196f9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:34:08.156799 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:34:08.156609 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e49752e2-a116-4dd9-b6c6-712153a196f9-credential-socket" pod "e49752e2-a116-4dd9-b6c6-712153a196f9" (UID: "e49752e2-a116-4dd9-b6c6-712153a196f9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:34:08.156984 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:34:08.156833 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e49752e2-a116-4dd9-b6c6-712153a196f9-istiod-ca-cert" pod "e49752e2-a116-4dd9-b6c6-712153a196f9" (UID: "e49752e2-a116-4dd9-b6c6-712153a196f9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:34:08.156984 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:34:08.156964 5057 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e49752e2-a116-4dd9-b6c6-712153a196f9-credential-socket\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:34:08.156984 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:34:08.156981 5057 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e49752e2-a116-4dd9-b6c6-712153a196f9-workload-certs\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:34:08.157159 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:34:08.156991 5057 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e49752e2-a116-4dd9-b6c6-712153a196f9-istiod-ca-cert\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:34:08.157159 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:34:08.156998 5057 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e49752e2-a116-4dd9-b6c6-712153a196f9-istio-data\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:34:08.157159 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:34:08.157006 5057 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e49752e2-a116-4dd9-b6c6-712153a196f9-workload-socket\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:34:08.158546 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:34:08.158521 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e49752e2-a116-4dd9-b6c6-712153a196f9-istio-token" pod "e49752e2-a116-4dd9-b6c6-712153a196f9" (UID: "e49752e2-a116-4dd9-b6c6-712153a196f9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:34:08.158681 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:34:08.158543 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e49752e2-a116-4dd9-b6c6-712153a196f9-kube-api-access-4kt26" pod "e49752e2-a116-4dd9-b6c6-712153a196f9" (UID: "e49752e2-a116-4dd9-b6c6-712153a196f9"). InnerVolumeSpecName "kube-api-access-4kt26". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:34:08.158681 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:34:08.158672 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e49752e2-a116-4dd9-b6c6-712153a196f9-istio-podinfo" pod "e49752e2-a116-4dd9-b6c6-712153a196f9" (UID: "e49752e2-a116-4dd9-b6c6-712153a196f9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:34:08.158809 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:34:08.158747 5057 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e49752e2-a116-4dd9-b6c6-712153a196f9-istio-envoy" pod "e49752e2-a116-4dd9-b6c6-712153a196f9" (UID: "e49752e2-a116-4dd9-b6c6-712153a196f9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:34:08.258561 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:34:08.258534 5057 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4kt26\" (UniqueName: \"kubernetes.io/projected/e49752e2-a116-4dd9-b6c6-712153a196f9-kube-api-access-4kt26\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:34:08.258561 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:34:08.258559 5057 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e49752e2-a116-4dd9-b6c6-712153a196f9-istio-podinfo\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:34:08.258726 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:34:08.258569 5057 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e49752e2-a116-4dd9-b6c6-712153a196f9-istio-token\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:34:08.258726 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:34:08.258579 5057 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e49752e2-a116-4dd9-b6c6-712153a196f9-istio-envoy\") on node \"ip-10-0-140-74.ec2.internal\" DevicePath \"\"" Oct 29 23:34:08.823143 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:34:08.823079 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:34:08.827371 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:34:08.827320 5057 generic.go:356] "Generic (PLEG): container finished" podID="e49752e2-a116-4dd9-b6c6-712153a196f9" containerID="64e48b5dcfb6775775fef1fe42be0e24654acb0887fa1144196a619c0ff1c256" exitCode=0 Oct 29 23:34:08.827494 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:34:08.827395 5057 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-dmw4m" Oct 29 23:34:08.827494 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:34:08.827405 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-dmw4m" event={"ID":"e49752e2-a116-4dd9-b6c6-712153a196f9","Type":"ContainerDied","Data":"64e48b5dcfb6775775fef1fe42be0e24654acb0887fa1144196a619c0ff1c256"} Oct 29 23:34:08.827494 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:34:08.827461 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-dmw4m" event={"ID":"e49752e2-a116-4dd9-b6c6-712153a196f9","Type":"ContainerDied","Data":"4cbbdbf442485c76719fc3470187146a5203c9b9c3825384ba05b344f9d595ab"} Oct 29 23:34:08.827494 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:34:08.827484 5057 scope.go:122] "RemoveContainer" containerID="64e48b5dcfb6775775fef1fe42be0e24654acb0887fa1144196a619c0ff1c256" Oct 29 23:34:08.841608 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:34:08.841588 5057 scope.go:122] "RemoveContainer" containerID="64e48b5dcfb6775775fef1fe42be0e24654acb0887fa1144196a619c0ff1c256" Oct 29 23:34:08.841942 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:34:08.841914 5057 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64e48b5dcfb6775775fef1fe42be0e24654acb0887fa1144196a619c0ff1c256\": container with ID starting with 64e48b5dcfb6775775fef1fe42be0e24654acb0887fa1144196a619c0ff1c256 not found: ID does not exist" containerID="64e48b5dcfb6775775fef1fe42be0e24654acb0887fa1144196a619c0ff1c256" Oct 29 23:34:08.842060 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:34:08.841948 5057 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64e48b5dcfb6775775fef1fe42be0e24654acb0887fa1144196a619c0ff1c256"} err="failed to get container status \"64e48b5dcfb6775775fef1fe42be0e24654acb0887fa1144196a619c0ff1c256\": rpc error: code = NotFound desc = could not find container \"64e48b5dcfb6775775fef1fe42be0e24654acb0887fa1144196a619c0ff1c256\": container with ID starting with 64e48b5dcfb6775775fef1fe42be0e24654acb0887fa1144196a619c0ff1c256 not found: ID does not exist" Oct 29 23:34:08.892466 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:34:08.892437 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-dmw4m"] Oct 29 23:34:08.903024 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:34:08.902993 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-dmw4m"] Oct 29 23:34:09.329163 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:34:09.329120 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e49752e2-a116-4dd9-b6c6-712153a196f9" path="/var/lib/kubelet/pods/e49752e2-a116-4dd9-b6c6-712153a196f9/volumes" Oct 29 23:34:19.027535 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:34:19.027495 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d2dnn" containerName="extractor" Oct 29 23:34:19.269947 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:34:19.269916 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-59d4d9f4d8-v7rk9_bb8ee125-7b3a-48c1-8a10-0d2bf17a4eb0/manager/0.log" Oct 29 23:34:20.705158 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:34:20.705069 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-5k5vp_494d8090-9803-4be0-a442-6108c1ebc201/manager/0.log" Oct 29 23:34:21.661885 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:34:21.661848 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-8457c8746d-4lscp_f1fa4ed3-77b6-4d9b-b31d-08c26c2bd492/maas-api/0.log" Oct 29 23:34:21.898347 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:34:21.898308 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-35e58700-pxt8s_5db04267-4c0f-493e-8a0d-f4d37995cd2b/revoke-keys/0.log" Oct 29 23:34:22.264077 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:34:22.264041 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-5babc19a-cjh8n_2cb5a4ef-c702-49a4-83e0-0d1f0ff1a549/revoke-keys/0.log" Oct 29 23:34:22.460455 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:34:22.460423 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-65a313ba-f92ch_5be39f9d-8c3a-4450-bd67-b1a17ee56be7/revoke-keys/0.log" Oct 29 23:34:22.836462 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:34:22.836432 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-7b1f808d-jv69s_d60d011e-d215-4b9d-ae66-62cdf573a9ed/revoke-keys/0.log" Oct 29 23:34:23.400096 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:34:23.400059 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-94e8042f-pxbc5_189db1b7-a3e4-4dd9-90aa-811ec21845e3/revoke-keys/0.log" Oct 29 23:34:23.591123 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:34:23.591086 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-9e50149e-slmz8_a1851a9f-eb0f-493f-87f3-6246aaa79dde/revoke-keys/0.log" Oct 29 23:34:24.137825 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:34:24.137792 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-b18b56d5-jmkc5_ce4fe136-400e-4285-9af1-bfcd51109ae2/revoke-keys/0.log" Oct 29 23:34:24.673974 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:34:24.673943 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-c1ce6b81-pdp5g_36aa03cd-9d75-41dd-9e2d-401a3792be1b/revoke-keys/0.log" Oct 29 23:34:25.214141 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:34:25.214110 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-e26a6b27-6wjrq_81e3720d-b019-448f-9a08-49d5ccad4cb1/revoke-keys/0.log" Oct 29 23:34:29.119611 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:34:29.119574 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-2hpht_bd4e5443-0ff9-4a71-b6d1-bbddbbe457ab/manager/0.log" Oct 29 23:34:29.303108 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:34:29.303076 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-v2tfm_5e0f62f7-e32b-45f1-ba71-88bbb389039b/limitador/0.log" Oct 29 23:34:31.416107 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:34:31.416070 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6db586dd6c-cm4wj_4e861691-eac4-4f06-a6aa-b6553efdd358/istio-proxy/0.log" Oct 29 23:34:31.968542 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:34:31.968511 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6db586dd6c-nqs5j_e92e0a25-60cc-4250-a75d-537ed970fd75/istio-proxy/0.log" Oct 29 23:34:32.154880 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:34:32.154839 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6db586dd6c-vm822_ef4e2b67-0954-4fe1-a2b4-223771c24e79/istio-proxy/0.log" Oct 29 23:34:32.698724 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:34:32.698692 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-vsszj_6b4fbaae-ed3b-4a78-b38c-b3a245bf77da/payload-pre-processing/0.log" Oct 29 23:34:33.099362 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:34:33.099269 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5d8d7cc579-8cwjp_71d80eec-b5e7-41d0-b58d-8a8d9a6c769c/router/0.log" Oct 29 23:34:34.106873 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:34:34.106840 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-dbdjq_3ee5747c-e4e9-48f6-ad11-ab3673116724/main/0.log" Oct 29 23:34:34.490551 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:34:34.490522 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-vp6zb_3254fb15-5180-4c0e-afdb-69062103f3e9/main/0.log" Oct 29 23:34:34.857237 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:34:34.857163 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-jqfzk_cee3c70c-dc5a-4fae-9d28-820fc4dea236/main/0.log" Oct 29 23:34:35.256294 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:34:35.256261 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-w67ch_35c5b0e2-86a7-4408-83e3-1799bf6d66c8/main/0.log" Oct 29 23:34:35.442143 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:34:35.442084 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-2wlhw_8c5a5036-5eab-4356-833d-ffde96608b46/main/0.log" Oct 29 23:34:37.156358 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:34:37.156294 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:34:45.069269 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:34:45.069226 5057 scope.go:122] "RemoveContainer" containerID="6bb78445931729089d2b4c79d44dd7df626154a2304a24581aaa5db386360beb" Oct 29 23:34:46.011385 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:34:46.011317 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:34:46.152093 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:34:46.152052 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:34:49.026816 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:34:49.026775 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d2dnn" containerName="extractor" Oct 29 23:34:51.030128 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:34:51.030096 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b18b56d5-jmkc5"] Oct 29 23:34:51.041530 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:34:51.041503 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b18b56d5-jmkc5"] Oct 29 23:34:51.328818 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:34:51.328738 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce4fe136-400e-4285-9af1-bfcd51109ae2" path="/var/lib/kubelet/pods/ce4fe136-400e-4285-9af1-bfcd51109ae2/volumes" Oct 29 23:34:52.701927 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:34:52.701893 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-bp46g_6f50ee57-a710-4325-bd45-4f861452ede4/global-pull-secret-syncer/0.log" Oct 29 23:34:53.277902 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:34:53.277873 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-5fjlz_79ad3230-163e-447d-9f04-a072479ff957/connection-checker/0.log" Oct 29 23:34:53.406734 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:34:53.406708 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-jgvr6_587893f5-e85a-46a9-829f-545a42c10056/connection-checker/0.log" Oct 29 23:34:53.536269 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:34:53.536190 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-mrbgh_4ff372bf-82aa-4007-b74c-df0c779e5271/connection-checker/0.log" Oct 29 23:34:53.730884 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:34:53.730856 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" Oct 29 23:34:54.481699 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:34:54.481661 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-140-74.ec2.internal_ef9ac0a87cdfc5ee9b8eaa2797bc55b1/haproxy/0.log" Oct 29 23:35:02.032614 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:35:02.032571 5057 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1ce6b81-pdp5g"] Oct 29 23:35:02.044508 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:35:02.044472 5057 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1ce6b81-pdp5g"] Oct 29 23:35:02.189140 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:35:02.189107 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-2hpht_bd4e5443-0ff9-4a71-b6d1-bbddbbe457ab/manager/0.log" Oct 29 23:35:02.296461 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:35:02.296382 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-v2tfm_5e0f62f7-e32b-45f1-ba71-88bbb389039b/limitador/0.log" Oct 29 23:35:03.327596 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:35:03.327558 5057 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="36aa03cd-9d75-41dd-9e2d-401a3792be1b" path="/var/lib/kubelet/pods/36aa03cd-9d75-41dd-9e2d-401a3792be1b/volumes" Oct 29 23:35:04.189994 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:35:04.189968 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_f2fed222-16ad-4e9c-99c4-23f64db91c21/alertmanager/0.log" Oct 29 23:35:04.272050 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:35:04.271958 5057 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-zfxq6/perf-node-gather-daemonset-7hc5q"] Oct 29 23:35:04.272782 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:35:04.272762 5057 cpu_manager.go:414] "containerMap: removing container" podUID="63e3a68d-90a1-4e09-8967-a1fb5d99b87a" containerName="main" Oct 29 23:35:04.272858 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:35:04.272785 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63e3a68d-90a1-4e09-8967-a1fb5d99b87a" containerName="main" Oct 29 23:35:04.272858 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:35:04.272811 5057 cpu_manager.go:414] "containerMap: removing container" podUID="59602ea3-b37d-479e-96f3-9f72411920b4" containerName="payload-pre-processing" Oct 29 23:35:04.272858 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:35:04.272820 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59602ea3-b37d-479e-96f3-9f72411920b4" containerName="payload-pre-processing" Oct 29 23:35:04.272858 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:35:04.272832 5057 cpu_manager.go:414] "containerMap: removing container" podUID="862fe3ac-f842-48d3-a722-ab1689671efd" containerName="payload-pre-processing" Oct 29 23:35:04.272858 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:35:04.272841 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="862fe3ac-f842-48d3-a722-ab1689671efd" containerName="payload-pre-processing" Oct 29 23:35:04.272858 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:35:04.272853 5057 cpu_manager.go:414] "containerMap: removing container" podUID="f15821b0-cb92-4759-aba2-a4fe6f8e62f9" containerName="payload-processing" Oct 29 23:35:04.273064 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:35:04.272860 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f15821b0-cb92-4759-aba2-a4fe6f8e62f9" containerName="payload-processing" Oct 29 23:35:04.273064 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:35:04.272872 5057 cpu_manager.go:414] "containerMap: removing container" podUID="630519d9-72be-441c-b013-eff2d5f360cc" containerName="main" Oct 29 23:35:04.273064 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:35:04.272880 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="630519d9-72be-441c-b013-eff2d5f360cc" containerName="main" Oct 29 23:35:04.273064 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:35:04.272897 5057 cpu_manager.go:414] "containerMap: removing container" podUID="aa71d8ce-5a91-4d79-bd0a-8f9997a3c9c4" containerName="main" Oct 29 23:35:04.273064 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:35:04.272903 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa71d8ce-5a91-4d79-bd0a-8f9997a3c9c4" containerName="main" Oct 29 23:35:04.273064 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:35:04.272925 5057 cpu_manager.go:414] "containerMap: removing container" podUID="e664f012-3b80-4a8a-ac19-d767d92c0a33" containerName="main" Oct 29 23:35:04.273064 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:35:04.272932 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e664f012-3b80-4a8a-ac19-d767d92c0a33" containerName="main" Oct 29 23:35:04.273064 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:35:04.272948 5057 cpu_manager.go:414] "containerMap: removing container" podUID="176dd45c-5cd1-452e-bbb9-3cb0ae30205e" containerName="revoke-keys" Oct 29 23:35:04.273064 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:35:04.272955 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="176dd45c-5cd1-452e-bbb9-3cb0ae30205e" containerName="revoke-keys" Oct 29 23:35:04.273064 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:35:04.272972 5057 cpu_manager.go:414] "containerMap: removing container" podUID="8508fdad-a139-4803-affb-b63a6d52bd13" containerName="maas-api" Oct 29 23:35:04.273064 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:35:04.272979 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8508fdad-a139-4803-affb-b63a6d52bd13" containerName="maas-api" Oct 29 23:35:04.273064 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:35:04.272991 5057 cpu_manager.go:414] "containerMap: removing container" podUID="b9d46ee5-61dd-4a34-93bc-bcf127bb983a" containerName="istio-proxy" Oct 29 23:35:04.273064 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:35:04.272999 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9d46ee5-61dd-4a34-93bc-bcf127bb983a" containerName="istio-proxy" Oct 29 23:35:04.273064 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:35:04.273015 5057 cpu_manager.go:414] "containerMap: removing container" podUID="05cd9ea1-8a8c-4434-a32d-540ccf6bfd4c" containerName="payload-processing" Oct 29 23:35:04.273064 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:35:04.273022 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05cd9ea1-8a8c-4434-a32d-540ccf6bfd4c" containerName="payload-processing" Oct 29 23:35:04.273064 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:35:04.273043 5057 cpu_manager.go:414] "containerMap: removing container" podUID="e49752e2-a116-4dd9-b6c6-712153a196f9" containerName="istio-proxy" Oct 29 23:35:04.273064 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:35:04.273050 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e49752e2-a116-4dd9-b6c6-712153a196f9" containerName="istio-proxy" Oct 29 23:35:04.273645 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:35:04.273071 5057 cpu_manager.go:414] "containerMap: removing container" podUID="684cec70-0bb2-4eff-917e-6a04c8734d74" containerName="payload-pre-processing" Oct 29 23:35:04.273645 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:35:04.273077 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="684cec70-0bb2-4eff-917e-6a04c8734d74" containerName="payload-pre-processing" Oct 29 23:35:04.273645 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:35:04.273093 5057 cpu_manager.go:414] "containerMap: removing container" podUID="f6c2cdc5-4b87-43a0-a1a0-258e925d849c" containerName="istio-proxy" Oct 29 23:35:04.273645 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:35:04.273100 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6c2cdc5-4b87-43a0-a1a0-258e925d849c" containerName="istio-proxy" Oct 29 23:35:04.273645 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:35:04.273112 5057 cpu_manager.go:414] "containerMap: removing container" podUID="3c334eeb-957b-47ba-9944-c363e0f6b35e" containerName="maas-api" Oct 29 23:35:04.273645 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:35:04.273119 5057 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c334eeb-957b-47ba-9944-c363e0f6b35e" containerName="maas-api" Oct 29 23:35:04.273645 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:35:04.273250 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="176dd45c-5cd1-452e-bbb9-3cb0ae30205e" containerName="revoke-keys" Oct 29 23:35:04.273645 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:35:04.273264 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="8508fdad-a139-4803-affb-b63a6d52bd13" containerName="maas-api" Oct 29 23:35:04.273645 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:35:04.273273 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9d46ee5-61dd-4a34-93bc-bcf127bb983a" containerName="istio-proxy" Oct 29 23:35:04.273645 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:35:04.273285 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="05cd9ea1-8a8c-4434-a32d-540ccf6bfd4c" containerName="payload-processing" Oct 29 23:35:04.273645 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:35:04.273295 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="e49752e2-a116-4dd9-b6c6-712153a196f9" containerName="istio-proxy" Oct 29 23:35:04.273645 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:35:04.273304 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="684cec70-0bb2-4eff-917e-6a04c8734d74" containerName="payload-pre-processing" Oct 29 23:35:04.273645 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:35:04.273313 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6c2cdc5-4b87-43a0-a1a0-258e925d849c" containerName="istio-proxy" Oct 29 23:35:04.273645 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:35:04.273320 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c334eeb-957b-47ba-9944-c363e0f6b35e" containerName="maas-api" Oct 29 23:35:04.273645 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:35:04.273347 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="63e3a68d-90a1-4e09-8967-a1fb5d99b87a" containerName="main" Oct 29 23:35:04.273645 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:35:04.273358 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="59602ea3-b37d-479e-96f3-9f72411920b4" containerName="payload-pre-processing" Oct 29 23:35:04.273645 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:35:04.273365 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="862fe3ac-f842-48d3-a722-ab1689671efd" containerName="payload-pre-processing" Oct 29 23:35:04.273645 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:35:04.273374 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="f15821b0-cb92-4759-aba2-a4fe6f8e62f9" containerName="payload-processing" Oct 29 23:35:04.273645 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:35:04.273384 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="630519d9-72be-441c-b013-eff2d5f360cc" containerName="main" Oct 29 23:35:04.273645 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:35:04.273395 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa71d8ce-5a91-4d79-bd0a-8f9997a3c9c4" containerName="main" Oct 29 23:35:04.273645 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:35:04.273406 5057 memory_manager.go:365] "RemoveStaleState removing state" podUID="e664f012-3b80-4a8a-ac19-d767d92c0a33" containerName="main" Oct 29 23:35:04.278981 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:35:04.278960 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zfxq6/perf-node-gather-daemonset-7hc5q" Oct 29 23:35:04.294779 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:35:04.293149 5057 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-zfxq6\"/\"default-dockercfg-gxv8f\"" Oct 29 23:35:04.294779 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:35:04.293715 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-zfxq6/perf-node-gather-daemonset-7hc5q"] Oct 29 23:35:04.301380 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:35:04.301359 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-zfxq6\"/\"kube-root-ca.crt\"" Oct 29 23:35:04.304848 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:35:04.304829 5057 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-zfxq6\"/\"openshift-service-ca.crt\"" Oct 29 23:35:04.323422 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:35:04.323394 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_f2fed222-16ad-4e9c-99c4-23f64db91c21/config-reloader/0.log" Oct 29 23:35:04.356797 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:35:04.356764 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zh65\" (UniqueName: \"kubernetes.io/projected/192233b7-923a-43fb-84c3-8e3cbd0d93b3-kube-api-access-5zh65\") pod \"perf-node-gather-daemonset-7hc5q\" (UID: \"192233b7-923a-43fb-84c3-8e3cbd0d93b3\") " pod="openshift-must-gather-zfxq6/perf-node-gather-daemonset-7hc5q" Oct 29 23:35:04.357211 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:35:04.356996 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/192233b7-923a-43fb-84c3-8e3cbd0d93b3-proc\") pod \"perf-node-gather-daemonset-7hc5q\" (UID: \"192233b7-923a-43fb-84c3-8e3cbd0d93b3\") " pod="openshift-must-gather-zfxq6/perf-node-gather-daemonset-7hc5q" Oct 29 23:35:04.357211 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:35:04.357066 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/192233b7-923a-43fb-84c3-8e3cbd0d93b3-sys\") pod \"perf-node-gather-daemonset-7hc5q\" (UID: \"192233b7-923a-43fb-84c3-8e3cbd0d93b3\") " pod="openshift-must-gather-zfxq6/perf-node-gather-daemonset-7hc5q" Oct 29 23:35:04.357211 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:35:04.357135 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/192233b7-923a-43fb-84c3-8e3cbd0d93b3-podres\") pod \"perf-node-gather-daemonset-7hc5q\" (UID: \"192233b7-923a-43fb-84c3-8e3cbd0d93b3\") " pod="openshift-must-gather-zfxq6/perf-node-gather-daemonset-7hc5q" Oct 29 23:35:04.357211 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:35:04.357174 5057 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/192233b7-923a-43fb-84c3-8e3cbd0d93b3-lib-modules\") pod \"perf-node-gather-daemonset-7hc5q\" (UID: \"192233b7-923a-43fb-84c3-8e3cbd0d93b3\") " pod="openshift-must-gather-zfxq6/perf-node-gather-daemonset-7hc5q" Oct 29 23:35:04.457729 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:35:04.457692 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_f2fed222-16ad-4e9c-99c4-23f64db91c21/kube-rbac-proxy-web/0.log" Oct 29 23:35:04.458016 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:35:04.457992 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zh65\" (UniqueName: \"kubernetes.io/projected/192233b7-923a-43fb-84c3-8e3cbd0d93b3-kube-api-access-5zh65\") pod \"perf-node-gather-daemonset-7hc5q\" (UID: \"192233b7-923a-43fb-84c3-8e3cbd0d93b3\") " pod="openshift-must-gather-zfxq6/perf-node-gather-daemonset-7hc5q" Oct 29 23:35:04.458135 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:35:04.458120 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/192233b7-923a-43fb-84c3-8e3cbd0d93b3-proc\") pod \"perf-node-gather-daemonset-7hc5q\" (UID: \"192233b7-923a-43fb-84c3-8e3cbd0d93b3\") " pod="openshift-must-gather-zfxq6/perf-node-gather-daemonset-7hc5q" Oct 29 23:35:04.458202 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:35:04.458175 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/192233b7-923a-43fb-84c3-8e3cbd0d93b3-sys\") pod \"perf-node-gather-daemonset-7hc5q\" (UID: \"192233b7-923a-43fb-84c3-8e3cbd0d93b3\") " pod="openshift-must-gather-zfxq6/perf-node-gather-daemonset-7hc5q" Oct 29 23:35:04.458266 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:35:04.458205 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/192233b7-923a-43fb-84c3-8e3cbd0d93b3-podres\") pod \"perf-node-gather-daemonset-7hc5q\" (UID: \"192233b7-923a-43fb-84c3-8e3cbd0d93b3\") " pod="openshift-must-gather-zfxq6/perf-node-gather-daemonset-7hc5q" Oct 29 23:35:04.458266 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:35:04.458230 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/192233b7-923a-43fb-84c3-8e3cbd0d93b3-proc\") pod \"perf-node-gather-daemonset-7hc5q\" (UID: \"192233b7-923a-43fb-84c3-8e3cbd0d93b3\") " pod="openshift-must-gather-zfxq6/perf-node-gather-daemonset-7hc5q" Oct 29 23:35:04.458266 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:35:04.458242 5057 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/192233b7-923a-43fb-84c3-8e3cbd0d93b3-lib-modules\") pod \"perf-node-gather-daemonset-7hc5q\" (UID: \"192233b7-923a-43fb-84c3-8e3cbd0d93b3\") " pod="openshift-must-gather-zfxq6/perf-node-gather-daemonset-7hc5q" Oct 29 23:35:04.458266 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:35:04.458247 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/192233b7-923a-43fb-84c3-8e3cbd0d93b3-sys\") pod \"perf-node-gather-daemonset-7hc5q\" (UID: \"192233b7-923a-43fb-84c3-8e3cbd0d93b3\") " pod="openshift-must-gather-zfxq6/perf-node-gather-daemonset-7hc5q" Oct 29 23:35:04.458590 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:35:04.458288 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/192233b7-923a-43fb-84c3-8e3cbd0d93b3-podres\") pod \"perf-node-gather-daemonset-7hc5q\" (UID: \"192233b7-923a-43fb-84c3-8e3cbd0d93b3\") " pod="openshift-must-gather-zfxq6/perf-node-gather-daemonset-7hc5q" Oct 29 23:35:04.458590 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:35:04.458375 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/192233b7-923a-43fb-84c3-8e3cbd0d93b3-lib-modules\") pod \"perf-node-gather-daemonset-7hc5q\" (UID: \"192233b7-923a-43fb-84c3-8e3cbd0d93b3\") " pod="openshift-must-gather-zfxq6/perf-node-gather-daemonset-7hc5q" Oct 29 23:35:04.480906 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:35:04.480879 5057 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zh65\" (UniqueName: \"kubernetes.io/projected/192233b7-923a-43fb-84c3-8e3cbd0d93b3-kube-api-access-5zh65\") pod \"perf-node-gather-daemonset-7hc5q\" (UID: \"192233b7-923a-43fb-84c3-8e3cbd0d93b3\") " pod="openshift-must-gather-zfxq6/perf-node-gather-daemonset-7hc5q" Oct 29 23:35:04.595287 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:35:04.595253 5057 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zfxq6/perf-node-gather-daemonset-7hc5q" Oct 29 23:35:04.599803 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:35:04.599778 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_f2fed222-16ad-4e9c-99c4-23f64db91c21/kube-rbac-proxy/0.log" Oct 29 23:35:04.611466 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:35:04.611428 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-59d4d9f4d8-v7rk9_bb8ee125-7b3a-48c1-8a10-0d2bf17a4eb0/manager/0.log" Oct 29 23:35:04.754058 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:35:04.753930 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_f2fed222-16ad-4e9c-99c4-23f64db91c21/kube-rbac-proxy-metric/0.log" Oct 29 23:35:04.761341 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:35:04.761304 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-59d4d9f4d8-v7rk9_bb8ee125-7b3a-48c1-8a10-0d2bf17a4eb0/manager/0.log" Oct 29 23:35:04.880308 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:35:04.880278 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_f2fed222-16ad-4e9c-99c4-23f64db91c21/prom-label-proxy/0.log" Oct 29 23:35:04.902630 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:35:04.902606 5057 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-zfxq6/perf-node-gather-daemonset-7hc5q"] Oct 29 23:35:04.904560 ip-10-0-140-74 kubenswrapper[5057]: W1029 23:35:04.904530 5057 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod192233b7_923a_43fb_84c3_8e3cbd0d93b3.slice/crio-3a8e3e657572757ab8a90a09590d31d6cba8d360acb3df7bfbf942099ccd4757 WatchSource:0}: Error finding container 3a8e3e657572757ab8a90a09590d31d6cba8d360acb3df7bfbf942099ccd4757: Status 404 returned error can't find the container with id 3a8e3e657572757ab8a90a09590d31d6cba8d360acb3df7bfbf942099ccd4757 Oct 29 23:35:05.015526 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:35:05.015458 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_f2fed222-16ad-4e9c-99c4-23f64db91c21/init-config-reloader/0.log" Oct 29 23:35:05.182019 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:35:05.181986 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-884576c57-8c5hr_641f9ec1-c8e2-4774-9d4f-6a7a2da97b6f/cluster-monitoring-operator/0.log" Oct 29 23:35:05.198232 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:35:05.198192 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zfxq6/perf-node-gather-daemonset-7hc5q" event={"ID":"192233b7-923a-43fb-84c3-8e3cbd0d93b3","Type":"ContainerStarted","Data":"2ac45629878bba7390a52b079b6a9f7a7045ee809fa43db0adfabc7db6f2d429"} Oct 29 23:35:05.198465 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:35:05.198255 5057 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zfxq6/perf-node-gather-daemonset-7hc5q" event={"ID":"192233b7-923a-43fb-84c3-8e3cbd0d93b3","Type":"ContainerStarted","Data":"3a8e3e657572757ab8a90a09590d31d6cba8d360acb3df7bfbf942099ccd4757"} Oct 29 23:35:05.198465 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:35:05.198285 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-zfxq6/perf-node-gather-daemonset-7hc5q" Oct 29 23:35:05.240816 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:35:05.240754 5057 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-zfxq6/perf-node-gather-daemonset-7hc5q" podStartSLOduration=1.240738892 podStartE2EDuration="1.240738892s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 23:35:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:35:05.239494378 +0000 UTC m=+3686.022707805" watchObservedRunningTime="2026-10-29 23:35:05.240738892 +0000 UTC m=+3686.023952346" Oct 29 23:35:05.294784 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:35:05.294712 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-pfvrr_689c465f-09b8-42cb-a382-5a59526bb3b5/kube-state-metrics/0.log" Oct 29 23:35:05.409279 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:35:05.409244 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-pfvrr_689c465f-09b8-42cb-a382-5a59526bb3b5/kube-rbac-proxy-main/0.log" Oct 29 23:35:05.534273 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:35:05.534222 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-pfvrr_689c465f-09b8-42cb-a382-5a59526bb3b5/kube-rbac-proxy-self/0.log" Oct 29 23:35:05.667726 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:35:05.667630 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-6f9dbf8897-m6njl_e02743b7-7cc7-44d9-a69b-26cd1a0c7f55/metrics-server/0.log" Oct 29 23:35:05.795770 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:35:05.795737 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-77f4f76594-mjnxx_957c3493-0edc-4c9a-a85b-9ed7b23cb7ba/monitoring-plugin/0.log" Oct 29 23:35:05.927975 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:35:05.927893 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-pcsj6_a5b6a0e8-ad7a-4a47-a333-48e0d3b9eced/node-exporter/0.log" Oct 29 23:35:06.050415 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:35:06.050380 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-pcsj6_a5b6a0e8-ad7a-4a47-a333-48e0d3b9eced/kube-rbac-proxy/0.log" Oct 29 23:35:06.174231 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:35:06.174204 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-pcsj6_a5b6a0e8-ad7a-4a47-a333-48e0d3b9eced/init-textfile/0.log" Oct 29 23:35:06.597522 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:35:06.597476 5057 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54f9x_2d2d7446-0741-4144-9b19-608854f8a04e/konnectivity-agent/0.log" limit=1024 Oct 29 23:35:07.048609 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:35:07.048566 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-qjcwh_3cdc2c68-0a75-426d-ab02-ab7f6af525be/kube-rbac-proxy-main/0.log" Oct 29 23:35:07.173772 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:35:07.173729 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-qjcwh_3cdc2c68-0a75-426d-ab02-ab7f6af525be/kube-rbac-proxy-self/0.log" Oct 29 23:35:07.299908 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:35:07.299814 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-qjcwh_3cdc2c68-0a75-426d-ab02-ab7f6af525be/openshift-state-metrics/0.log" Oct 29 23:35:07.437849 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:35:07.437817 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_ef110980-d6dc-44d9-8cbf-ec396e99db44/prometheus/0.log" Oct 29 23:35:07.563360 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:35:07.563257 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_ef110980-d6dc-44d9-8cbf-ec396e99db44/config-reloader/0.log" Oct 29 23:35:07.688358 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:35:07.688299 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_ef110980-d6dc-44d9-8cbf-ec396e99db44/thanos-sidecar/0.log" Oct 29 23:35:07.811896 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:35:07.811865 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_ef110980-d6dc-44d9-8cbf-ec396e99db44/kube-rbac-proxy-web/0.log" Oct 29 23:35:07.916033 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:35:07.915939 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-5k5vp_494d8090-9803-4be0-a442-6108c1ebc201/manager/0.log" Oct 29 23:35:07.935880 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:35:07.935850 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_ef110980-d6dc-44d9-8cbf-ec396e99db44/kube-rbac-proxy/0.log" Oct 29 23:35:08.053221 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:35:08.053187 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-5k5vp_494d8090-9803-4be0-a442-6108c1ebc201/manager/0.log" Oct 29 23:35:08.067915 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:35:08.067891 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_ef110980-d6dc-44d9-8cbf-ec396e99db44/kube-rbac-proxy-thanos/0.log" Oct 29 23:35:08.192574 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:35:08.192512 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_ef110980-d6dc-44d9-8cbf-ec396e99db44/init-config-reloader/0.log" Oct 29 23:35:08.330949 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:35:08.330921 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-jz2pw_4f1d0a5c-7cc7-4f1f-88d1-f1b1e6945963/prometheus-operator/0.log" Oct 29 23:35:08.444738 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:35:08.444669 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-jz2pw_4f1d0a5c-7cc7-4f1f-88d1-f1b1e6945963/kube-rbac-proxy/0.log" Oct 29 23:35:08.568599 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:35:08.568574 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-6489d8bf89-hsxj9_b83c5586-450b-46e6-9759-c3d94b50da13/prometheus-operator-admission-webhook/0.log" Oct 29 23:35:08.700061 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:35:08.700038 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-559f47964f-9wc6l_47845ebc-dce7-4409-9fc9-ffe9ce036c77/telemeter-client/0.log" Oct 29 23:35:08.823293 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:35:08.823265 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-559f47964f-9wc6l_47845ebc-dce7-4409-9fc9-ffe9ce036c77/reload/0.log" Oct 29 23:35:08.947511 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:35:08.947476 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-559f47964f-9wc6l_47845ebc-dce7-4409-9fc9-ffe9ce036c77/kube-rbac-proxy/0.log" Oct 29 23:35:09.080831 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:35:09.080753 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-74bbd64f85-jjdcv_cfc225f9-52b5-4e36-916c-2467e3fe6e0b/thanos-query/0.log" Oct 29 23:35:09.203487 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:35:09.203464 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-74bbd64f85-jjdcv_cfc225f9-52b5-4e36-916c-2467e3fe6e0b/kube-rbac-proxy-web/0.log" Oct 29 23:35:09.328738 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:35:09.328707 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-74bbd64f85-jjdcv_cfc225f9-52b5-4e36-916c-2467e3fe6e0b/kube-rbac-proxy/0.log" Oct 29 23:35:09.406760 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:35:09.406678 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-59d4d9f4d8-v7rk9_bb8ee125-7b3a-48c1-8a10-0d2bf17a4eb0/manager/0.log" Oct 29 23:35:09.456797 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:35:09.456769 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-74bbd64f85-jjdcv_cfc225f9-52b5-4e36-916c-2467e3fe6e0b/prom-label-proxy/0.log" Oct 29 23:35:09.590598 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:35:09.590570 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-74bbd64f85-jjdcv_cfc225f9-52b5-4e36-916c-2467e3fe6e0b/kube-rbac-proxy-rules/0.log" Oct 29 23:35:09.648772 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:35:09.648742 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-59d4d9f4d8-v7rk9_bb8ee125-7b3a-48c1-8a10-0d2bf17a4eb0/manager/0.log" Oct 29 23:35:09.723905 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:35:09.723877 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-74bbd64f85-jjdcv_cfc225f9-52b5-4e36-916c-2467e3fe6e0b/kube-rbac-proxy-metrics/0.log" Oct 29 23:35:11.217374 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:35:11.217314 5057 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-zfxq6/perf-node-gather-daemonset-7hc5q" Oct 29 23:35:11.449830 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:35:11.449789 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-86c478bb88-6rr4f_2accd775-2173-4bfe-a413-7d243bd0820f/networking-console-plugin/0.log" Oct 29 23:35:12.562865 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:35:12.562828 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-5k5vp_494d8090-9803-4be0-a442-6108c1ebc201/manager/0.log" Oct 29 23:35:12.641565 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:35:12.641531 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-vj6md_aa6a190e-3283-458a-9dfd-2de7cb193fc5/console-operator/2.log" Oct 29 23:35:12.652076 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:35:12.652041 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-vj6md_aa6a190e-3283-458a-9dfd-2de7cb193fc5/console-operator/3.log" Oct 29 23:35:12.835965 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:35:12.835863 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-5k5vp_494d8090-9803-4be0-a442-6108c1ebc201/manager/0.log" Oct 29 23:35:13.898290 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:35:13.898262 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-9bbcb6fd8-w6wx5_cd69f88c-fed7-4f6a-8c88-8d5ac7642061/console/0.log" Oct 29 23:35:14.047238 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:35:14.047207 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-6dd789c8b7-9s8rp_612c3068-fbdf-4cd3-bf23-4099b112c655/download-server/0.log" Oct 29 23:35:14.634565 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:35:14.634533 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-59d4d9f4d8-v7rk9_bb8ee125-7b3a-48c1-8a10-0d2bf17a4eb0/manager/0.log" Oct 29 23:35:14.776453 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:35:14.776422 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-59d4d9f4d8-v7rk9_bb8ee125-7b3a-48c1-8a10-0d2bf17a4eb0/manager/0.log" Oct 29 23:35:15.728453 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:35:15.728395 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-58c9f5f46d-w5d4m_c54ef754-c91e-4bc5-8e83-08523aa09d37/volume-data-source-validator/0.log" Oct 29 23:35:17.521381 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:35:17.521322 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-5k5vp_494d8090-9803-4be0-a442-6108c1ebc201/manager/0.log" Oct 29 23:35:18.378554 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:35:18.378517 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-jjz47_6b34b3b1-84af-45f3-92ee-1ca720e8b263/dns/0.log" Oct 29 23:35:18.494393 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:35:18.494361 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-5k5vp_494d8090-9803-4be0-a442-6108c1ebc201/manager/0.log" Oct 29 23:35:18.505811 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:35:18.505791 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-jjz47_6b34b3b1-84af-45f3-92ee-1ca720e8b263/kube-rbac-proxy/0.log" Oct 29 23:35:19.027533 ip-10-0-140-74 kubenswrapper[5057]: E1029 23:35:19.027495 5057 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d2dnn" containerName="extractor" Oct 29 23:35:19.261429 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:35:19.261396 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-dpsd9_0e1c2279-b6d4-44e0-84e6-94eb1206d2a0/dns-node-resolver/0.log" Oct 29 23:35:20.812638 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:35:20.812605 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-d49c68d56-vpbg5_e2fb7b48-ff14-42e7-86cc-b77746dcccb0/registry/0.log" Oct 29 23:35:21.168914 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:35:21.168819 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-pcjl7_93dbae94-f92d-4157-a89b-d3c984a56013/node-ca/0.log" Oct 29 23:35:22.363923 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:35:22.363885 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6db586dd6c-cm4wj_4e861691-eac4-4f06-a6aa-b6553efdd358/istio-proxy/0.log" Oct 29 23:35:22.469770 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:35:22.469731 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6db586dd6c-nqs5j_e92e0a25-60cc-4250-a75d-537ed970fd75/istio-proxy/0.log" Oct 29 23:35:22.527738 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:35:22.527710 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6db586dd6c-vm822_ef4e2b67-0954-4fe1-a2b4-223771c24e79/istio-proxy/0.log" Oct 29 23:35:22.707117 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:35:22.707085 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-vsszj_6b4fbaae-ed3b-4a78-b38c-b3a245bf77da/payload-pre-processing/0.log" Oct 29 23:35:22.842732 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:35:22.842693 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5d8d7cc579-8cwjp_71d80eec-b5e7-41d0-b58d-8a8d9a6c769c/router/0.log" Oct 29 23:35:23.424623 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:35:23.424580 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6db586dd6c-cm4wj_4e861691-eac4-4f06-a6aa-b6553efdd358/istio-proxy/0.log" Oct 29 23:35:23.543087 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:35:23.543053 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6db586dd6c-nqs5j_e92e0a25-60cc-4250-a75d-537ed970fd75/istio-proxy/0.log" Oct 29 23:35:23.612054 ip-10-0-140-74 kubenswrapper[5057]: I1029 23:35:23.612024 5057 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6db586dd6c-vm822_ef4e2b67-0954-4fe1-a2b4-223771c24e79/istio-proxy/0.log"